At approximately 8:00AM PST, our outbound traffic was greatly reduced by an overloaded NAT server. Our outbound bandwidth in the us-west-2 region of AWS was saturated and we were unable to contact services outside our internal network.

• By 8:12AM PST, our on-call staff was notified to the problem.
• By 8:20AM PST, we had identified the inability to establish an outside connection as the primary cause of the loss of functionality.
• By 8:50AM PST, we had escalated to the Atlassian global incident management team.
• By 9:20AM PST, the incident was escalated to the Atlassian network engineering team.
• By 10:34AM PST, a network engineer managed to identify and began to mitigate the cause. Service started to return to normal.
• By 11:22AM PST, full functionality had resumed.

During the affected time, notifications and metric updates were severely delayed. Users making use of Google authentication would have been unable to access the management dashboard. Users logged into the management dashboard would have been unable to search for subscribers and incidents.

Since restoring service, we have implemented a large number of changes to improve how we allocate capacity and manage consumption of our network. Additionally, we are homing in on the several factors that led to this incident occurring, and we are working the teams involved to ensure this does not occur in the future.

Going forward, we are implementing the ability to fail-over to another AWS region. This would have allowed us to fully restore functionality on the order of minutes, not hours.