Summary

On 03/06/2020, between 02:38pm - 02:49pm, public pages had high level of errors and private status pages were unavailable. This event was triggered by a burst of requests erroneously bypassing our caching layer.

Cause

Requests to status pages are cached to ease the load on the web servers. On 03/06/2020 at 2:38pm, a large volume of requests failed to be appropriately normalized by our caching layer and ended up bypassing it completely. This in turn caused a backlog of web requests, resulting in 500 errors being served to public status pages and private pages being unavailable for 11 minutes.

What are we changing going forward?

Reliability and uptime for our services remain top priority. We have already adjusted our caching configuration to account for this pattern and are planning more work hardening our request caching process.

We apologize for the disruption in our service as a result of this incident and thank you for trusting us with your incident communication. Please contact us in case you have any further questions.