Summary

Two new servers were added to our environment to replace two existing servers. Each of these servers run a secondary copy of our PostgreSQL and Redis databases along with running the management portal and API for StatusPage.

After verifying that both PostgreSQL and Redis were healthy the two servers that were being replaced were stopped. From the naming scheme used it was not entirely clear that these servers were serving requests and once the existing servers failed the health checks in the load balancer, they were marked as down and then the management portal and API went offline.

Cause

Two servers were stopped and then the load balancer severing the management portal and API traffic did not have any healthy servers to handle their traffic.

The naming scheme used did not make it absolutely clear that these servers handled public traffic.

This lead to the two new servers were never placed into the load balancer and when the two existing servers were stopped it was never a concern that they were serving traffic. Only the health of the PostgreSQL and Redis databases was considered during the change.

Resolution

Once we were notified by our monitoring it was clear that this was caused from the change and it was obvious that the two new servers were not placed into the load balancer. They were then added into it and then the management portal and API came back online.

Mitigation

[COMPLETE] We have moved the secondary copies of the PostgreSQL and Redis databases onto a new server and have renamed the two existing servers to make it clear what is running on them from the server name.

[IN PROGRESS] We are in the process of updating our load balancers to use a failover pool when it detects all servers are marked as down. In this case the load balancer would have seen the two existing servers were down and it would then start using the servers handling traffic to the status pages to serve traffic for the management portal and API.

[FUTURE] We are also planning to build tools around making these types of changes using statuspage-specific tools instead of using the web UI from our hosting provider so we can implement safeguards to prevent this from happening again.

Timeline

3:05 PM MDT Two new servers are brought online and start replicating the PostgreSQL and Redis databases.

3:17 PM MDT The two existing servers are stopped using the web UI from AWS.

3:23 PM MDT We are alerted that the management portal and API was down.

3:24 PM MDT We see that the two new servers are not in the load balancer and that the two stopped instances had been marked down by the load balancer.

3:25 PM MDT The two new servers are added back into the load balancer.

3:26 PM MDT Our monitoring reported that the management portal and API was back online.