Summary

After memory issues experienced on 3/28 we decided to move Memcached off of our primary database server and onto its own infrastructure. This change required a configuration change and then a code deploy. We also increased the maximum number of connections our PostgreSQL database server would allow. This required a restart of the database service.

The maintenance went longer than expected because of two separate issues, but no data was lost, and only web-tier downtime occurred.

Cause

After stopping all traffic and background workers to perform the maintenance, our deploy process was sluggish due to new gems needing to be installed and new code needing to be fetched from Github. A lack of "pre-deploy process" on our end meant that more downtime was incurred while we waited for these two tasks to complete.

Once the initial code deploy was completed, the web tier processes failed to boot due to a misconfiguration with the security groups allocated with our web host and the new Memcached infrastructure. This was not immediately apparent since some nodes were able to connect while others were not. We also failed to test this change on one server before moving forward with pushing it out to all of our production fleet.

Resolution

[FUTURE] The deploy had stalled when trying run some of the installation and building steps. We are looking into a couple of options we have to reduce the amount of time needed for these steps of a deploy. This includes staging the code, running the bundle install and not updating the server to start using the new code until all new dependencies have been verified.

[COMPLETE] Moving forward any changes that involve infrastructure will be done in a way to use a canary deploy to validate the change in production and limit the possible impact. This means picking a server involved with the change in production, removing it from the load balancer, making the change and then verifying the change is working correctly before deploying it to the entire production environment.

Timeline

7:00 PM MDT The new environment variable has been added and verified across production.

8:00 PM MDT Maintenance has started. A graceful stop of all background workers and the Unicorn HTTP server. Stopped PostgreSQL replication, restarted the PostgreSQL services one at a time with the change to maximum connections, and then verified the health of the database. Turned replication back on and verified replication was working.

08:20 PM MDT (time is approximate) started code deploy and waited for the code to be checked out and the bundle install steps to complete. Encountered errors where the bundle install and asset building took an extremely long time. It appeared that the deploy had stalled and we decided to stop and restart the deploy.

09:12 PM MDT The deploy completed and production was unable to communicate with the new Memcached infrastructure.

09:30 PM MDT Determined that the new Memcached cluster was configured to use the wrong security group. Updated the cluster to use the correct group and service was restored.