Summary

The memory usage for our Redis database increased from a normal 150MB to 6GB within a period of two hours. While there is monitoring of the memory usage of Redis, there were no alerts configured to warn us of the increase in memory usage. As Redis sustained 6GB of memory usage it was eventually force-killed by the kernel. This brought the entire site down while Redis was restarted by Upstart and then the database was loaded back into memory. After the restart the memory usage for Redis stayed consistent at our normal 150MB.

Cause

Our infrastructure layout at the time of incident had a primary Redis database located in our primary data center, and that Redis database is then replicated out to two secondary Redis databases also located in our primary data center and one additional database located in our disaster recovery center. After testing and reviewing the data, we believe that the spike in memory was caused by an increase in latency between the Redis databases located in our primary and disaster recovery data centers.

As latency, and possibly the connection quality degraded, commands waiting to be streamed to the Redis database in our disaster recovery center would queue up in memory until they could be successfully flushed to the follower databases. This accounted for the increase in memory from our steady state value to the extremely large value.

Resolution

[COMPLETE] The stability of the site was restored after Linux restarted Redis and it’s memory usage returned to normal. The following Monday (3/30) we moved the Memcached service off of the primary database server and onto its own infrastructure to reduce the normal memory usage on the server and allow for future increases and unexpected spikes.

[COMPLETE] Alerting around the memory usage for these services was put into place.

[COMPLETE] Our disaster recovery data center has also been moved to closer proximity to our primary data center in an effort to decrease latency and increase network connection health.

Timeline

11:00 AM MDT Redis memory usage spiked to 600MB from our normal 140MB and continued to grow.

12:00 PM MDT Redis memory was over 2GB and continuing to grow

12:54 PM MDT Our monitoring starts seeing the site and service starting to flap and alerts our team. We start looking into the services and see the increase in memory on our primary database server running PostgreSQL, Redis, and Memcached.

1:26 PM MDT While we look to reduce the amount of memory used by Redis the service was OOM-killed by Linux’s out-of-memory process and then the database was loaded back into memory.

1:27 PM MDT The database was loaded back into memory and services returned to normal.