Summary

A PostgreSQL node met an out-of-memory condition that caused the database processes to be restarted. This caused a brief two minute outage.

Cause

Memory pressure on a database server caused the kernel to force-kill the database processes. This brought down the site for two minutes while PostgreSQL was restarted by Upstart.

Resolution

[PLANNED] Migrate the database to a server with a larger amount of memory.

[COMPLETE] To help reduce the memory pressure on the server we’ve lowered the TTL on some of our caches to reduce the amount of items stored in Redis. This reduces the amount of memory used by Redis on the same server that serves the PostgreSQL database.

[TESTED on STAGE] We will be making some virtual memory changes to help reduce the impact until the server migration is completed.

Timeline

4:11 PM MST A scheduled task tries to run and is unable to locate memory.

4:12 PM MST The PostgreSQL processes our OOM-killed by Linux’s out-of-memory process and then the database was started by Upstart.

4:14 PM MST The database was back online and we were successfully serving traffic again.