Last Tuesday, Statuspage inadvertently sent many SMS messages to people who did not expect any notifications. We realize these messages would have been concerning to anyone affected, whether as a direct user of Statuspage or as a subscriber to any one of our customers' pages. I'm very sorry for any disruption this may have caused you or your customers, and I wanted to detail how we're committed to avoiding further unwanted communication.
What happened
On March 13th at 17:05 UTC, an engineer inadvertently sent notifications to 36,981 US SMS subscribers while attempting to simplify subscription management for older subscribers, some of who had since ended their subscription. These affected subscribers received an SMS from "78774" indicating that they were now subscribed to status updates, even if they had never sent an SMS to that specific number or otherwise opted in to receive messages from it.
We have a standard policy of requiring peer review on all changes to production code; however, an engineer used a mechanism intended for resolving support cases which bypassed this safeguard, and the changes made affected significantly more subscribers than intended. In addition, because we currently store historical subscriber data to aid in troubleshooting support cases and understand our customers' usage of Statuspage, these notifications also impacted some subscribers who had previously ended their subscription.
Remediation
Subscribers who received this message should not be concerned - they will not receive further such messages. Existing subscribers should receive updates as normal as their subscription status was not affected. People who have unsubscribed, or who subscribed only to incidents that since been resolved, should not receive any further messages. Page owners do not need to take any action.
Moving forward, we are implementing the following changes to ensure this does not happen again:
- We are building stricter safe guards against accidentally modifying the production data without proper review, including the support mechanism used here.
- We are changing the code that caused this problem to make it difficult to accidentally trigger side effects such as sending notifications.
- We are evaluating how best to clean up old subscribers to make it difficult to accidentally contact someone who has not recently interacted with our service through SMS.
Next steps
If you have any questions or concerns about this incident, please reach out to hi@statuspage.io for further conversation. Thank you all for your continued support.
-Duane Bailey, Senior Software Engineer