Unexpected SMS notifications
Incident Report for Atlassian Statuspage
Postmortem

Last Tuesday, Statuspage inadvertently sent many SMS messages to people who did not expect any notifications. We realize these messages would have been concerning to anyone affected, whether as a direct user of Statuspage or as a subscriber to any one of our customers' pages. I'm very sorry for any disruption this may have caused you or your customers, and I wanted to detail how we're committed to avoiding further unwanted communication.

What happened

On March 13th at 17:05 UTC, an engineer inadvertently sent notifications to 36,981 US SMS subscribers while attempting to simplify subscription management for older subscribers, some of who had since ended their subscription. These affected subscribers received an SMS from "78774" indicating that they were now subscribed to status updates, even if they had never sent an SMS to that specific number or otherwise opted in to receive messages from it.

We have a standard policy of requiring peer review on all changes to production code; however, an engineer used a mechanism intended for resolving support cases which bypassed this safeguard, and the changes made affected significantly more subscribers than intended. In addition, because we currently store historical subscriber data to aid in troubleshooting support cases and understand our customers' usage of Statuspage, these notifications also impacted some subscribers who had previously ended their subscription.

Remediation

Subscribers who received this message should not be concerned - they will not receive further such messages. Existing subscribers should receive updates as normal as their subscription status was not affected. People who have unsubscribed, or who subscribed only to incidents that since been resolved, should not receive any further messages. Page owners do not need to take any action.

Moving forward, we are implementing the following changes to ensure this does not happen again:

  1. We are building stricter safe guards against accidentally modifying the production data without proper review, including the support mechanism used here.
  2. We are changing the code that caused this problem to make it difficult to accidentally trigger side effects such as sending notifications.
  3. We are evaluating how best to clean up old subscribers to make it difficult to accidentally contact someone who has not recently interacted with our service through SMS.

Next steps

If you have any questions or concerns about this incident, please reach out to hi@statuspage.io for further conversation. Thank you all for your continued support.

-Duane Bailey, Senior Software Engineer

Posted 4 months ago. Mar 19, 2018 - 10:40 PDT

Resolved
No further erroneous SMS messages have been sent, and this issue is resolved. We will be posting a postmortem with further details shortly.
Posted 4 months ago. Mar 13, 2018 - 15:49 PDT
Monitoring
We have determined that no further erroneous confirmation messages are being sent. We are continuing to examine the overall impact.
Posted 4 months ago. Mar 13, 2018 - 14:07 PDT
Identified
During a data migration, SMS subscription confirmation messages were erroneously sent to a set of phone numbers that were already subscribed to notifications from various pages. We are working to ensure that no further erroneous confirmations are sent. Those that did receive this message may disregard its call to action.
Posted 4 months ago. Mar 13, 2018 - 13:20 PDT
Investigating
We have received reports of unexpected SMS notifications being sent. We are investigating this now and will update this incident with additional information as it becomes available.
Posted 4 months ago. Mar 13, 2018 - 12:52 PDT
This incident affected: Notifications (SMS).