Unexpected SMS notifications
Incident Report for Atlassian Statuspage

Last Tuesday, Statuspage inadvertently sent many SMS messages to people who did not expect any notifications. We realize these messages would have been concerning to anyone affected, whether as a direct user of Statuspage or as a subscriber to any one of our customers' pages. I'm very sorry for any disruption this may have caused you or your customers, and I wanted to detail how we're committed to avoiding further unwanted communication.

What happened

On March 13th at 17:05 UTC, an engineer inadvertently sent notifications to 36,981 US SMS subscribers while attempting to simplify subscription management for older subscribers, some of who had since ended their subscription. These affected subscribers received an SMS from "78774" indicating that they were now subscribed to status updates, even if they had never sent an SMS to that specific number or otherwise opted in to receive messages from it.

We have a standard policy of requiring peer review on all changes to production code; however, an engineer used a mechanism intended for resolving support cases which bypassed this safeguard, and the changes made affected significantly more subscribers than intended. In addition, because we currently store historical subscriber data to aid in troubleshooting support cases and understand our customers' usage of Statuspage, these notifications also impacted some subscribers who had previously ended their subscription.


Subscribers who received this message should not be concerned - they will not receive further such messages. Existing subscribers should receive updates as normal as their subscription status was not affected. People who have unsubscribed, or who subscribed only to incidents that since been resolved, should not receive any further messages. Page owners do not need to take any action.

Moving forward, we are implementing the following changes to ensure this does not happen again:

  1. We are building stricter safe guards against accidentally modifying the production data without proper review, including the support mechanism used here.
  2. We are changing the code that caused this problem to make it difficult to accidentally trigger side effects such as sending notifications.
  3. We are evaluating how best to clean up old subscribers to make it difficult to accidentally contact someone who has not recently interacted with our service through SMS.
Next steps

If you have any questions or concerns about this incident, please reach out to hi@statuspage.io for further conversation. Thank you all for your continued support.

-Duane Bailey, Senior Software Engineer

Posted Mar 19, 2018 - 10:40 PDT

No further erroneous SMS messages have been sent, and this issue is resolved. We will be posting a postmortem with further details shortly.
Posted Mar 13, 2018 - 15:49 PDT
We have determined that no further erroneous confirmation messages are being sent. We are continuing to examine the overall impact.
Posted Mar 13, 2018 - 14:07 PDT
During a data migration, SMS subscription confirmation messages were erroneously sent to a set of phone numbers that were already subscribed to notifications from various pages. We are working to ensure that no further erroneous confirmations are sent. Those that did receive this message may disregard its call to action.
Posted Mar 13, 2018 - 13:20 PDT
We have received reports of unexpected SMS notifications being sent. We are investigating this now and will update this incident with additional information as it becomes available.
Posted Mar 13, 2018 - 12:52 PDT
This incident affected: Notifications (SMS).