Message Loading Issue
Incident Report for SEDNA Systems
Postmortem

Message Loading Issue

28 January 2019

Summary

On 28 January 2019 at 09:28 UTC our team discovered that the Message List and Search were not loading any content and that this affected all customers.

Why this happened

At approximately 09:20 an update to SEDNA was released. This update was not believed to have any impact on SEDNA’s existing features or functionality. Unfortunately, the search endpoint was affected resulting in a temporary loss of service.

Actions and opportunities for improvement

Once the issue was identified as a faulty deployment it was rolled back to the last known good version. As a result of this incident we have undertaken the following actions:

Increase the speed of application deployments to ensure that rollbacks take as little time as possible - IN PROGRESS

Further increase test coverage of new deployments to ensure that any fault prevents the deployment from succeeding - IN PROGRESS

Improve alerting thresholds to ensure application errors are raised more rapidly - COMPLETED

Posted Jan 30, 2019 - 11:09 UTC

Resolved
The list not loading incident has been resolved.
Posted Jan 28, 2019 - 10:18 UTC
Monitoring
The team has rectified the issue causing the list not to load and all lists should now be loading as expected. We will continue to monitor.
Posted Jan 28, 2019 - 10:05 UTC
Identified
Our team has identified the cause of the list not loading issue and is working urgently to resolve this.
Posted Jan 28, 2019 - 09:45 UTC
Investigating
The team is currently investigating an issue wherein no messages are loading in SEDNA.
Posted Jan 28, 2019 - 09:35 UTC