28 January 2019
On 28 January 2019 at 09:28 UTC our team discovered that the Message List and Search were not loading any content and that this affected all customers.
At approximately 09:20 an update to SEDNA was released. This update was not believed to have any impact on SEDNA’s existing features or functionality. Unfortunately, the search endpoint was affected resulting in a temporary loss of service.
Once the issue was identified as a faulty deployment it was rolled back to the last known good version. As a result of this incident we have undertaken the following actions:
Increase the speed of application deployments to ensure that rollbacks take as little time as possible - IN PROGRESS
Further increase test coverage of new deployments to ensure that any fault prevents the deployment from succeeding - IN PROGRESS
Improve alerting thresholds to ensure application errors are raised more rapidly - COMPLETED