Message Loading Issue
Incident Report for Sedna Systems
Postmortem

04 Feb 2019

Summary

On February 4th SEDNA experienced a serious and long-lasting connectivity issue with our third-party provider of Elasticsearch services in the Eastern United States (AWS us-east-1) which caused a prolonged period of downtime for G2O and almost two-thirds of SEDNA's customers. Despite this connectivity issue, we restored partial service within two hours, including receiving your most recent messages, the ability to send messages, and access to the address book.

Our use of this infrastructure has been at a considerable cost and was intended to provide redundancy for such outages. It is not achieving this result.

Actions and opportunities for improvement

We have provisioned an emergency backup of SEDNA using an alternative third-party provider (Amazon - AWS ESS) which will allow us to have SEDNA up and running, with minimal disruption, in the event of another connectivity issue. Our technical team has begun evaluating a permanent move to AWS ESS infrastructure.

A painful learning from these two incidents has been our need to be more resilient to third-party service outages, especially beyond the redundancy those service providers commit to us contractually. Following the immediate mitigation steps above, we will put in place mechanisms to improve our ability to recover from these kinds of incidents and ensure that SEDNA's core functionality remains available and usable.

Posted Feb 06, 2019 - 17:56 GMT

Resolved
We are now seeing all activity returned to normal. We are incredibly sorry for the downtime experienced and will be conducting an internal review to assess the root cause of the issue. Once complete we will post our post mortem to our status page.

For any outstanding questions please reach out to support@sedna.com. Thank you again,.
Posted Feb 04, 2019 - 16:58 GMT
Monitoring
Service has now been partially restored through the implementation of an interim fix by SEDNA technical teams. Recent and new content (inbound and outbound) will be visible and users can now access, send and receive content in SEDNA.

We are still awaiting resolution of the underlying incident reported by our third-party provider. As a result, while access to SEDNA has been restored, historical email may not yet be available.

We will continue to monitor service and provide further updates as they become available.
Posted Feb 04, 2019 - 06:55 GMT
Identified
An issue has been identified with a third-party service used by SEDNA which is resulting in message lists failing to load for all users on our production cluster hosted in the US-East region. An interim solution is in the process of being implemented by our internal teams now with the aim of restoring service to users until the underlying root cause is identified and resolved.

We will provide further updates as they become available.
Posted Feb 04, 2019 - 06:00 GMT
Investigating
The team is currently investigating an issue wherein SEDNA is not properly loading messages.
Posted Feb 04, 2019 - 04:55 GMT
This incident affected: SEDNA - www.sednanetwork.com.