On February 4th SEDNA experienced a serious and long-lasting connectivity issue with our third-party provider of Elasticsearch services in the Eastern United States (AWS us-east-1) which caused a prolonged period of downtime for G2O and almost two-thirds of SEDNA's customers. Despite this connectivity issue, we restored partial service within two hours, including receiving your most recent messages, the ability to send messages, and access to the address book.
Our use of this infrastructure has been at a considerable cost and was intended to provide redundancy for such outages. It is not achieving this result.
We have provisioned an emergency backup of SEDNA using an alternative third-party provider (Amazon - AWS ESS) which will allow us to have SEDNA up and running, with minimal disruption, in the event of another connectivity issue. Our technical team has begun evaluating a permanent move to AWS ESS infrastructure.
A painful learning from these two incidents has been our need to be more resilient to third-party service outages, especially beyond the redundancy those service providers commit to us contractually. Following the immediate mitigation steps above, we will put in place mechanisms to improve our ability to recover from these kinds of incidents and ensure that SEDNA's core functionality remains available and usable.