Summary

On the 28th June, 2023, the SEDNA Platform experienced a partial loss of service. A number of customers served by the North American server infrastructure were not able to view their recently sent messages or see new messages in their Inbox. We deeply regret the inconvenience caused during the five-hour and four-minute duration of this issue.

‌

We want to assure you that no messages were lost during this incident. However, there was a delay in processing both inbound and outbound messages as some sent messages were temporarily held in a staging queue, which were then delivered correctly at the end of the incident window.

‌

Remediation and Prevention

‌

An incident of this nature receives Sedna’s highest level of scrutiny to ensure we can provide our customers with full confidence in the system. Following the incident the team conducted a series of follow ups:

‌

• Engineering Retro - documentation of the specific actions which caused the incident, the remediation, and the alerts through out to identify where / how improvements are needed
• Engineering / CS Coordination Retro - confirmation of coordination between Support and Engineering to ensure we were providing the best possible support to our customers given the difficult situation
• Leadership Team Retro - review with Sedna leadership on the issue, and specific actions which will be taken to ensure it is not repeated

‌

As a result of the reto’s listed above. SEDNA has taken specific action to ensure we reduce the likelihood of incidents like this in the future, and has developed a full plan of action 

Those actions include:

‌

• A full detailed code review of the issue in question to identify the offending sections, how the code was developed, and what requirements were being targeted which produced such an error
• A full detailed review of the QA process for the offending code to determine if additional measures could / should have been taken to ensure such a bug cannot be introduced in the future
• A full review of the queueing and alerting systems used to identify similar issues 
• A review of potential rollback procedures to include analysis of queue impacts so that we might faster resolve any similar issues in the future
• A scheduled review of our autotagging specific development process to reduce the risk of inappropriate tagging and improved testing procedures
• Consideration of enhancements to improve the queue processing approach that impacts this area of service

‌

This is only an initial list of actions taken. As we work through the above we expect additional specific actions to be identified and remedied. 

‌

What you can expect from SEDNA

‌

We understand the critical nature of the services SEDNA provides your business. We will continue to communicate with customers to answer any questions and ensure we do our best to provide a seamless customer experience. We apologize for any issues these events may have caused. 

‌

Please reach out directly to SEDNA Support (support@sedna.com) with any questions.