SEDNA Not Loading Messages

Incident Report for Sedna Systems

Postmortem

Incident Report

SEDNA Incident

11 Jan 2022

Summary

On January 11, 2022 users on our default cluster experienced issues with SEDNA not loading messages being received by teams.

Why this happened

As part of improvements SEDNA was making to our Search feature there was a temporary issue with importing data which caused a spike in load on our servers which resulted in the degraded performance users experienced.

Actions and opportunities for improvement

Improve logging and reporting on the search functionality - COMPLETE
Revisit and improve change management process - IN PROGRESS

Posted Jan 27, 2022 - 14:56 GMT

Resolved

This incident has been resolved.

Posted Jan 11, 2022 - 15:58 GMT

Update

Our Engineering team has resolved the issue and all messages have now been loaded in SEDNA.

Posted Jan 11, 2022 - 15:58 GMT

Update

We are continuing to monitor the situation with SEDNA not loading messages.

Posted Jan 11, 2022 - 14:57 GMT

Monitoring

Our Engineering team has deployed a fix and are seeing that SEDNA is stabilising. We are continuing to monitor the situation.

Posted Jan 11, 2022 - 14:28 GMT

Identified

Our team has identified the issue that is causing the message to not load in SEDNA and are actively looking at ways to resolve this.

Posted Jan 11, 2022 - 14:17 GMT

Update

Our team continues to investigate the issue which is causing a long delay in messages being loaded in SEDNA

Posted Jan 11, 2022 - 13:51 GMT

Investigating

Our team is actively investigating an issue that is causing messages not to load in SEDNA.

Posted Jan 11, 2022 - 13:22 GMT

This incident affected: SEDNA - www.sednanetwork.com and SEDNA - Private Hosting.