SEDNA Not Loading

Incident Report for SEDNA Systems

Postmortem

24 November 2020

Summary

On November 24th 2020 between the hours 8am GMT and 10am GMT SEDNA was inaccessible to some users and some users were experiencing degraded services.

Why this happened

SEDNA relies on a cloud provider for our search functionality and they began experiencing difficulties which in turn had an impact on SEDNA’s ability to load messages and function as expected. One node in a regional search cluster became inaccessible in a way that led to inconsistent and delayed responses. We worked with AWS to resolve the problem and in the mean-time initiated a redeployment of the search cluster - that redeployment succeeded and service was fully restored.

Actions and opportunities for improvement

Investigate root cause with AWS - IN PROGRESS
Improve degraded search mode to mitigate reliance on the search service - IN PROGRESS

‌ 3. Enhanced alarms around search service - COMPLETED

Posted Nov 26, 2020 - 10:18 UTC

Resolved

This incident has been resolved and all users will see SEDNA loading as expected, please reach out to support@sedna.com if you run into any issues.

Posted Nov 24, 2020 - 10:36 UTC

Monitoring

Our team has resolved the issue that was causing SEDNA not to load, users may need to refresh or log out/in. Incoming mail is being imported with some delay as the system catches up, we are actively monitoring the situation.

Posted Nov 24, 2020 - 10:17 UTC

Update

Our team is working to get SEDNA back up and running, some users may experience intermittent loading of SEDNA while our team works on resolving the backend issue. This is only affecting some tenants who login via URLs such as sedna.sednanetwork.com.

Posted Nov 24, 2020 - 09:58 UTC

Update

Our team continues to work towards resolving the issue with SEDNA not loading or being able to log in. Some users may experience intermittent loading of the list in SEDNA.

Posted Nov 24, 2020 - 09:25 UTC

Update

Our team is working towards a full resolution on the issue of SEDNA not loading or being unable to login. Some users will experience intermittent loading of SEDNA at this time.

Posted Nov 24, 2020 - 09:14 UTC

Update

Our team continues to works towards resolving the issue of SEDNA not loading, current performance is degraded and some users may experience issues with loading the list and logging in.

Posted Nov 24, 2020 - 09:01 UTC

Update

Our team continues to work towards resolving the issue with logging into and using SEDNA. Most users will be able to log in and use SEDNA at this time though performance may be degraded.

Posted Nov 24, 2020 - 08:46 UTC

Update

Our team continues to work towards resolving the issue with users being unable to login to or use SEDNA. Some users may experience intermittent loading of SEDNA during this time.

Posted Nov 24, 2020 - 08:38 UTC

Identified

Our team has identified the issue causing SEDNA to not load and are actively working towards resolving.

Posted Nov 24, 2020 - 08:32 UTC

Investigating

Our team is investigating an issue where users receive a spinner when using SEDNA or are unable to login. This is only affecting some users who login via a URL such as sedna.sednanetwork.com.

Posted Nov 24, 2020 - 08:25 UTC

This incident affected: SEDNA - Private Hosting.