Unable to load SEDNA

Incident Report for Sedna Systems

Postmortem

Incident Report

SEDNA Incident

09 Sep 2021

Summary

On September 9th 2021 between 08:34 UTC and 09:59 UTC users were unable to login to or use SEDNA. This was due to a failure in our database storage layer which caused our database to failover and the application failed to re-connect gracefully.

Why this happened

There was an unforeseen failure in our storage layer and our redundancy setup meant that we should have seen minimal downtime however this was not the case. Our application server's DNS cache did not allow for our application instances to reconnect to our failover database until we manually intervened.

Actions and opportunities for improvement

We internally reviewed this incident and have agreed to the following action:

Ensure application instances can recover gracefully from a database failover. - IN PROGRESS

‌ 2. Improve reporting and alerting so we know about these issues earlier - IN PROGRESS

Posted Sep 22, 2021 - 23:59 BST

Resolved

Our team has confirmed all instances of SEDNA are loading and working as expected. Users affected may need to refresh their browsers. If you continue to see any issues please reach out to support@sedna.com.

Posted Sep 09, 2021 - 10:59 BST

Monitoring

We can confirm the incident has been isolated and only impacted our customer hosted on our Dublin cluster. We have now deployed a fix, and restarting the instances and are seeing speedy successful logins.

Posted Sep 09, 2021 - 10:38 BST

Identified

Our team has identified what is causing the issue where SEDNA is showing spinners and is actively working on it.

Posted Sep 09, 2021 - 10:17 BST

Investigating

We are currently experiencing an issue where users are unable to load SEDNA. The team is currently investigating the issue.

Posted Sep 09, 2021 - 09:34 BST

This incident affected: SEDNA - Private Hosting.