Unable to load SEDNA

Incident Report for Sedna Systems

Postmortem

SEDNA Incident

14 Oct 2021

Summary
On October 14th 2021 between 14:43 UTC and 15:14 UTC an upgrade to the SEDNA infrastructure resulted in our search platform not returning any results and no messages were being returned to the Message List. As a result, teams could not view messages during this time.

Why this happened
This update in our application configuration resulted in too few servers trying to handle traffic for one of our production clusters in North America causing the interruption to service and degraded performance. The number of servers was then scaled to a value appropriate for that cluster and full service was restored.

Actions and opportunities for improvement
We internally reviewed this incident and have agreed to the following action:

Added correct autoscaling configurations for ALL clusters to prevent this specific issue from happening again.

Posted Oct 18, 2021 - 19:56 BST

Resolved

A change in application configuration led to too few servers trying to handle traffic for one of our production clusters in North America causing a brief interruption to service and degraded performance. The number of servers has been scaled to a value appropriate for that cluster and service has been restored

Posted Oct 14, 2021 - 16:14 BST

Update

We are continuing to monitor for any further issues.

Posted Oct 14, 2021 - 16:13 BST

Monitoring

We can confirm the incident has been isolated and only impacted our customer hosted on our default cluster. We are currently monitoring the issue and will continue to share any updates.

Posted Oct 14, 2021 - 16:09 BST

Investigating

We are currently experiencing an issue where users are unable to load SEDNA. The team is currently investigating the issue.

Posted Oct 14, 2021 - 16:04 BST

This incident affected: SEDNA - www.sednanetwork.com.