Unable to load SEDNA
Incident Report for SEDNA Systems
Postmortem

SEDNA Incident

14 Oct 2021

Summary
On October 14th 2021 between 14:43 UTC and 15:14 UTC an upgrade to the SEDNA infrastructure resulted in our search platform not returning any results and no messages were being returned to the Message List. As a result, teams could not view messages during this time.

Why this happened
This update in our application configuration resulted in too few servers trying to handle traffic for one of our production clusters in North America causing the interruption to service and degraded performance. The number of servers was then scaled to a value appropriate for that cluster and full service was restored.

Actions and opportunities for improvement
We internally reviewed this incident and have agreed to the following action:

  1. Added correct autoscaling configurations for ALL clusters to prevent this specific issue from happening again.
Posted Oct 18, 2021 - 18:56 UTC

Resolved
A change in application configuration led to too few servers trying to handle traffic for one of our production clusters in North America causing a brief interruption to service and degraded performance. The number of servers has been scaled to a value appropriate for that cluster and service has been restored
Posted Oct 14, 2021 - 15:14 UTC
Update
We are continuing to monitor for any further issues.
Posted Oct 14, 2021 - 15:13 UTC
Monitoring
We can confirm the incident has been isolated and only impacted our customer hosted on our default cluster. We are currently monitoring the issue and will continue to share any updates.
Posted Oct 14, 2021 - 15:09 UTC
Investigating
We are currently experiencing an issue where users are unable to load SEDNA. The team is currently investigating the issue.
Posted Oct 14, 2021 - 15:04 UTC
This incident affected: SEDNA - www.sednanetwork.com.