SEDNA not Loading
Incident Report for SEDNA Systems
Postmortem

SEDNA Incident

4 November 2019

Summary

On November 4th, 2019 SEDNA experienced a degradation of the platform that made users unable to search within SEDNA or log in. This affected users on our privately hosted cluster meaning users who login via a custom URL such as sedna.sednanetwork.com.

Why this happened

A mechanism designed to monitor the health of SEDNA’s search infrastructure identified a potential problem with ElasticSearch - the database SEDNA uses to index messages. An automated response to that potential fault was activated which had the unintended consequence of degrading SEDNA’s performance.

Actions and opportunities for improvement

An investigation revealed that the health monitoring system activated unnecessarily and measures have been taken to ensure that it only responds to genuine problems with the search service’s health.

  1. Improve database performance metrics - COMPLETE
  2. Ensure elastic search health monitoring only triggers alerts under appropriate conditions - COMPLETE
  3. Optimised database access queries to ensure that in the event that the search service falls back to direct database access (i.e. Elastic Search is unavailable) that the queries used are performant. - COMPLETE
  4. Elastic Search service version upgrades to enhance resiliency - IN PROGRESS
Posted Nov 06, 2019 - 16:03 UTC

Resolved
SEDNA is now working as expected and users are able to login, search and load messages. If you run into any further issues please reach out to support@sedna.com.
Posted Nov 04, 2019 - 14:58 UTC
Monitoring
Our team have been able to resolve the issue that was causing SEDNA to not load and resolved the issue where users were unable to login. We continue to monitor performance.
Posted Nov 04, 2019 - 14:47 UTC
Identified
Our team have identified the issue causing SEDNA to be slow and have rectified the issue. We are seeing SEDNA performance coming back as expected and continue to monitor performance.
Posted Nov 04, 2019 - 14:34 UTC
Investigating
Our team are currently investigating an issue with SEDNA with private hosted SEDNA tenants, those that login with a custom URL such as sedna.sednanetwork.com. Users will be unable to login to SEDNA or experience slow load or pages not loading.
Posted Nov 04, 2019 - 14:28 UTC
This incident affected: SEDNA - Private Hosting.