SEDNA Incident

4 November 2019

Summary

On November 4th, 2019 SEDNA experienced a degradation of the platform that made users unable to search within SEDNA or log in. This affected users on our privately hosted cluster meaning users who login via a custom URL such as sedna.sednanetwork.com.

Why this happened

A mechanism designed to monitor the health of SEDNA’s search infrastructure identified a potential problem with ElasticSearch - the database SEDNA uses to index messages. An automated response to that potential fault was activated which had the unintended consequence of degrading SEDNA’s performance.

Actions and opportunities for improvement

An investigation revealed that the health monitoring system activated unnecessarily and measures have been taken to ensure that it only responds to genuine problems with the search service’s health.

‌

1. Improve database performance metrics - COMPLETE
2. Ensure elastic search health monitoring only triggers alerts under appropriate conditions - COMPLETE
3. Optimised database access queries to ensure that in the event that the search service falls back to direct database access (i.e. Elastic Search is unavailable) that the queries used are performant. - COMPLETE
4. Elastic Search service version upgrades to enhance resiliency - IN PROGRESS