SEDNA Currently Unavailable on www.sednanetwork.com
Incident Report for SEDNA Systems
Postmortem

SEDNA Incident

04 July 2019

Summary

On Thursday, July 4th, our internal monitoring system alerted us that SEDNA was experiencing higher than expected load times. This resulted in a major outage for customers who access SEDNA via www.sednanetwork.com with some users not able to log in to SEDNA or observed long load times while logging in or searching within SEDNA.

Why this happened

Our team released an update to SEDNA to improve the responsiveness of the message lists refresh functionality. Once this update reached our production environment, it triggered an increase in calls to our elastic search infrastructure that were not produced during testing on our test environment.

Actions and opportunities for improvement

Due to this unforeseen spike in search traffic our team rolled back the update to return SEDNA to expected performance.

  1. Increase our performance testing on AWS - IN PROGRESS
  2. Improve internal dark launch process on feature launch - IN PROGRESS
Posted Jul 10, 2019 - 00:05 UTC

Resolved
This incident has now been resolved and SEDNA has returned to full functionality. We are sorry for any users that were impacted during this incident. We will be conducting an internal review in the coming days and will publish our post mortem via the SEDNA Status Page. If you have any questions please reach out to support@sedna.com. Thank you.
Posted Jul 04, 2019 - 21:32 UTC
Monitoring
We have implemented a fix to resolve the issue preventing users from logging into SEDNA. Some users may experience longer load times and we are actively monitoring the situation and will advise once we return to optimal performance. Thank you for your patience. If you have any questions please reach out to support@sedna.com.
Posted Jul 04, 2019 - 20:45 UTC
Identified
We have currently identified an issue that is preventing users from logging into SEDNA. Users that sign in via www.sednanetwork.com/launch are currently impacted. We will update as soon as we have more information. If you have any questions please reach out to support@sedna.com.
Posted Jul 04, 2019 - 20:30 UTC
This incident affected: SEDNA - www.sednanetwork.com.