On Thursday, July 4th, our internal monitoring system alerted us that SEDNA was experiencing higher than expected load times. This resulted in a major outage for customers who access SEDNA via www.sednanetwork.com with some users not able to log in to SEDNA or observed long load times while logging in or searching within SEDNA.
Our team released an update to SEDNA to improve the responsiveness of the message lists refresh functionality. Once this update reached our production environment, it triggered an increase in calls to our elastic search infrastructure that were not produced during testing on our test environment.
Due to this unforeseen spike in search traffic our team rolled back the update to return SEDNA to expected performance.