On March 5th at approximately 1510 PST (2310 UTC) customers on our European cluster were unable to access SEDNA. Our Development Team was alerted that our event API had seen a larger than expected volume of calls which resulted in SEDNA becoming inaccessible. At 1557 PST (2357 UTC) the task queue was back to zero and customers regained full access to SEDNA. We are greatly sorry for the disruption to the customers on our European cluster and have taken immediate action to prevent this specific issue from recurring.
Why this happened.
Our event API was being heavily which caused the database to be overused and eventually stopped responding to requests. This was caused by a combination of higher than normal mail volumes (due to historical import) and traffic from a third party integration. We are now in the process of improving the the performance of the event API and the 3rd party integration which we expect to prevent any recurrence.
1. Improve the efficiency of our event API which will result in better handling of greater volumes of mail ingestion - IN PROGRESS
2. Improved database monitoring to ensure we are aware of this issue before any customers are impacted - COMPLETE