SEDNA Incident

23 July 2019

Summary

On July 23rd at ~1000PDT (~1700 UTC) SEDNA's message ingest experienced a delay in sending and receiving messages. During this time, some users may have observed messages arriving later than expected along with intermittent delays in sending messages. All other functions within SEDNA, search, compose, tagging, etc, were fully operational.

Why this happened

An inbound message containing irregular message content failed our import process. Our team attempted to manually import said message which resulted in our ingest servers to throttle processing items in our task queue.

Actions and opportunities for improvement

To prevent this issue from happening again, our team has either begun, or completed the following:

1. Update our internal process guides for failed message import. This will prevent the manual re-import of messages of this type - COMPLETE
2. Improve our internal monitoring processes to capture this type of failure event earlier - IN PROGRESS
3. Improve import process to better parse messages of this content type - IN PROGRESS