Mail arrival delays on www.sednanetwork.com
Incident Report for SEDNA Systems
Postmortem

SEDNA Incident

23 July 2019

Summary

On July 23rd at ~1000PDT (~1700 UTC) SEDNA's message ingest experienced a delay in sending and receiving messages. During this time, some users may have observed messages arriving later than expected along with intermittent delays in sending messages. All other functions within SEDNA, search, compose, tagging, etc, were fully operational.

Why this happened

An inbound message containing irregular message content failed our import process. Our team attempted to manually import said message which resulted in our ingest servers to throttle processing items in our task queue.

Actions and opportunities for improvement

To prevent this issue from happening again, our team has either begun, or completed the following:

  1. Update our internal process guides for failed message import. This will prevent the manual re-import of messages of this type - COMPLETE
  2. Improve our internal monitoring processes to capture this type of failure event earlier - IN PROGRESS
  3. Improve import process to better parse messages of this content type - IN PROGRESS
Posted Jul 26, 2019 - 22:22 UTC

Resolved
Our team has fully resolved the issue with inbound mail delays. We are sorry for the delays and will be conducting an internal review on what happened and why it happened. Once we've completed our internal review we will post it to our status page. If you have any questions please reach out to support@sedna.com.

Thank you
Posted Jul 23, 2019 - 18:19 UTC
Monitoring
Our team has resolved the issue and mail flow has returned to normal. All impacted messages are now arriving and will be timestamped at their initial intended point of arrival. We will continue to monitor the situation and will provide an update shortly. If you have any questions please reach out to support@sedna.com. Thank you.
Posted Jul 23, 2019 - 17:56 UTC
Investigating
Since approximately 1000 PDT (1700 UTC) we are experiencing sporadic mail arrival delays for customers on www.sednanetwork.com. Sending messages, accessing mail within SEDNA, and Search are fully operational. Our team is currently working towards a resolution and we will post updates as we progress. For any questions please reach out to support@sedna.com. Thank you for your patience.
Posted Jul 23, 2019 - 17:52 UTC
This incident affected: SEDNA - www.sednanetwork.com.