Outbound Messages Are Not Being Delivered

Incident Report for Sedna Systems

Postmortem

SEDNA Incident

27 Jan 2021

Summary

On January 27th, 2021 at approximately 1648 PST (0048 UTC) outbound messages from SEDNA were halted and sent messages were being returned with a non delivery report stating an issue with the SMTP API header. Our Engineering team deployed an update as a hotfix for the issue which restored service at approximately 1723 PST (0123 UTC) resulting in a ~35 minute period of dropped messages.

Why this happened

Our third-party mail delivery service experienced an issue with how they handle a specific string in the SMTP API that handles bounce and blocked email suppression lists. This resulted in outbound messages from SEDNA being dropped until our team deployed the hotfix. Once our third-party provider resolved the issue, we reverted our hotfix to restore our previous suppression list management via the SMTP API.

Actions and opportunities for improvement

A key takeaway from this issue is that our delivery pipeline prevented us from deploying our hotfix sooner.

Review delivery pipeline to improve deployment speeds - IN PROGRESS

Posted Jan 30, 2021 - 00:20 GMT

Resolved

The team has resolved the issue with the outbound message failures and SEDNA is fully operational. Any messages that failed, and subsequently received a non-delivery report (NDR) will need to be resent. We will be conducting a full review of this issue and will post our findings on the SEDNA Status Page in the coming days. Thank you for your patience. For any inquiries please reach out to support@SEDNA.com.

Posted Jan 28, 2021 - 02:25 GMT

Update

The team has deployed the hot fix and we've restored outbound message sending. All messages that failed to deliver will need to be resent. We are continuing to investigate the root cause of the issue and will post updates as we continue to investigate. As of now, full service has been returned to SEDNA. If you have any questions, please reach out to support@SEDNA.com. Thank you.

Posted Jan 28, 2021 - 01:15 GMT

Update

We are continuing to investigate the issue that is causing outbound message failures. We do have a hot fix currently being deployed that will restore outbound message functionality. This temporary fix will temporarily prevent non delivery reports while we continue to investigate. Messages that failed to deliver will need to be resent. Thank you for your patience while we continue to resolve this issue. For any questions please reach out to support@SEDNA.com.

Posted Jan 28, 2021 - 01:02 GMT

Investigating

We are currently investigating an issue where outbound messages are not being delivered. The rest of SEDNA functionality is currently working as expected.

Posted Jan 28, 2021 - 00:39 GMT

This incident affected: SEDNA - www.sednanetwork.com and SEDNA - Private Hosting.