Mail Ingestion Outage
Incident Report for SEDNA Systems
Postmortem

SEDNA Incident

28 May 2019

Summary

On 28 May 2019 at around 08:55 BST SEDNA was unable to ingest mail on our sednanetwork.com tenants (those who login at sednanetwork.com) and did not affect any tenants that are privately hosted. Emails that were being ingested ended up in all teams within the tenant.

Why this happened

This issue occurred due to a bug within our Update API, the changing of a team setting resulted in ingestion addresses being removed from all teams.

Actions and opportunities for improvement

The bug within the Update API has been resolved - COMPLETE

All ingested messages were updated to only show in the intended teams - COMPLETE

Posted May 31, 2019 - 09:20 UTC

Resolved
The incident has been resolved, all queued mail has now come into SEDNA.
Posted May 28, 2019 - 10:23 UTC
Update
Queued mail is now coming into SEDNA.
Posted May 28, 2019 - 09:54 UTC
Update
We are continuing to investigate this issue.
Posted May 28, 2019 - 09:03 UTC
Investigating
No new messages are arriving in SEDNA but are being queued up, minutes prior to the outage some teams observed mail being delivered to all teams on their own tenant, rather than intended teams.
Posted May 28, 2019 - 08:31 UTC
This incident affected: SEDNA - www.sednanetwork.com.