09 Sep 2021
On September 9th 2021 between 08:34 UTC and 09:59 UTC users were unable to login to or use SEDNA. This was due to a failure in our database storage layer which caused our database to failover and the application failed to re-connect gracefully.
There was an unforeseen failure in our storage layer and our redundancy setup meant that we should have seen minimal downtime however this was not the case. Our application server's DNS cache did not allow for our application instances to reconnect to our failover database until we manually intervened.
Actions and opportunities for improvement
We internally reviewed this incident and have agreed to the following action:
2. Improve reporting and alerting so we know about these issues earlier - IN PROGRESS