Resolved -
All systems are fully operational again and all incoming data that was buffered during the outage is fully available again.
We are going to conduct a thorough RCA and post a Postmortem explaining the incident as well as the actions derived to prevent similar issues in the future.
Nov 6, 21:02 UTC
Update -
Cluster recovery is still underway and we are working with our Database vendor to recover the issue.
Unfortunately we don't have an ETA for when the issue will be resolved but will post another update at 12:45
Nov 6, 10:51 UTC
Update -
We are still working on recovering the cluster
Unfortunately we don't have an ETA for when the issue will be resolved but will post another update at 11:45
Nov 6, 09:45 UTC
Identified -
Our main analytics database cluster is currently offline due to a failed upgrade operation.
We are currently working to recover but currently query and export functionality is unavailable.
All incoming data is being stored and will be backfilled once the incident is resolved.
Nov 6, 08:10 UTC