All systems are fully operational again and all incoming data that was buffered during the outage is fully available again. We are going to conduct a thorough RCA and post a Postmortem explaining the incident as well as the actions derived to prevent similar issues in the future.
Posted Nov 06, 2024 - 21:02 UTC
Update
Cluster recovery is still underway and we are working with our Database vendor to recover the issue. Unfortunately we don't have an ETA for when the issue will be resolved but will post another update at 12:45
Posted Nov 06, 2024 - 10:51 UTC
Update
We are still working on recovering the cluster Unfortunately we don't have an ETA for when the issue will be resolved but will post another update at 11:45
Posted Nov 06, 2024 - 09:45 UTC
Identified
Our main analytics database cluster is currently offline due to a failed upgrade operation. We are currently working to recover but currently query and export functionality is unavailable. All incoming data is being stored and will be backfilled once the incident is resolved.
Posted Nov 06, 2024 - 08:10 UTC
This incident affected: Analytics Service (Analytics Ingress, Query Service, Export Service).