All data is backfilled and queries and exports are now returning up to date data. An RCA will be published once we have concluded the investigation as to what caused this incident.
Posted May 09, 2023 - 22:39 UTC
Update
The issue is resolved and we are currently backfilling the buffered data. We expect this process to take another 2 hours. Until then querying/exporting data collected between 19:10 UTC and 20:22 UTC will yield incomplete data. Current data is not affected and everything is working normally.
Posted May 09, 2023 - 21:34 UTC
Monitoring
We have resolved the issue - all inserts and queries are working normally again. Query latencies have also recovered although queries might still serve incomplete data that was buffered during the incident. We will monitor the situation and will start the process of backfilling the buffered data soon.
Posted May 09, 2023 - 21:01 UTC
Investigating
We are currently investigating an elevated error rate in our analytics backend database. All incoming data is being buffered and will be inserted once everything is fully stabilised but exports and queries might be serving stale data at the moment and/or fail.
Posted May 09, 2023 - 20:06 UTC
This incident affected: Analytics Service (Analytics Ingress, Query Service, Export Service).