Incident Post-Mortem – Encoding Cleanup Gap-Lock

Date: 2025-10-16
Duration: 07:49–08:30 UTC (41 minutes)
Customer Impact: Calls to the https://api.bitmovin.com/v1/encoding/encodings/{encoding_id}/muxings/ endpoints failed, leading to workflow stoppages. The failure rate remained below monitoring thresholds, so the issue was not immediately detected.

Summary

At 07:49 UTC, our encoding cleanup process (responsible for deleting old encodings outside of retention) triggered a gap-lock on one of our database tables. While the long-running delete transaction was active, the API was unable to insert new records into the table.

This directly impacted the muxings endpoints, which failed during this period and caused encoding workflows to stop. Because the error rate was below our monitoring alert thresholds, the incident went unnoticed until the team was alerted at 08:20 UTC.

Root Cause

The issue was caused by outdated MySQL table statistics, which led the MySQL query optimizer to select an inefficient execution plan:

• Instead of using an index scan on the large table, the query optimizer chose a full table scan.

• Under REPEATABLE_READ isolation, this resulted in gap-locks across wide portions of the table.

• In practice, this behaved like a table-level lock, blocking all inserts until the cleanup query was stopped.

Timeline

• 07:49 UTC – Encoding cleanup process starts. Optimizer chooses full table scan → gap-locks prevent inserts.

• 07:49–08:20 UTC – Calls to /encoding/encodings/{encoding_id}/muxings/ fail, workflows stop. Monitoring does not alert as errors are below configured threshold.

• 08:20 UTC – Engineering team is alerted.

• 08:20–08:30 UTC – Cleanup process identified as root cause. Process is stopped, lock released.

• 08:30 UTC – Incident resolved, API resumes normal operation.

Immediate Actions

After resolving the incident, we immediately stopped the cleanup process and investigated why it suddenly caused issues despite having run successfully for years.

Since then, we have:

• Tightened timeouts for deletion commands in the cleanup process.

• Changed to a more forgiving isolation level to prevent broad blocking locks.

• Adjusted the cleanup process to use smaller batch sizes and higher concurrency, which reduced database impact and increased throughput.

• Added significantly more monitoring to the cleanup process to better track query performance and catch anomalies early.

Next Steps

• Update MySQL statistics so the query optimizer selects the correct execution plan.

  • Expected completion: by end of this week (2025-10-24)

• Rework alerting strategy to detect low-rate but workflow-blocking errors earlier.

  • Expected completion: within 3 weeks (by 2025-11-06)

We experienced elevated API error rates on the encoding endpoints between 07:50 and 08:30 UTC. The issue, caused by the encoding service during muxing creation, was identified and promptly resolved. The engineering team is preparing a post-mortem and will share it here once available.

We have identified the root cause of the encoding failures: a misconfiguration in our S3 bucket.
The S3 configuration has been corrected, and encoding jobs on AWS are now recovering. We are seeing encoding tasks completing successfully again.
We are actively monitoring the system to confirm full recovery.

Root Cause Analysis: Encoding Failures on AWS

Summary

On September 4, 2025, between 06:31 AM and 08:10 AM CEST, encoding jobs running on AWS failed due to an issue accessing our S3 storage.

Root Cause

The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage.

Impact

• Only encoding jobs running on AWS were affected.

• Encodings on other cloud providers and all other Bitmovin services were not impacted.

Resolution

The configuration was corrected at 08:10 AM CEST, restoring access to S3. Encoding operations on AWS recovered immediately and have been stable since.

Preventive Measures

To prevent this from happening again, we are:

• Updating our key rotation procedure to ensure keys are not deleted prematurely.

• Automating the key rotation process to reduce the chance of operator error.