Root Cause Analysis — Encoding Configuration & Start Outage — 2026-07-07

Summary

On 07 July 2026, a non-backwards-compatible database migration was applied to a relational database schema at 11:40 UTC. The migration renamed several columns on a table. The currently running microservice responsible for the configuration of encoding resources still referenced the old column names in its entity mappings, so every read referencing the affected table began failing with a "Unknown column" error.

As a result, customers could not configure or start encodings. Affected API calls returned HTTP 500 responses. The first alert fired internally at 11:41 UTC, customer reports arrived shortly after. 

The root cause which was the non-backwards compatible database migration which has been applied was identified at 12:15 UTC, a migration reverting the previous change was applied at 12:31 UTC, and the incident was marked Resolved at 12:40 UTC.

Incident Timeline

Customer Impact

Customers could not configure and/or start encoding jobs. Affected requests returned an HTTP 500 Internal Server Error with error code 1004 — "An undefined error occurred". 

Root Cause

A migration renaming columns within a relational database table was applied to the production database.

An in-place CHANGE COLUMN rename was included in the database migration. The version of the service accessing the affected table running in production did still expect the old column names in its database entity mappings.

This bubbled up as an unhandled exception and was returned to callers as an HTTP 500 error. Given that the affected database serves as the primary repository for the encoding API, the resulting disruption was widespread, impacting configuration workflows for our entire customer base.

The database schema change and the corresponding service code change were not coordinated as a backwards-compatible, multi-step rollout. A schema-and-code change of this kind requires the database to remain compatible with the currently deployed service version at every step.

Next Steps and Preventive Actions

Detection & response

• Faster escalation — we've shortened the on-call escalation window so incidents reach the full response team within minutes of detection.

• Real-time error-rate alerting — we've reduced time to alert on API error rates to 2 minutes.

Prevention

• Formalized migration procedure — all database schema changes now follow a documented, mandatory playbook requiring backwards-compatible, multi-step rollouts, keeping the database compatible with the running service version at every step. This standard is applied uniformly across all engineering teams.

• Automated safeguards in CI and code review — we've added automated checks and compatibility tests that detect schema/service mismatches before a change can reach production.

The issue affecting Bitmovin Encoding has been resolved, and full functionality has been restored.

Our systems are stable, and we are no longer observing any errors or delays.

A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption.

We apologize for all customers that were affected by this outage, unfortunately failed encodings cannot be restarted on our side but need to be recreated in your workflow.

A fix has been implemented, and we are currently monitoring the system to ensure full recovery.

Early indicators are positive, but we are validating to confirm that services are fully restored.

We will provide a final resolution update shortly.

We have identified the root cause and our Engineering team is actively working on implementing a fix for the identified issue impacting encoding start calls.

We are making progress and will share an update as soon as validation is underway or the fix is rolled out.

Thank you for your continued understanding.

This incident is now resolved. As of 17:32 UTC, all buffered data has been successfully backfilled into our central Observability data store, and ingestion has been operating normally on the alternative transport protocol since the failover at 16:52 UTC.

There was no data loss. Records buffered during the connectivity issue are now fully available for querying.

The incident began at 16:26 UTC and was fully resolved at 17:32 UTC. We apologize for any inconvenience and will follow up with a post-incident review once we have performed a full root cause analysis.

We have identified the cause of the connectivity issue between our US datacenter and our central Observability data store and have failed over to an alternative transport protocol. Real-time ingestion of US analytics data has been fully restored.

We are now draining the backlog of buffered data into the database. Some recently buffered records may appear with a short delay until this process completes. No data loss is expected.

We will continue to monitor the system and will post a final update once all buffered data has been backfilled.