Root Cause Analysis: AWS Image Authorization Issues for Cloud Connect Customers

On August 11, AWS Cloud Connect customers temporarily lost access to the encoder AMIs, causing encodings to fail at instance startup. The cause was a recent change to our infrastructure management API that altered internal request routing. As a result, an AMI access synchronization step in our encoder release process received an empty customer list and revoked customer AMI permissions. A first, partially completed run affected a small number of customers starting at 14:06 UTC; a second run at 14:58 UTC revoked access for all remaining customers. Access was fully restored by 15:46 UTC. No data was lost.

Timeline (UTC)

• 14:06 - An encoder release process runs. Its AMI access synchronization step queries the list of Cloud Connect customers, receives an empty result, and begins revoking customer access to the encoder AMIs. The release process is restarted, canceling the run partway through. A small number of customers are affected at this point.

• 14:46 - The first customer reports about failing encodings on AWS come in. Since only a small number of customers are affected at this point, the initial investigation focuses on customer-specific causes.

• 14:58 - The restarted release process completes. The same synchronization step again receives an empty customer list and revokes AMI access for all remaining customers.

• ~15:28 - After the second synchronization run, internal monitoring alerts on failing encoding startups across all AWS Cloud Connect customers, making the wider scope of the issue visible.

• 15:30 - The revoked AMI permissions are identified as the cause and traced back to the release process.

• 15:34 - We trigger a re-run of the grant process with the correct customer list.

• 15:46 - The grant process completes. AMI access is restored and encodings start normally again. We continue monitoring to confirm stable operation.

Root cause

Our encoder release process includes a step that synchronizes customer access permissions to the encoder AMIs. It queries our infrastructure management API for the list of Cloud Connect customers and grants or revokes AMI access accordingly.

In the week before the incident, we made changes to the infrastructure management API to improve resiliency and redundancy. These changes also altered how requests are routed internally. When the release process ran on August 11, its customer list query was routed to an endpoint that returned an empty list with a successful response. Because the response was technically valid, no error was raised, and the synchronization step treated the empty list as the desired state and revoked customer AMI permissions. This happened in two runs: a first run that was canceled partway through when the release process was restarted, and a second run that completed and revoked access for all remaining customers.

Resolution

We re-ran the grant process with the correct customer list, which restored AMI access for all affected customers. Encodings started normally again as of 15:46 UTC, and we confirmed stable operation over the following hours.

Preventive measures

• We are adding safeguards to the AMI access synchronization so it fails instead of acting when the customer list is empty or shrinks beyond a defined threshold.

• We are removing ACL management from the encoder release process entirely. Customer AMI access is already managed by our infrastructure service, and the release process was an unintended second place where these permissions could be changed. Consolidating this into a single owner eliminates this class of failure.

• We are improving our detection for this failure mode so that permission changes of this kind are alerted on directly, rather than indirectly through failing encodings. This will let us identify the scope of similar issues faster, even when only few customers are initially affected.

We apologize for the disruption this caused. If you have questions about this incident, please reach out to our support team.

This incident has been resolved.
Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been operating normally since 15:46 UTC.
We will publish a detailed root cause analysis within 24 hours.
We apologize for any inconvenience caused.

This incident has been resolved. Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been running normally since the fix was deployed.

We have identified the root cause of the AWS image authorization issue affecting Cloud Connect customers and are currently rolling out a fix. We will provide a further update once the fix has been fully deployed.

RCA: Bitmovin Observability query and export outage (27 July 2026)

On 27 July 2026, Bitmovin Observability was unable to serve queries or data exports for approximately 20 minutes, with reduced data completeness for a further 24 minutes. Data ingestion was never interrupted, incoming data was buffered throughout, and no customer data was lost. This post explains what happened, what we have done, and what we are changing.

Impact

Between 12:33 and 12:53 UTC, the Bitmovin Observability query API, dashboards and data export were unavailable. Queries and exports resumed at 12:53. Data collected during the outage was buffered and then backfilled, completing at 13:17 UTC. Until that point, queries covering the affected period could return incomplete results. Data ingestion and collection were not affected at any time, and there was no data loss.

What happened

An internal query executed against our observability database cluster triggered a defect in the database engine. The memory used by this query was not correctly accounted for, which meant the cluster's memory safeguards never engaged. Instead of the query being limited or rejected, memory was consumed until nodes became unresponsive, and the failure propagated across the cluster rather than staying contained to the node serving the query.

Automated alerting notified our on-call engineers within 30 seconds. The cluster was restarted at 12:43 and completed recovery of all shards, resuming full write acceptance, at 12:53. Buffered data was then backfilled and fully available at 13:17.

Timeline (UTC)

What we are changing

• The query that triggered the defect was an ad hoc analytical query run by an operator, not a query our application issues in normal operation. We have instructed our engineering team not to run this class of query against the production cluster until the underlying defect is resolved.

• We are reporting the defect upstream and validating whether a fix is available in a newer version of the database engine.

Our ingestion buffering worked exactly as designed and protected customer data. We apologise for the disruption and for the incomplete data during the backfill window.

As of 15:17 CEST, the backfill has completed and all buffered data is available.
All systems are operational again. No data was lost.

We will provide a root cause analysis as soon as possible.

The database cluster has recovered. Querying (dashboards and API) and data export have resumed, and incoming data is being written normally again.

We are now triggering the backfill of the buffered data. Until it completes, recent data may be incomplete in queries and exports.

We are monitoring the results and will confirm once the backfill has finished.

Query operations are back to normal and we continue to monitor the situation. We apologize for any inconvenience caused due to this incident.

Root Cause Analysis — Encoding Configuration & Start Outage — 2026-07-07

Summary

On 07 July 2026, a non-backwards-compatible database migration was applied to a relational database schema at 11:40 UTC. The migration renamed several columns on a table. The currently running microservice responsible for the configuration of encoding resources still referenced the old column names in its entity mappings, so every read referencing the affected table began failing with a "Unknown column" error.

As a result, customers could not configure or start encodings. Affected API calls returned HTTP 500 responses. The first alert fired internally at 11:41 UTC, customer reports arrived shortly after. 

The root cause which was the non-backwards compatible database migration which has been applied was identified at 12:15 UTC, a migration reverting the previous change was applied at 12:31 UTC, and the incident was marked Resolved at 12:40 UTC.

Incident Timeline

Customer Impact

Customers could not configure and/or start encoding jobs. Affected requests returned an HTTP 500 Internal Server Error with error code 1004 — "An undefined error occurred". 

Root Cause

A migration renaming columns within a relational database table was applied to the production database.

An in-place CHANGE COLUMN rename was included in the database migration. The version of the service accessing the affected table running in production did still expect the old column names in its database entity mappings.

This bubbled up as an unhandled exception and was returned to callers as an HTTP 500 error. Given that the affected database serves as the primary repository for the encoding API, the resulting disruption was widespread, impacting configuration workflows for our entire customer base.

The database schema change and the corresponding service code change were not coordinated as a backwards-compatible, multi-step rollout. A schema-and-code change of this kind requires the database to remain compatible with the currently deployed service version at every step.

Next Steps and Preventive Actions

Detection & response

• Faster escalation — we've shortened the on-call escalation window so incidents reach the full response team within minutes of detection.

• Real-time error-rate alerting — we've reduced time to alert on API error rates to 2 minutes.

Prevention

• Formalized migration procedure — all database schema changes now follow a documented, mandatory playbook requiring backwards-compatible, multi-step rollouts, keeping the database compatible with the running service version at every step. This standard is applied uniformly across all engineering teams.

• Automated safeguards in CI and code review — we've added automated checks and compatibility tests that detect schema/service mismatches before a change can reach production.

The issue affecting Bitmovin Encoding has been resolved, and full functionality has been restored.

Our systems are stable, and we are no longer observing any errors or delays.

A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption.

We apologize for all customers that were affected by this outage, unfortunately failed encodings cannot be restarted on our side but need to be recreated in your workflow.

A fix has been implemented, and we are currently monitoring the system to ensure full recovery.

Early indicators are positive, but we are validating to confirm that services are fully restored.

We will provide a final resolution update shortly.

We have identified the root cause and our Engineering team is actively working on implementing a fix for the identified issue impacting encoding start calls.

We are making progress and will share an update as soon as validation is underway or the fix is rolled out.

Thank you for your continued understanding.

This incident is now resolved. As of 17:32 UTC, all buffered data has been successfully backfilled into our central Observability data store, and ingestion has been operating normally on the alternative transport protocol since the failover at 16:52 UTC.

There was no data loss. Records buffered during the connectivity issue are now fully available for querying.

The incident began at 16:26 UTC and was fully resolved at 17:32 UTC. We apologize for any inconvenience and will follow up with a post-incident review once we have performed a full root cause analysis.

We have identified the cause of the connectivity issue between our US datacenter and our central Observability data store and have failed over to an alternative transport protocol. Real-time ingestion of US analytics data has been fully restored.

We are now draining the backlog of buffered data into the database. Some recently buffered records may appear with a short delay until this process completes. No data loss is expected.

We will continue to monitor the system and will post a final update once all buffered data has been backfilled.