Bitmovin Status - Incident history https://status.bitmovin.com Bitmovin Tue, 7 Jul 2026 11:45:00 +0000 Bitmovin Encodings cannot be started Type: Incident Duration: 55 minutes Affected Components: Encoding API Jul 7, 11:45:00 GMT+0 - Investigating - We are currently investigating a potential service disruption impacting encoding start calls. Our team is actively reviewing alerts and customer reports. We’ll provide an update within 30 minutes or as soon as we have more information. Thank you for your patience. Jul 7, 12:25:37 GMT+0 - Identified - We have identified the root cause and our Engineering team is actively working on implementing a fix for the identified issue impacting encoding start calls. We are making progress and will share an update as soon as validation is underway or the fix is rolled out. Thank you for your continued understanding. Jul 7, 12:37:15 GMT+0 - Monitoring - A fix has been implemented, and we are currently monitoring the system to ensure full recovery. Early indicators are positive, but we are validating to confirm that services are fully restored. We will provide a final resolution update shortly. Jul 7, 12:39:32 GMT+0 - Resolved - The issue affecting Bitmovin Encoding has been resolved, and full functionality has been restored. Our systems are stable, and we are no longer observing any errors or delays. A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption. We apologize for all customers that were affected by this outage, unfortunately failed encodings cannot be restarted on our side but need to be recreated in your workflow. Jul 9, 08:59:44 GMT+0 - Postmortem - # Root Cause Analysis — Encoding Configuration & Start Outage — 2026-07-07 # Summary On 07 July 2026, a non-backwards-compatible database migration was applied to a relational database schema at **11:40 UTC**. The migration renamed several columns on a table. The currently running microservice responsible for the configuration of encoding resources still referenced the old column names in its entity mappings, so every read referencing the affected table began failing with a "Unknown column" error. As a result, customers could not configure or start encodings. Affected API calls returned **HTTP 500** responses. The first alert fired internally at **11:41 UTC**, customer reports arrived shortly after. The root cause which was the non-backwards compatible database migration which has been applied was identified at 12:15 UTC, a migration reverting the previous change was applied at 12:31 UTC, and the incident was marked **Resolved at 12:40 UTC**. # Incident Timeline | **Start time** | 11:40:51 UTC (breaking DB migration applied) | | -------------------------------------------------------------- | ----------------------------------------------- | | **First internal alert** | 11:41 UTC | | **Escalation** | 11:51 UTC | | **First customer reports and start of incident Investigation** | 11:58 UTC | | **Root Cause Identified** | 12:15 UTC | | **Mitigation Applied** | 12:31 UTC (migration reverting changes applied) | | **Resolution Time (Statuspage)** | 12:40 UTC | | **Total Duration** | \~ 50 minutes | | **Detection Method** | Internal monitoring/alerts + customer reports | # Customer Impact Customers could not configure and/or start encoding jobs. Affected requests returned an **HTTP 500 Internal Server Error** with error code 1004 — "An undefined error occurred". # Root Cause A migration renaming columns within a relational database table was applied to the production database. An in-place CHANGE COLUMN rename was included in the database migration. The version of the service accessing the affected table running in production did still expect the old column names in its database entity mappings. This bubbled up as an unhandled exception and was returned to callers as an HTTP 500 error. Given that the affected database serves as the primary repository for the encoding API, the resulting disruption was widespread, impacting configuration workflows for our entire customer base. The database schema change and the corresponding service code change were not coordinated as a backwards-compatible, multi-step rollout. A schema-and-code change of this kind requires the database to remain compatible with the currently deployed service version at every step. # Next Steps and Preventive Actions _Detection & response_ * **Faster escalation** — we've shortened the on-call escalation window so incidents reach the full response team within minutes of detection. * **Real-time error-rate alerting** — we've reduced time to alert on API error rates to 2 minutes. _Prevention_ * **Formalized migration procedure** — all database schema changes now follow a documented, mandatory playbook requiring backwards-compatible, multi-step rollouts, keeping the database compatible with the running service version at every step. This standard is applied uniformly across all engineering teams. * **Automated safeguards in CI and code review** — we've added automated checks and compatibility tests that detect schema/service mismatches before a change can reach production. Type: Incident

Duration: 55 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>11:45:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating a potential service disruption impacting encoding start calls. Our team is actively reviewing alerts and customer reports. We’ll provide an update within 30 minutes or as soon as we have more information. Thank you for your patience..</p> <p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>12:25:37</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause and our Engineering team is actively working on implementing a fix for the identified issue impacting encoding start calls. We are making progress and will share an update as soon as validation is underway or the fix is rolled out. Thank you for your continued understanding..</p> <p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>12:37:15</var> GMT+0</small><br><strong>Monitoring</strong> - A fix has been implemented, and we are currently monitoring the system to ensure full recovery. Early indicators are positive, but we are validating to confirm that services are fully restored. We will provide a final resolution update shortly..</p> <p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>12:39:32</var> GMT+0</small><br><strong>Resolved</strong> - The issue affecting Bitmovin Encoding has been resolved, and full functionality has been restored. Our systems are stable, and we are no longer observing any errors or delays. A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption. We apologize for all customers that were affected by this outage, unfortunately failed encodings cannot be restarted on our side but need to be recreated in your workflow..</p> <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>08:59:44</var> GMT+0</small><br><strong>Postmortem</strong> - # Root Cause Analysis — Encoding Configuration & Start Outage — 2026-07-07 # Summary On 07 July 2026, a non-backwards-compatible database migration was applied to a relational database schema at **11:40 UTC**. The migration renamed several columns on a table. The currently running microservice responsible for the configuration of encoding resources still referenced the old column names in its entity mappings, so every read referencing the affected table began failing with a "Unknown column" error. As a result, customers could not configure or start encodings. Affected API calls returned **HTTP 500** responses. The first alert fired internally at **11:41 UTC**, customer reports arrived shortly after. The root cause which was the non-backwards compatible database migration which has been applied was identified at 12:15 UTC, a migration reverting the previous change was applied at 12:31 UTC, and the incident was marked **Resolved at 12:40 UTC**. # Incident Timeline | **Start time** | 11:40:51 UTC (breaking DB migration applied) | | -------------------------------------------------------------- | ----------------------------------------------- | | **First internal alert** | 11:41 UTC | | **Escalation** | 11:51 UTC | | **First customer reports and start of incident Investigation** | 11:58 UTC | | **Root Cause Identified** | 12:15 UTC | | **Mitigation Applied** | 12:31 UTC (migration reverting changes applied) | | **Resolution Time (Statuspage)** | 12:40 UTC | | **Total Duration** | \~ 50 minutes | | **Detection Method** | Internal monitoring/alerts + customer reports | # Customer Impact Customers could not configure and/or start encoding jobs. Affected requests returned an **HTTP 500 Internal Server Error** with error code 1004 — "An undefined error occurred". # Root Cause A migration renaming columns within a relational database table was applied to the production database. An in-place CHANGE COLUMN rename was included in the database migration. The version of the service accessing the affected table running in production did still expect the old column names in its database entity mappings. This bubbled up as an unhandled exception and was returned to callers as an HTTP 500 error. Given that the affected database serves as the primary repository for the encoding API, the resulting disruption was widespread, impacting configuration workflows for our entire customer base. The database schema change and the corresponding service code change were not coordinated as a backwards-compatible, multi-step rollout. A schema-and-code change of this kind requires the database to remain compatible with the currently deployed service version at every step. # Next Steps and Preventive Actions _Detection & response_ * **Faster escalation** — we've shortened the on-call escalation window so incidents reach the full response team within minutes of detection. * **Real-time error-rate alerting** — we've reduced time to alert on API error rates to 2 minutes. _Prevention_ * **Formalized migration procedure** — all database schema changes now follow a documented, mandatory playbook requiring backwards-compatible, multi-step rollouts, keeping the database compatible with the running service version at every step. This standard is applied uniformly across all engineering teams. * **Automated safeguards in CI and code review** — we've added automated checks and compatibility tests that detect schema/service mismatches before a change can reach production..</p> ]]> Tue, 7 Jul 2026 11:45:00 +0000 https://status.bitmovin.com/incident/cmrametut07bi1ansiul2hkrj https://status.bitmovin.com/incident/cmrametut07bi1ansiul2hkrj Delayed Observability Data Ingestion (US Region) Type: Incident Duration: 58 minutes Affected Components: Data Ingress, Query Service Jun 7, 16:37:55 GMT+0 - Investigating - We are currently investigating connectivity issues between our US datacenter and our central Observability data store happening since 16:26 UTC. As a result, approximately 20% of analytics requests originating in the US are not being ingested in real time. Affected data is being buffered and will be inserted into the database once connectivity is restored, so no data is being lost. Querying of previously ingested data is unaffected. We are working with our cloud providers to identify the root cause and will provide an update as soon as we have more information. Jun 7, 16:59:10 GMT+0 - Identified - We have identified the cause of the connectivity issue between our US datacenter and our central Observability data store and have failed over to an alternative transport protocol. Real-time ingestion of US analytics data has been fully restored. We are now draining the backlog of buffered data into the database. Some recently buffered records may appear with a short delay until this process completes. No data loss is expected. We will continue to monitor the system and will post a final update once all buffered data has been backfilled. Jun 7, 17:35:52 GMT+0 - Resolved - This incident is now resolved. As of 17:32 UTC, all buffered data has been successfully backfilled into our central Observability data store, and ingestion has been operating normally on the alternative transport protocol since the failover at 16:52 UTC. There was no data loss. Records buffered during the connectivity issue are now fully available for querying. The incident began at 16:26 UTC and was fully resolved at 17:32 UTC. We apologize for any inconvenience and will follow up with a post-incident review once we have performed a full root cause analysis. Type: Incident

Duration: 58 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>16:37:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating connectivity issues between our US datacenter and our central Observability data store happening since 16:26 UTC. As a result, approximately 20% of analytics requests originating in the US are not being ingested in real time. Affected data is being buffered and will be inserted into the database once connectivity is restored, so no data is being lost. Querying of previously ingested data is unaffected. We are working with our cloud providers to identify the root cause and will provide an update as soon as we have more information..</p> <p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>16:59:10</var> GMT+0</small><br><strong>Identified</strong> - We have identified the cause of the connectivity issue between our US datacenter and our central Observability data store and have failed over to an alternative transport protocol. Real-time ingestion of US analytics data has been fully restored. We are now draining the backlog of buffered data into the database. Some recently buffered records may appear with a short delay until this process completes. No data loss is expected. We will continue to monitor the system and will post a final update once all buffered data has been backfilled..</p> <p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>17:35:52</var> GMT+0</small><br><strong>Resolved</strong> - This incident is now resolved. As of 17:32 UTC, all buffered data has been successfully backfilled into our central Observability data store, and ingestion has been operating normally on the alternative transport protocol since the failover at 16:52 UTC. There was no data loss. Records buffered during the connectivity issue are now fully available for querying. The incident began at 16:26 UTC and was fully resolved at 17:32 UTC. We apologize for any inconvenience and will follow up with a post-incident review once we have performed a full root cause analysis..</p> ]]> Sun, 7 Jun 2026 16:37:55 +0000 https://status.bitmovin.com/incident/cmq409v1000kbp8r0qm5a51bs https://status.bitmovin.com/incident/cmq409v1000kbp8r0qm5a51bs Observability API — Historic Data Queries Returning Empty Results Type: Incident Duration: 1 day, 4 hours and 52 minutes Affected Components: Query Service, Export Service Apr 15, 11:38:55 GMT+0 - Identified - We are now performing the cut-over for the 90 day retention data store. We expect this to take approximately 30-45 minutes. There may be brief inconsistencies in high-granularity data during the process. We will post a final update once the operation is complete and all data is fully restored. Apr 15, 12:23:44 GMT+0 - Resolved - The data recovery is now fully complete. All data across both 30 day and 90 day retention data stores is fully available again in the Observability Dashboard and via the API. We will perform a full root cause analysis and publish it by tomorrow (April 16, 2026) at the latest. We apologize for any inconvenience caused and thank you for your patience. Apr 14, 15:24:45 GMT+0 - Identified - We are currently working on mitigating the issue affecting historic data in Bitmovin Observability. The underlying bug in our database software has been identified and fixed. However, the corrupted metadata is already present in the system, which requires us to recover data from backup. No data has been lost. As part of the recovery process, we will need to perform a short table switch during which high granularity data may contain a small gap for a few minutes. Once the data recovery is complete, all historic data will be fully available and everything will be back to normal. We will post an update once the recovery operation is complete. Apr 14, 18:16:05 GMT+0 - Identified - We are now performing a cut-over for the 30 day retention data store to the recovered data. Historic data will be available again within the next 10 minutes at 2026-04-14 18:25 UTC. During the cut-over time you might see short gaps in the data while we reconcile the recovered data set with the live data. We will post a final update once the operation is complete and 30 day retention data is fully restored. Apr 15, 15:28:34 GMT+0 - Postmortem - ### Summary On April 14, 2026, starting at approximately 06:00 UTC, historic data in Bitmovin Observability became partially unavailable. API queries for timeframes before 06:00 UTC returned empty or incomplete results, and minute-level granularity in the Dashboard was unavailable for affected timeframes. Hourly granularity for historic data in the Dashboard remained unaffected. API queries for data after 06:00 UTC were fully operational throughout the incident. ### Root Cause On April 13, 2026, we upgraded our database software to a new version. This version had been running successfully in our QA environment for over one month prior to the production rollout. On April 14, our daily retention job, which removes data that has exceeded its retention period, executed normally but triggered a pre-existing bug in the new database version. This bug caused a metadata corruption that led the database to misidentify the storage location of existing data. As a result, queries against historic data returned empty results even though the underlying data was still intact on disk. The bug was specific to tables that had been created on an older version of the database and subsequently upgraded, which is why it did not surface in our QA environment. We worked together with our database vendor to identify the root cause and determine the best recovery path. The vendor has confirmed the issue and developed a fix, which we expect to be released today (April 15, 2026). ### Impact * API queries for timeframes before 06:00 UTC on April 14, 2026 returned empty or incomplete results. * Minute-level granularity in the Observability Dashboard was unavailable for affected timeframes. * Session details and unique viewers metrics were unavailable for affected timeframes. * Data exports for timeframes before 06:00 UTC were also affected. * During the final recovery steps, brief data inconsistencies were visible in the Dashboard while recovered data was being reconciled with live data. * No data was lost. All real-time monitoring, alerting, and data ingestion remained fully operational throughout the incident. Hourly granularity for historic data in the Dashboard was unaffected. All functionality related to data after 06:00 UTC, including API queries, exports, and Dashboard views at all granularities, was fully operational. The impact was limited to minute-level granularity, session details, unique viewers, and exports for timeframes before 06:00 UTC. ### Timeline (all times UTC) | Time | Event | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | April 13 | Database software upgraded to new version | | April 14, 06:00 | Daily retention job runs, triggering the metadata corruption | | April 14, 06:35 | Routine data integrity check triggers an alarm, investigation begins | | April 14, 07:08 | Issue escalated to database vendor support | | April 14, 07:18 | Database vendor joins investigation call | | April 14, 09:20 | Retention job identified as the trigger for the issue | | April 14, 10:00 | Issue successfully reproduced | | April 14, 12:19 | Root cause identified and bug fixed. Analysis concludes that recovery via code change alone is not feasible. Recovery plan using backups and secondary data stores initiated | | April 14, 13:00 | Data recovery begins for the 30 day retention data store | | April 14, 19:10 | 30 day retention data store recovery completed. 90 day retention data store recovery begins | | April 15, 12:20 | 90 day retention data store recovery completed. Incident resolved | ### Mitigation & Recovery Once the root cause was identified, we worked with our database vendor who developed a fix to prevent future occurrences. Since the corrupted metadata was already present in the system, recovery via a code change alone was not feasible. We performed a full data recovery using backups and secondary data stores. Recovery was carried out in two phases: first for the 30 day retention data store, then for the 90 day retention data store. Both were completed successfully with all historic data fully restored. ### Preventive Measures * The database vendor has developed a fix that prevents the metadata corruption from occurring during partition deletions. We expect the fix to be released today (April 15, 2026). * We will apply the fix to our production systems once it is available. * We will ensure our QA environment includes tables that mirror production conditions, including tables originally created on older database versions and subsequently upgraded, to catch similar edge cases before production rollouts. Apr 14, 19:11:24 GMT+0 - Identified - We have completed the data recovery for the 30 day retention data store. All historic data for affected customers on this retention tier is now fully available again. We will now continue with recovery for the 90 day retention data store. We expect this recovery to be completed by tomorrow morning (April 15, 2026). We will post a final update once all data has been fully restored. Apr 14, 07:31:15 GMT+0 - Investigating - We have identified an issue affecting the Observability platform where historic data is currently not being served correctly. API queries targeting timeframes prior to today may return empty or incomplete results. Additionally, session details and unique viewers metrics in the Observability Dashboard are impacted for historic timeframes. Real-time data and all data for today remain fully operational and unaffected, both in the Observability Dashboard and via the API. Our engineering team is actively investigating the root cause and working on a resolution. We will provide updates as more information becomes available. Apr 14, 09:00:01 GMT+0 - Identified - We are continuing to investigate the issue affecting historic data on the Observability platform. We can now confirm that all data prior to 06:00 UTC today (April 14, 2026) is affected. Currently impacted: * API queries for timeframes before 06:00 UTC today are returning empty or incomplete results. * Zooming into minute granularity in the Observability Dashboard for timeframes before 06:00 UTC today will not display data. * Session details and unique viewers metrics in the Dashboard are unavailable for affected timeframes. Viewing data at hourly granularity in the Dashboard is working normally. All data from 06:00 UTC today onwards remains fully operational and unaffected, both in the Dashboard and via the API. We are actively investigating the root cause and working on mitigation. We will continue to provide updates as we make progress. Apr 14, 11:04:05 GMT+0 - Identified - We have identified the root cause of the issue affecting historic data in our Observability product. A bug in our database software caused a metadata mismatch, making historic data in our high granularity data store unavailable for timeframes prior to 06:00 UTC today (April 14, 2026). We have successfully reproduced the issue and our database vendor is currently working on a hotfix to resolve it. We will provide a further update once we have a timeline for mitigation. Type: Incident

Duration: 1 day, 4 hours and 52 minutes

Affected Components: ,

<p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>11:38:55</var> GMT+0</small><br><strong>Identified</strong> - We are now performing the cut-over for the 90 day retention data store. We expect this to take approximately 30-45 minutes. There may be brief inconsistencies in high-granularity data during the process. We will post a final update once the operation is complete and all data is fully restored..</p> <p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>12:23:44</var> GMT+0</small><br><strong>Resolved</strong> - The data recovery is now fully complete. All data across both 30 day and 90 day retention data stores is fully available again in the Observability Dashboard and via the API. We will perform a full root cause analysis and publish it by tomorrow (April 16, 2026) at the latest. We apologize for any inconvenience caused and thank you for your patience..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>15:24:45</var> GMT+0</small><br><strong>Identified</strong> - We are currently working on mitigating the issue affecting historic data in Bitmovin Observability. The underlying bug in our database software has been identified and fixed. However, the corrupted metadata is already present in the system, which requires us to recover data from backup. No data has been lost. As part of the recovery process, we will need to perform a short table switch during which high granularity data may contain a small gap for a few minutes. Once the data recovery is complete, all historic data will be fully available and everything will be back to normal. We will post an update once the recovery operation is complete..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>18:16:05</var> GMT+0</small><br><strong>Identified</strong> - We are now performing a cut-over for the 30 day retention data store to the recovered data. Historic data will be available again within the next 10 minutes at 2026-04-14 18:25 UTC. During the cut-over time you might see short gaps in the data while we reconcile the recovered data set with the live data. We will post a final update once the operation is complete and 30 day retention data is fully restored..</p> <p><small>Apr <var data-var='date'> 15</var>, <var data-var='time'>15:28:34</var> GMT+0</small><br><strong>Postmortem</strong> - ### Summary On April 14, 2026, starting at approximately 06:00 UTC, historic data in Bitmovin Observability became partially unavailable. API queries for timeframes before 06:00 UTC returned empty or incomplete results, and minute-level granularity in the Dashboard was unavailable for affected timeframes. Hourly granularity for historic data in the Dashboard remained unaffected. API queries for data after 06:00 UTC were fully operational throughout the incident. ### Root Cause On April 13, 2026, we upgraded our database software to a new version. This version had been running successfully in our QA environment for over one month prior to the production rollout. On April 14, our daily retention job, which removes data that has exceeded its retention period, executed normally but triggered a pre-existing bug in the new database version. This bug caused a metadata corruption that led the database to misidentify the storage location of existing data. As a result, queries against historic data returned empty results even though the underlying data was still intact on disk. The bug was specific to tables that had been created on an older version of the database and subsequently upgraded, which is why it did not surface in our QA environment. We worked together with our database vendor to identify the root cause and determine the best recovery path. The vendor has confirmed the issue and developed a fix, which we expect to be released today (April 15, 2026). ### Impact * API queries for timeframes before 06:00 UTC on April 14, 2026 returned empty or incomplete results. * Minute-level granularity in the Observability Dashboard was unavailable for affected timeframes. * Session details and unique viewers metrics were unavailable for affected timeframes. * Data exports for timeframes before 06:00 UTC were also affected. * During the final recovery steps, brief data inconsistencies were visible in the Dashboard while recovered data was being reconciled with live data. * No data was lost. All real-time monitoring, alerting, and data ingestion remained fully operational throughout the incident. Hourly granularity for historic data in the Dashboard was unaffected. All functionality related to data after 06:00 UTC, including API queries, exports, and Dashboard views at all granularities, was fully operational. The impact was limited to minute-level granularity, session details, unique viewers, and exports for timeframes before 06:00 UTC. ### Timeline (all times UTC) | Time | Event | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | April 13 | Database software upgraded to new version | | April 14, 06:00 | Daily retention job runs, triggering the metadata corruption | | April 14, 06:35 | Routine data integrity check triggers an alarm, investigation begins | | April 14, 07:08 | Issue escalated to database vendor support | | April 14, 07:18 | Database vendor joins investigation call | | April 14, 09:20 | Retention job identified as the trigger for the issue | | April 14, 10:00 | Issue successfully reproduced | | April 14, 12:19 | Root cause identified and bug fixed. Analysis concludes that recovery via code change alone is not feasible. Recovery plan using backups and secondary data stores initiated | | April 14, 13:00 | Data recovery begins for the 30 day retention data store | | April 14, 19:10 | 30 day retention data store recovery completed. 90 day retention data store recovery begins | | April 15, 12:20 | 90 day retention data store recovery completed. Incident resolved | ### Mitigation & Recovery Once the root cause was identified, we worked with our database vendor who developed a fix to prevent future occurrences. Since the corrupted metadata was already present in the system, recovery via a code change alone was not feasible. We performed a full data recovery using backups and secondary data stores. Recovery was carried out in two phases: first for the 30 day retention data store, then for the 90 day retention data store. Both were completed successfully with all historic data fully restored. ### Preventive Measures * The database vendor has developed a fix that prevents the metadata corruption from occurring during partition deletions. We expect the fix to be released today (April 15, 2026). * We will apply the fix to our production systems once it is available. * We will ensure our QA environment includes tables that mirror production conditions, including tables originally created on older database versions and subsequently upgraded, to catch similar edge cases before production rollouts..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>19:11:24</var> GMT+0</small><br><strong>Identified</strong> - We have completed the data recovery for the 30 day retention data store. All historic data for affected customers on this retention tier is now fully available again. We will now continue with recovery for the 90 day retention data store. We expect this recovery to be completed by tomorrow morning (April 15, 2026). We will post a final update once all data has been fully restored..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>07:31:15</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue affecting the Observability platform where historic data is currently not being served correctly. API queries targeting timeframes prior to today may return empty or incomplete results. Additionally, session details and unique viewers metrics in the Observability Dashboard are impacted for historic timeframes. Real-time data and all data for today remain fully operational and unaffected, both in the Observability Dashboard and via the API. Our engineering team is actively investigating the root cause and working on a resolution. We will provide updates as more information becomes available..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>09:00:01</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to investigate the issue affecting historic data on the Observability platform. We can now confirm that all data prior to 06:00 UTC today (April 14, 2026) is affected. Currently impacted: * API queries for timeframes before 06:00 UTC today are returning empty or incomplete results. * Zooming into minute granularity in the Observability Dashboard for timeframes before 06:00 UTC today will not display data. * Session details and unique viewers metrics in the Dashboard are unavailable for affected timeframes. Viewing data at hourly granularity in the Dashboard is working normally. All data from 06:00 UTC today onwards remains fully operational and unaffected, both in the Dashboard and via the API. We are actively investigating the root cause and working on mitigation. We will continue to provide updates as we make progress..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>11:04:05</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause of the issue affecting historic data in our Observability product. A bug in our database software caused a metadata mismatch, making historic data in our high granularity data store unavailable for timeframes prior to 06:00 UTC today (April 14, 2026). We have successfully reproduced the issue and our database vendor is currently working on a hotfix to resolve it. We will provide a further update once we have a timeline for mitigation..</p> ]]> Tue, 14 Apr 2026 07:31:15 +0000 https://status.bitmovin.com/incident/cmnyayu8s01jxqh1a4ah0hegf https://status.bitmovin.com/incident/cmnyayu8s01jxqh1a4ah0hegf Encoding Job Processing in AWS eu-west-1 (Ireland) Type: Incident Duration: 2 hours and 9 minutes Jan 28, 17:15:00 GMT+0 - Identified - Jobs started in AWS eu-west-1 (Ireland) may be affected by an upstream networking issue on AWS EC2. Jan 28, 17:48:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jan 28, 19:56:54 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 hours and 9 minutes

<p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>17:15:00</var> GMT+0</small><br><strong>Identified</strong> - Jobs started in AWS eu-west-1 (Ireland) may be affected by an upstream networking issue on AWS EC2..</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>17:48:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>19:56:54</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 28 Jan 2026 17:15:00 +0000 https://status.bitmovin.com/incident/cmkybiou403osah4d8a4gzskw https://status.bitmovin.com/incident/cmkybiou403osah4d8a4gzskw Scheduled Database Maintenance Type: Maintenance Duration: 5 minutes Affected Components: Dashboard, Query Service, Export Service, Alerting Service, , Encoding API, , Amazon Web Services, Oracle Cloud Infrastructure, Microsoft Azure, Akamai Cloud, Google Cloud, Encoding Scheduler, , Statistics Service, Account Service, Billing Service, Notification Service, Encoding → Cloud Provisioning → Bitmovin Dashboard → Dec 9, 09:00:00 GMT+0 - Identified - Our database will undergo scheduled maintenance on Tuesday, 2025-12-09, between 09:00 AM and 10:00 AM (UTC) for routine security updates. **Expected Impact: None**. Thanks to our improved infrastructure, we don't anticipate any service disruption. However, we're notifying you as a precaution. Your encodings and workflows should continue without interruption. Thank you for your understanding. Dec 9, 09:00:01 GMT+0 - Identified - Maintenance is now in progress Dec 9, 09:05:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 5 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our database will undergo scheduled maintenance on Tuesday, 2025-12-09, between 09:00 AM and 10:00 AM (UTC) for routine security updates. **Expected Impact: None**. Thanks to our improved infrastructure, we don't anticipate any service disruption. However, we're notifying you as a precaution. Your encodings and workflows should continue without interruption. Thank you for your understanding..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>09:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>09:05:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Tue, 9 Dec 2025 09:00:00 +0000 https://status.bitmovin.com/maintenance/cmiov56zv001a1m4njhuuupno https://status.bitmovin.com/maintenance/cmiov56zv001a1m4njhuuupno Increased Error Rates on Encoding API Type: Incident Duration: 5 hours and 25 minutes Affected Components: , Encoding API, , Amazon Web Services, Oracle Cloud Infrastructure, Microsoft Azure, Akamai Cloud, Google Cloud, Encoding Scheduler, Notification Service, Encoding → Cloud Provisioning → Nov 23, 21:30:18 GMT+0 - Identified - We have identified the root cause of the issue and implemented a fix, encoding backlog processing has sped up but will take more time to fully recover the overall system state. Nov 23, 22:02:10 GMT+0 - Identified - Our internal system state has recovered and we are slowly increasing parallel processing throughput for the queued encoding backlog. Nov 23, 22:24:37 GMT+0 - Monitoring - We are increasing parallel encoding slots every 10 minutes and the queued encoding backlog built up in during the incident will be finished soon. We encourage our customers to restart potentially stopped workflows again. Nov 23, 22:31:53 GMT+0 - Resolved - Our systems are stable, and we are no longer observing any errors or delays. A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption. Nov 23, 17:07:07 GMT+0 - Investigating - We are currently investigating this incident. Nov 23, 17:39:54 GMT+0 - Identified - We are currently experiencing a **partial outage** of the Encoding API. Error rates remain elevated and queued encodings continue to accumulate. To stabilise the platform, we have temporarily reduced maximum processing concurrency. While this measure helps improve system stability, it also impacts throughput and leads to longer wait times for queued jobs. Our engineering team is actively working on restoring normal performance. We will provide further updates as we make progress. Nov 23, 18:18:50 GMT+0 - Identified - We have successfully reduced the Encoding API error rate; however, our services are still struggling to process the backlog of queued service messages. To help stabilise the system, we have further lowered the VOD processing limits and continue to actively work on clearing the queue and restoring full service performance. Webhook notifications are also lagging behind. Further updates will follow as we make progress. Nov 25, 10:45:28 GMT+0 - Postmortem - ## **Root Cause Analysis – Encoding Service Outage 23 Nov 2025** **Summary** On 23 Nov 2025, the Bitmovin Encoding Platform experienced a service outage affecting our VoD and Live encoding pipeline. The issue resulted in temporary unavailability of encoding operations including starting and stopping of encodings and unavailability of status updates of running encodings for the time of the outage. ## **Root Cause** A **bug in one of our encoding services** caused memory usage and database (DB) data transfer volumes to increase slowly but steadily over the course of approximately one month. This resulted in: * Gradually **escalating DB read traffic** from the encoding services due to increasing object size with time * Overwhelming the service with **growing message queues** for this specific service due to decreased throughput * Data transfer reaching **\~900 MB/s outbound** from the database to the service instances As memory usage kept rising, multiple encoding service instances ultimately **crashed simultaneously**, leading to an abrupt stop in processing and further service degradation due to accumulation of **\~300,000 unprocessed service messages during the outage from 18:00 CET**. ## **Impact** * Encoding jobs failed to start after queueing * Running encoding jobs failed to update their status to “In Progress” or “Finished” during the incident ## **Detection** * 18:06 CET – Investigation initiated based on internal monitoring and alerts. * 18:07 CET – Outage confirmed and posted on status page. Internally, teams observed service pods **crashing due to memory limit exhaustion**, causing message processing interruptions and queue growth. ## **Investigation, Mitigation & Recovery** After the service crashes, we initiated a controlled process: 1. Increased service memory limits on our kubernetes cluster 18:25 CET 2. Reduced parallel message queue worker logic of the affected service to reduce DB read load by about 30% 19:02 CET 3. Implemented further monitoring and tracing in the service to increase visibility into the massive DB data streaming causes 19:38 CET 4. Re-routed several messages on our messaging system to further decrease DB read load 19:51 CET 5. Sped up processing of messages causing excessive read operations through service based skipping with a new service version 20:23 CET 6. Deployed new service which decreased DB read load to normal levels 20:56 7. Increased message throughput for services to restore the system state by processing the message backlog of about 350k messages in RabbitMQ 21:30 CET 8. Restored all service configuration and gradually increased encoding throughput 23:00 CET 9. Full service restored 23:31 CET ## **Next Steps and Preventive Actions** To prevent recurrence, we are implementing the following: * Implement a holistic fix for the bug causing excessive DB read load * Expand message queue and memory usage monitoring with stricter alerts for read amplification * Comprehensive review of encoding service code paths related to heavy DB reads * Message queue pressure safeguards including throttling for struggling services Nov 23, 20:33:10 GMT+0 - Investigating - Our team has likely identified the root cause of high DB load causing delays in encoding status updates in our API and dashboard. We are deploying a new version of the affected service and will have an update in a few minutes. Nov 23, 19:24:41 GMT+0 - Investigating - Our team is continuing to investigate the issue impacting slow API requests, delayed encoding status updates and delayed manifest generations (manifests generated with the encoding start calls). Encoding output is not affected. At this time, we have not yet identified the root cause. Diagnostic efforts are ongoing, and all necessary teams are engaged. We sincerely apologize for the inconvenience and will provide updates as soon as we have more information. Thank you for your patience and understanding. Type: Incident

Duration: 5 hours and 25 minutes

Affected Components: , , , , , , , , ,

<p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>21:30:18</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause of the issue and implemented a fix, encoding backlog processing has sped up but will take more time to fully recover the overall system state..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>22:02:10</var> GMT+0</small><br><strong>Identified</strong> - Our internal system state has recovered and we are slowly increasing parallel processing throughput for the queued encoding backlog..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>22:24:37</var> GMT+0</small><br><strong>Monitoring</strong> - We are increasing parallel encoding slots every 10 minutes and the queued encoding backlog built up in during the incident will be finished soon. We encourage our customers to restart potentially stopped workflows again..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>22:31:53</var> GMT+0</small><br><strong>Resolved</strong> - Our systems are stable, and we are no longer observing any errors or delays. A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>17:07:07</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>17:39:54</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a **partial outage** of the Encoding API. Error rates remain elevated and queued encodings continue to accumulate. To stabilise the platform, we have temporarily reduced maximum processing concurrency. While this measure helps improve system stability, it also impacts throughput and leads to longer wait times for queued jobs. Our engineering team is actively working on restoring normal performance. We will provide further updates as we make progress..</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>18:18:50</var> GMT+0</small><br><strong>Identified</strong> - We have successfully reduced the Encoding API error rate; however, our services are still struggling to process the backlog of queued service messages. To help stabilise the system, we have further lowered the VOD processing limits and continue to actively work on clearing the queue and restoring full service performance. Webhook notifications are also lagging behind. Further updates will follow as we make progress..</p> <p><small>Nov <var data-var='date'> 25</var>, <var data-var='time'>10:45:28</var> GMT+0</small><br><strong>Postmortem</strong> - ## **Root Cause Analysis – Encoding Service Outage 23 Nov 2025** **Summary** On 23 Nov 2025, the Bitmovin Encoding Platform experienced a service outage affecting our VoD and Live encoding pipeline. The issue resulted in temporary unavailability of encoding operations including starting and stopping of encodings and unavailability of status updates of running encodings for the time of the outage. ## **Root Cause** A **bug in one of our encoding services** caused memory usage and database (DB) data transfer volumes to increase slowly but steadily over the course of approximately one month. This resulted in: * Gradually **escalating DB read traffic** from the encoding services due to increasing object size with time * Overwhelming the service with **growing message queues** for this specific service due to decreased throughput * Data transfer reaching **\~900 MB/s outbound** from the database to the service instances As memory usage kept rising, multiple encoding service instances ultimately **crashed simultaneously**, leading to an abrupt stop in processing and further service degradation due to accumulation of **\~300,000 unprocessed service messages during the outage from 18:00 CET**. ## **Impact** * Encoding jobs failed to start after queueing * Running encoding jobs failed to update their status to “In Progress” or “Finished” during the incident ## **Detection** * 18:06 CET – Investigation initiated based on internal monitoring and alerts. * 18:07 CET – Outage confirmed and posted on status page. Internally, teams observed service pods **crashing due to memory limit exhaustion**, causing message processing interruptions and queue growth. ## **Investigation, Mitigation & Recovery** After the service crashes, we initiated a controlled process: 1. Increased service memory limits on our kubernetes cluster 18:25 CET 2. Reduced parallel message queue worker logic of the affected service to reduce DB read load by about 30% 19:02 CET 3. Implemented further monitoring and tracing in the service to increase visibility into the massive DB data streaming causes 19:38 CET 4. Re-routed several messages on our messaging system to further decrease DB read load 19:51 CET 5. Sped up processing of messages causing excessive read operations through service based skipping with a new service version 20:23 CET 6. Deployed new service which decreased DB read load to normal levels 20:56 7. Increased message throughput for services to restore the system state by processing the message backlog of about 350k messages in RabbitMQ 21:30 CET 8. Restored all service configuration and gradually increased encoding throughput 23:00 CET 9. Full service restored 23:31 CET ## **Next Steps and Preventive Actions** To prevent recurrence, we are implementing the following: * Implement a holistic fix for the bug causing excessive DB read load * Expand message queue and memory usage monitoring with stricter alerts for read amplification * Comprehensive review of encoding service code paths related to heavy DB reads * Message queue pressure safeguards including throttling for struggling services.</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>20:33:10</var> GMT+0</small><br><strong>Investigating</strong> - Our team has likely identified the root cause of high DB load causing delays in encoding status updates in our API and dashboard. We are deploying a new version of the affected service and will have an update in a few minutes. .</p> <p><small>Nov <var data-var='date'> 23</var>, <var data-var='time'>19:24:41</var> GMT+0</small><br><strong>Investigating</strong> - Our team is continuing to investigate the issue impacting slow API requests, delayed encoding status updates and delayed manifest generations (manifests generated with the encoding start calls). Encoding output is not affected. At this time, we have not yet identified the root cause. Diagnostic efforts are ongoing, and all necessary teams are engaged. We sincerely apologize for the inconvenience and will provide updates as soon as we have more information. Thank you for your patience and understanding..</p> ]]> Sun, 23 Nov 2025 17:07:07 +0000 https://status.bitmovin.com/incident/cmibz0g6g01pz2zk9q7sp00oj https://status.bitmovin.com/incident/cmibz0g6g01pz2zk9q7sp00oj Encoding Failures in us-east-1 Due to EC2 Instance Creation Issues Type: Incident Duration: 9 hours and 18 minutes Affected Components: Amazon Web Services Oct 20, 10:30:00 GMT+0 - Monitoring - Since 12:30 UTC on October 20, our encoding service in the us-east-1 region has again been impacted by an AWS issue preventing the creation of new EC2 instances. As a result, new encoding jobs in this region may fail to start. Running encodings on existing capacity are not affected. Workaround: We recommend all customers switch to another cloud region that is not affected or make use of the fallbackRegion setting when creating encodings, as described in our Documentation: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings>. We are monitoring AWS’s recovery efforts closely and will provide further updates as more information becomes available. Oct 20, 13:33:00 GMT+0 - Monitoring - The AWS incident affecting EC2 instance creation in us-east-1 is still ongoing. As a result, encoding jobs in this region may continue to fail to start. There are currently no actions we can take on our side until AWS resolves the underlying issue. Therefore, we will pause posting further updates until AWS has marked the incident as resolved. For the latest information, we recommend monitoring the AWS Health Dashboard: <https://health.aws.amazon.com/health/status> In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings> Oct 20, 12:58:00 GMT+0 - Monitoring - The incident is still ongoing. Our encoding service in us-east-1 continues to be impacted by AWS EC2 instance creation issues. We are waiting for further updates from AWS and will continue to monitor their recovery efforts. For more detailed and up-to-date information, we recommend customers check the AWS Health Dashboard: <https://health.aws.amazon.com/health/status> In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings> Oct 20, 19:48:00 GMT+0 - Resolved - The AWS incident affecting EC2 instance creation in us-east-1 has been resolved by AWS as of Oct 20 21:48 UTC Type: Incident

Duration: 9 hours and 18 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>10:30:00</var> GMT+0</small><br><strong>Monitoring</strong> - Since 12:30 UTC on October 20, our encoding service in the us-east-1 region has again been impacted by an AWS issue preventing the creation of new EC2 instances. As a result, new encoding jobs in this region may fail to start. Running encodings on existing capacity are not affected. Workaround: We recommend all customers switch to another cloud region that is not affected or make use of the fallbackRegion setting when creating encodings, as described in our Documentation: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings>. We are monitoring AWS’s recovery efforts closely and will provide further updates as more information becomes available..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>13:33:00</var> GMT+0</small><br><strong>Monitoring</strong> - The AWS incident affecting EC2 instance creation in us-east-1 is still ongoing. As a result, encoding jobs in this region may continue to fail to start. There are currently no actions we can take on our side until AWS resolves the underlying issue. Therefore, we will pause posting further updates until AWS has marked the incident as resolved. For the latest information, we recommend monitoring the AWS Health Dashboard: <https://health.aws.amazon.com/health/status> In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings>.</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>12:58:00</var> GMT+0</small><br><strong>Monitoring</strong> - The incident is still ongoing. Our encoding service in us-east-1 continues to be impacted by AWS EC2 instance creation issues. We are waiting for further updates from AWS and will continue to monitor their recovery efforts. For more detailed and up-to-date information, we recommend customers check the AWS Health Dashboard: <https://health.aws.amazon.com/health/status> In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: <https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings>.</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>19:48:00</var> GMT+0</small><br><strong>Resolved</strong> - The AWS incident affecting EC2 instance creation in us-east-1 has been resolved by AWS as of Oct 20 21:48 UTC.</p> ]]> Mon, 20 Oct 2025 10:30:00 +0000 https://status.bitmovin.com/incident/cmhd90qjf00vnvapz2tvu5vb9 https://status.bitmovin.com/incident/cmhd90qjf00vnvapz2tvu5vb9 Elevated API Error Rate on Muxing creation Type: Incident Duration: 40 minutes Affected Components: Encoding API Oct 20, 22:04:00 GMT+0 - Postmortem - # Incident Post-Mortem – Encoding Cleanup Gap-Lock **Date:** 2025-10-16 **Duration:** 07:49–08:30 UTC (41 minutes) **Customer Impact:** Calls to the `https://api.bitmovin.com/v1/encoding/encodings/{encoding_id}/muxings/` endpoints failed, leading to workflow stoppages. The failure rate remained below monitoring thresholds, so the issue was not immediately detected. ## Summary At 07:49 UTC, our encoding cleanup process (responsible for deleting old encodings outside of retention) triggered a **gap-lock** on one of our database tables. While the long-running delete transaction was active, the API was unable to insert new records into the table. This directly impacted the **muxings endpoints**, which failed during this period and caused encoding workflows to stop. Because the error rate was below our monitoring alert thresholds, the incident went unnoticed until the team was alerted at 08:20 UTC. ## Root Cause The issue was caused by **outdated MySQL table statistics**, which led the **MySQL query optimizer** to select an inefficient execution plan: * Instead of using an index scan on the large table, the query optimizer chose a **full table scan**. * Under **REPEATABLE\_READ isolation**, this resulted in **gap-locks** across wide portions of the table. * In practice, this behaved like a **table-level lock**, blocking all inserts until the cleanup query was stopped. ## Timeline * **07:49 UTC** – Encoding cleanup process starts. Optimizer chooses full table scan → gap-locks prevent inserts. * **07:49–08:20 UTC** – Calls to `/encoding/encodings/{encoding_id}/muxings/` fail, workflows stop. Monitoring does not alert as errors are below configured threshold. * **08:20 UTC** – Engineering team is alerted. * **08:20–08:30 UTC** – Cleanup process identified as root cause. Process is stopped, lock released. * **08:30 UTC** – Incident resolved, API resumes normal operation. ## Immediate Actions After resolving the incident, we immediately stopped the cleanup process and investigated why it suddenly caused issues despite having run successfully for years. Since then, we have: * **Tightened timeouts** for deletion commands in the cleanup process. * **Changed to a more forgiving isolation level** to prevent broad blocking locks. * **Adjusted the cleanup process to use smaller batch sizes and higher concurrency**, which reduced database impact and increased throughput. * **Added significantly more monitoring** to the cleanup process to better track query performance and catch anomalies early. ## Next Steps * **Update MySQL statistics** so the query optimizer selects the correct execution plan. * _Expected completion:_ **by end of this week (2025-10-24)** * **Rework alerting strategy** to detect low-rate but workflow-blocking errors earlier. * _Expected completion:_ **within 3 weeks (by 2025-11-06)** Oct 16, 05:50:00 GMT+0 - Investigating - We are currently investigating this incident. Oct 16, 06:30:00 GMT+0 - Resolved - We experienced elevated API error rates on the encoding endpoints between 07:50 and 08:30 UTC. The issue, caused by the encoding service during muxing creation, was identified and promptly resolved. The engineering team is preparing a post-mortem and will share it here once available. Type: Incident

Duration: 40 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>22:04:00</var> GMT+0</small><br><strong>Postmortem</strong> - # Incident Post-Mortem – Encoding Cleanup Gap-Lock **Date:** 2025-10-16 **Duration:** 07:49–08:30 UTC (41 minutes) **Customer Impact:** Calls to the `https://api.bitmovin.com/v1/encoding/encodings/{encoding_id}/muxings/` endpoints failed, leading to workflow stoppages. The failure rate remained below monitoring thresholds, so the issue was not immediately detected. ## Summary At 07:49 UTC, our encoding cleanup process (responsible for deleting old encodings outside of retention) triggered a **gap-lock** on one of our database tables. While the long-running delete transaction was active, the API was unable to insert new records into the table. This directly impacted the **muxings endpoints**, which failed during this period and caused encoding workflows to stop. Because the error rate was below our monitoring alert thresholds, the incident went unnoticed until the team was alerted at 08:20 UTC. ## Root Cause The issue was caused by **outdated MySQL table statistics**, which led the **MySQL query optimizer** to select an inefficient execution plan: * Instead of using an index scan on the large table, the query optimizer chose a **full table scan**. * Under **REPEATABLE\_READ isolation**, this resulted in **gap-locks** across wide portions of the table. * In practice, this behaved like a **table-level lock**, blocking all inserts until the cleanup query was stopped. ## Timeline * **07:49 UTC** – Encoding cleanup process starts. Optimizer chooses full table scan → gap-locks prevent inserts. * **07:49–08:20 UTC** – Calls to `/encoding/encodings/{encoding_id}/muxings/` fail, workflows stop. Monitoring does not alert as errors are below configured threshold. * **08:20 UTC** – Engineering team is alerted. * **08:20–08:30 UTC** – Cleanup process identified as root cause. Process is stopped, lock released. * **08:30 UTC** – Incident resolved, API resumes normal operation. ## Immediate Actions After resolving the incident, we immediately stopped the cleanup process and investigated why it suddenly caused issues despite having run successfully for years. Since then, we have: * **Tightened timeouts** for deletion commands in the cleanup process. * **Changed to a more forgiving isolation level** to prevent broad blocking locks. * **Adjusted the cleanup process to use smaller batch sizes and higher concurrency**, which reduced database impact and increased throughput. * **Added significantly more monitoring** to the cleanup process to better track query performance and catch anomalies early. ## Next Steps * **Update MySQL statistics** so the query optimizer selects the correct execution plan. * _Expected completion:_ **by end of this week (2025-10-24)** * **Rework alerting strategy** to detect low-rate but workflow-blocking errors earlier. * _Expected completion:_ **within 3 weeks (by 2025-11-06)**.</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>05:50:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Resolved</strong> - We experienced elevated API error rates on the encoding endpoints between 07:50 and 08:30 UTC. The issue, caused by the encoding service during muxing creation, was identified and promptly resolved. The engineering team is preparing a post-mortem and will share it here once available..</p> ]]> Thu, 16 Oct 2025 05:50:00 +0000 https://status.bitmovin.com/incident/cmhd9c92501za2n0mmpo4us8c https://status.bitmovin.com/incident/cmhd9c92501za2n0mmpo4us8c Planned Database Downtime Type: Maintenance Duration: 2 hours Affected Components: Encoding API, , Amazon Web Services, Oracle Cloud Infrastructure, Microsoft Azure, Akamai Cloud, Google Cloud, Encoding Scheduler, Statistics Service, Account Service, Billing Service, Notification Service, Cloud Provisioning → Sep 27, 07:00:00 GMT+0 - Identified - Bitmovin’s Encoding services will undergo a database upgrade on Saturday September 27, 2025 from 09:00 - 11:00 UTC to improve performance and stability. Please do not start any new encoding jobs during the maintenance window, as they will not be processed. API status updates will be delayed, and the dashboard and Analytics will be unavailable. Please plan accordingly. Sep 27, 09:00:00 GMT+0 - Completed - Maintenance has completed successfully. Type: Maintenance

Duration: 2 hours

Affected Components: , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 27</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Identified</strong> - Bitmovin’s Encoding services will undergo a database upgrade on Saturday September 27, 2025 from 09:00 - 11:00 UTC to improve performance and stability. Please do not start any new encoding jobs during the maintenance window, as they will not be processed. API status updates will be delayed, and the dashboard and Analytics will be unavailable. Please plan accordingly..</p> <p><small>Sep <var data-var='date'> 27</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> Sat, 27 Sep 2025 07:00:00 +0000 https://status.bitmovin.com/maintenance/cmhdaczks00wsivprjfglc5ft https://status.bitmovin.com/maintenance/cmhdaczks00wsivprjfglc5ft Encoding failures on AWS Type: Incident Duration: 1 hour and 8 minutes Affected Components: Amazon Web Services Sep 4, 03:45:00 GMT+0 - Investigating - We are currently experiencing failures with encodings on AWS. Our engineering team is currently investigating the issue Sep 4, 04:53:00 GMT+0 - Resolved - We have identified the root cause of the encoding failures: a misconfiguration in our S3 bucket. The S3 configuration has been corrected, and encoding jobs on AWS are now recovering. We are seeing encoding tasks completing successfully again. We are actively monitoring the system to confirm full recovery. Sep 4, 04:53:00 GMT+0 - Resolved - # Root Cause Analysis: Encoding Failures on AWS ### Summary On **September 4, 2025**, between **06:31 AM and 08:10 AM CEST**, encoding jobs running on AWS failed due to an issue accessing our S3 storage. ### Root Cause The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage. ### Impact * Only encoding jobs running on **AWS** were affected. * Encodings on other cloud providers and all other Bitmovin services were **not impacted**. ### Resolution The configuration was corrected at **08:10 AM CEST**, restoring access to S3\. Encoding operations on AWS recovered immediately and have been stable since. ### Preventive Measures To prevent this from happening again, we are: * Updating our key rotation procedure to ensure keys are not deleted prematurely. * Automating the key rotation process to reduce the chance of operator error. Type: Incident

Duration: 1 hour and 8 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>03:45:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing failures with encodings on AWS. Our engineering team is currently investigating the issue.</p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>04:53:00</var> GMT+0</small><br><strong>Resolved</strong> - We have identified the root cause of the encoding failures: a misconfiguration in our S3 bucket. The S3 configuration has been corrected, and encoding jobs on AWS are now recovering. We are seeing encoding tasks completing successfully again. We are actively monitoring the system to confirm full recovery..</p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>04:53:00</var> GMT+0</small><br><strong>Resolved</strong> - # Root Cause Analysis: Encoding Failures on AWS ### Summary On **September 4, 2025**, between **06:31 AM and 08:10 AM CEST**, encoding jobs running on AWS failed due to an issue accessing our S3 storage. ### Root Cause The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage. ### Impact * Only encoding jobs running on **AWS** were affected. * Encodings on other cloud providers and all other Bitmovin services were **not impacted**. ### Resolution The configuration was corrected at **08:10 AM CEST**, restoring access to S3\. Encoding operations on AWS recovered immediately and have been stable since. ### Preventive Measures To prevent this from happening again, we are: * Updating our key rotation procedure to ensure keys are not deleted prematurely. * Automating the key rotation process to reduce the chance of operator error..</p> ]]> Thu, 4 Sep 2025 03:45:00 +0000 https://status.bitmovin.com/incident/cmhda393z0116g3728rzazrfs https://status.bitmovin.com/incident/cmhda393z0116g3728rzazrfs