Bitmovin - Encoding failures on AWS – Incident details

Encoding failures on AWS

Resolved
Major outage
Started 3 months agoLasted about 1 hour

Affected

Encoding

Major outage from 3:45 AM to 4:53 AM

Cloud Provisioning

Major outage from 3:45 AM to 4:53 AM

Amazon Web Services

Major outage from 3:45 AM to 4:53 AM

Updates
  • Update
    Update

    We have identified the root cause of the encoding failures: a misconfiguration in our S3 bucket.
    The S3 configuration has been corrected, and encoding jobs on AWS are now recovering. We are seeing encoding tasks completing successfully again.
    We are actively monitoring the system to confirm full recovery.

  • Resolved
    Resolved

    Root Cause Analysis: Encoding Failures on AWS

    Summary

    On September 4, 2025, between 06:31 AM and 08:10 AM CEST, encoding jobs running on AWS failed due to an issue accessing our S3 storage.

    Root Cause

    The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage.

    Impact

    • Only encoding jobs running on AWS were affected.

    • Encodings on other cloud providers and all other Bitmovin services were not impacted.

    Resolution

    The configuration was corrected at 08:10 AM CEST, restoring access to S3. Encoding operations on AWS recovered immediately and have been stable since.

    Preventive Measures

    To prevent this from happening again, we are:

    • Updating our key rotation procedure to ensure keys are not deleted prematurely.

    • Automating the key rotation process to reduce the chance of operator error.

  • Investigating
    Investigating

    We are currently experiencing failures with encodings on AWS.
    Our engineering team is currently investigating the issue