Bitmovin - Encoding Failures in us-east-1 Due to EC2 Instance Creation Issues – Incident details

All systems operational

Encoding Failures in us-east-1 Due to EC2 Instance Creation Issues

Resolved
Degraded performance
Started about 2 months agoLasted about 9 hours

Affected

Encoding

Degraded performance from 10:30 AM to 7:48 PM

Cloud Provisioning

Degraded performance from 10:30 AM to 7:48 PM

Amazon Web Services

Degraded performance from 10:30 AM to 7:48 PM

Updates
  • Resolved
    Resolved

    The AWS incident affecting EC2 instance creation in us-east-1 has been resolved by AWS as of Oct 20 21:48 UTC

  • Update
    Update

    The AWS incident affecting EC2 instance creation in us-east-1 is still ongoing. As a result, encoding jobs in this region may continue to fail to start.

    There are currently no actions we can take on our side until AWS resolves the underlying issue. Therefore, we will pause posting further updates until AWS has marked the incident as resolved.

    For the latest information, we recommend monitoring the AWS Health Dashboard: https://health.aws.amazon.com/health/status

    In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings

  • Update
    Update

    The incident is still ongoing. Our encoding service in us-east-1 continues to be impacted by AWS EC2 instance creation issues.

    We are waiting for further updates from AWS and will continue to monitor their recovery efforts. For more detailed and up-to-date information, we recommend customers check the AWS Health Dashboard: https://health.aws.amazon.com/health/status

    In the meantime, we continue to recommend switching to another unaffected cloud region or using the fallbackRegion setting as described in our Encoding Incident Operational Playbook: https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings

  • Monitoring
    Monitoring

    Since 12:30 UTC on October 20, our encoding service in the us-east-1 region has again been impacted by an AWS issue preventing the creation of new EC2 instances.

    As a result, new encoding jobs in this region may fail to start. Running encodings on existing capacity are not affected.

    Workaround:
    We recommend all customers switch to another cloud region that is not affected or make use of the fallbackRegion setting when creating encodings, as described in our Documentation: https://developer.bitmovin.com/encoding/docs/encoding-incident-operational-playbook#1-add-fallback-regions-when-creating-encodings.

    We are monitoring AWS’s recovery efforts closely and will provide further updates as more information becomes available.