Root Cause Analysis: AWS Image Authorization Issues for Cloud Connect Customers

On August 11, AWS Cloud Connect customers temporarily lost access to the encoder AMIs, causing encodings to fail at instance startup. The cause was a recent change to our infrastructure management API that altered internal request routing. As a result, an AMI access synchronization step in our encoder release process received an empty customer list and revoked customer AMI permissions. A first, partially completed run affected a small number of customers starting at 14:06 UTC; a second run at 14:58 UTC revoked access for all remaining customers. Access was fully restored by 15:46 UTC. No data was lost.

Timeline (UTC)

• 14:06 - An encoder release process runs. Its AMI access synchronization step queries the list of Cloud Connect customers, receives an empty result, and begins revoking customer access to the encoder AMIs. The release process is restarted, canceling the run partway through. A small number of customers are affected at this point.

• 14:46 - The first customer reports about failing encodings on AWS come in. Since only a small number of customers are affected at this point, the initial investigation focuses on customer-specific causes.

• 14:58 - The restarted release process completes. The same synchronization step again receives an empty customer list and revokes AMI access for all remaining customers.

• ~15:28 - After the second synchronization run, internal monitoring alerts on failing encoding startups across all AWS Cloud Connect customers, making the wider scope of the issue visible.

• 15:30 - The revoked AMI permissions are identified as the cause and traced back to the release process.

• 15:34 - We trigger a re-run of the grant process with the correct customer list.

• 15:46 - The grant process completes. AMI access is restored and encodings start normally again. We continue monitoring to confirm stable operation.

Root cause

Our encoder release process includes a step that synchronizes customer access permissions to the encoder AMIs. It queries our infrastructure management API for the list of Cloud Connect customers and grants or revokes AMI access accordingly.

In the week before the incident, we made changes to the infrastructure management API to improve resiliency and redundancy. These changes also altered how requests are routed internally. When the release process ran on August 11, its customer list query was routed to an endpoint that returned an empty list with a successful response. Because the response was technically valid, no error was raised, and the synchronization step treated the empty list as the desired state and revoked customer AMI permissions. This happened in two runs: a first run that was canceled partway through when the release process was restarted, and a second run that completed and revoked access for all remaining customers.

Resolution

We re-ran the grant process with the correct customer list, which restored AMI access for all affected customers. Encodings started normally again as of 15:46 UTC, and we confirmed stable operation over the following hours.

Preventive measures

• We are adding safeguards to the AMI access synchronization so it fails instead of acting when the customer list is empty or shrinks beyond a defined threshold.

• We are removing ACL management from the encoder release process entirely. Customer AMI access is already managed by our infrastructure service, and the release process was an unintended second place where these permissions could be changed. Consolidating this into a single owner eliminates this class of failure.

• We are improving our detection for this failure mode so that permission changes of this kind are alerted on directly, rather than indirectly through failing encodings. This will let us identify the scope of similar issues faster, even when only few customers are initially affected.

We apologize for the disruption this caused. If you have questions about this incident, please reach out to our support team.

This incident has been resolved.
Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been operating normally since 15:46 UTC.
We will publish a detailed root cause analysis within 24 hours.
We apologize for any inconvenience caused.

This incident has been resolved. Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been running normally since the fix was deployed.

We have identified the root cause of the AWS image authorization issue affecting Cloud Connect customers and are currently rolling out a fix. We will provide a further update once the fix has been fully deployed.