Resolved -
This incident has been resolved.
Jul 16, 06:53 UTC
Monitoring -
We’re happy to report that the system has fully recovered and all encodings are now being processed normally across all regions and infrastructure environments.
As a next step, we are conducting a full root cause analysis to understand what led to the incident and to define measures that will prevent similar issues in the future.
We will provide a detailed postmortem once the analysis is completed.
Thank you again for your patience and trust throughout this incident.
Jul 15, 16:41 UTC
Update -
We have now manually recovered all previously stuck encodings and successfully cleaned up the instances that were blocking our GCP SSD quota.
As a result, GCP-based encoding jobs are once again processing successfully. We are currently performing a controlled ramp-up of system capacity to work through the backlog of affected jobs and ensure platform stability.
We continue to monitor closely and will provide further updates as full performance is restored.
Thank you for your patience during this incident.
Jul 15, 15:10 UTC
Update -
We have successfully recovered encoding operations for all non-GCP accounts and fully processed the existing backlog.
We are now beginning a controlled and gradual recovery of our GCP (Google Compute Engine) operations. Scheduling of encoding jobs on GCP remains halted as we work to stabilize the environment and resolve the underlying SSD quota constraints.
We will continue to monitor the situation closely and provide further updates as we increase GCP capacity and restore full service.
Thank you for your continued patience and understanding.
Jul 15, 10:15 UTC
Identified -
We have identified the root cause of the current encoding disruption as an exhaustion of SSD quota within our Google Cloud Platform (GCP) infrastructure. This quota issue is preventing us from provisioning additional instances required to scale encoding jobs, which is causing GCP-based workloads to become extremely slow and trigger our overall system safeguards.
To mitigate the impact, we have paused scheduling of encoding jobs on GCP and are actively working to restore service functionality for customers using non-GCP infrastructure.
Thank you for your patience as we work to fully restore encoding operations.
Jul 15, 09:14 UTC
Monitoring -
We are currently experiencing scheduling delays across our encoding platform. Our monitoring systems detected increased queue times and longer job scheduling intervals starting at approximately 04:00 UTC today.
Our engineering team has been notified and is actively investigating the root cause. We are seeing signs of recovery with scheduling times beginning to improve.
We will provide updates as more information becomes available and will post a follow-up once the issue has been fully resolved.
We apologize for any inconvenience this may have caused and appreciate your patience as we work to restore normal scheduling performance.
Jul 15, 06:37 UTC