We continued to monitor the situation and everything is working as expected and the error rates in our monitoring are at normal levels. As the duration of the incident was rather short our recommended retry behavior for 5xx errors on the API should have kept the impact minimal. The root cause of the incident was a database migration where a column was added to a huge table which caused it to lock for a few minutes. This table lock prevented successful execution on most encoding-related API calls. With our reduction of the database size which will start in July, this will not happen anymore. We additionally raised awareness in the team to pay extra attention when modifying huge database tables.
Posted Jun 22, 2023 - 11:27 UTC
Update
We see a recovery of error rates on our API and will continue to monitor the situation.
Posted Jun 22, 2023 - 11:16 UTC
Investigating
We're experiencing an elevated level of API errors on encoding endpoints and are currently looking into the issue.
Posted Jun 22, 2023 - 11:12 UTC
This incident affected: Bitmovin API (Encoding Service).