scalr.io Status

Postmortem: ## Summary

On October 22, 2026, between 15:45 and 16:35 UTC, some runs failed to start. Affected tasks failed with the error “Cannot provision execution environment”. The issue was caused by a sudden spike in run volume that exceeded a capacity limit in our cluster networking. It resolved on its own once the spike subsided, and the service is operating normally. A fix that removes the underlying cause is rolling out.

Customer impact

Between 15:45 and 16:35 UTC, ~2,000 tasks failed across ~1,800 runs, affecting roughly 100 accounts. Affected runs failed with an internal error because the agent worker never picked up the task, so no work was started. Tasks that were already running were not affected.

If any of your runs failed during this window, please re-run them.

Timeline

All times are UTC on October 22, 2026.

Root cause

Each agent task runs in its own short-lived pod, and each pod was labeled with values unique to that task. Our cluster networking creates a separate network identity for every unique set of labels, so each task created its own identity.

Those identities are tracked in a table with a fixed size limit. At roughly twice our normal hourly task volume, the number of identities exceeded that limit. Pods that started while the table was full came up without DNS access, so the agent worker could not pick up the task, and it failed after 300 seconds.

The limit existed before this event. A burst of runs from several customers across many workspaces was what pushed us over it.

Resolution

The issue cleared on its own as the burst subsided and unused network identities were cleaned up. In parallel, we began rolling out a release that removes the per-task unique values from task pod labels, so tasks no longer each create their own identity. This reduces identity usage by orders of magnitude, well below the limit.

Preventing a recurrence

1. Remove the cause. We are rolling out the label change described above, narrowing the DNS network rule on task pods so it consumes far less capacity, and further hardening our networking configuration, which we will validate outside production first.
2. Improve monitoring. We are adding alerts on networking capacity so we are warned well before the limit is reached.
3. Detect failures sooner. We are adding alerts on the rate of internal-error run failures and on the Task was not started by the agent worker within 300s error, so issues like this are caught quickly regardless of cause.

Our apology

We are sorry for the disruption this caused your teams and pipelines. Reliable run execution is core to what you depend on us for. If you have questions about how this affected your account, please contact Scalr support.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are investigating an issue causing runs to fail with the error: “Cannot provision execution environment.”

This does not impact runs on self-hosted agents.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On October 22, 2026, between 15:45 and 16:35 UTC, some runs failed to start. Affected tasks failed with the error “Cannot provision execution environment”. The issue was caused by a sudden spike in run volume that exceeded a capacity limit in our cluster networking. It resolved on its own once the spike subsided, and the service is operating normally. A fix that removes the underlying cause is rolling out.

Customer impact

Between 15:45 and 16:35 UTC, ~2,000 tasks failed across ~1,800 runs, affecting roughly 100 accounts. Affected runs failed with an internal error because the agent worker never picked up the task, so no work was started. Tasks that were already running were not affected.

If any of your runs failed during this window, please re-run them.

Timeline

All times are UTC on October 22, 2026.

Root cause

Each agent task runs in its own short-lived pod, and each pod was labeled with values unique to that task. Our cluster networking creates a separate network identity for every unique set of labels, so each task created its own identity.

Those identities are tracked in a table with a fixed size limit. At roughly twice our normal hourly task volume, the number of identities exceeded that limit. Pods that started while the table was full came up without DNS access, so the agent worker could not pick up the task, and it failed after 300 seconds.

The limit existed before this event. A burst of runs from several customers across many workspaces was what pushed us over it.

Resolution

The issue cleared on its own as the burst subsided and unused network identities were cleaned up. In parallel, we began rolling out a release that removes the per-task unique values from task pod labels, so tasks no longer each create their own identity. This reduces identity usage by orders of magnitude, well below the limit.

Preventing a recurrence

1. Remove the cause. We are rolling out the label change described above, narrowing the DNS network rule on task pods so it consumes far less capacity, and further hardening our networking configuration, which we will validate outside production first.
2. Improve monitoring. We are adding alerts on networking capacity so we are warned well before the limit is reached.
3. Detect failures sooner. We are adding alerts on the rate of internal-error run failures and on the Task was not started by the agent worker within 300s error, so issues like this are caught quickly regardless of cause.

Our apology

We are sorry for the disruption this caused your teams and pipelines. Reliable run execution is core to what you depend on us for. If you have questions about how this affected your account, please contact Scalr support.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are investigating an issue causing runs to fail with the error: “Cannot provision execution environment.”

This does not impact runs on self-hosted agents.

Investigating: We are currently investigating this issue.