scalr.io Status

Postmortem: Summary

A backend optimization deployed on April 24 inadvertently triggered a database performance issue that caused a platform-wide outage. The change has been fully reverted and the platform is stable. We are addressing the underlying query design before re-attempting the optimization.

What Happened

The optimization changed how policy code is delivered during runs, moving from an inline payload to a download from blob storage. This introduced an additional authorization check on each policy download that was not present before.

That authorization check relied on an existing database query with significant hidden complexity: under normal conditions it goes unnoticed, but at production policy-check volume it joined approximately 2 million rows per call, taking around 25 seconds to complete. The increased call frequency exposed this latency, drove the database to 100% CPU utilization, and exhausted the connection pool, making the platform unavailable.

Resolution

The optimization was fully reverted, restoring normal platform behavior. No data was lost or corrupted.

What We're Doing Next

The authorization query is being redesigned to use a direct indexed lookup, which eliminates the row-scan behavior that caused the spike. The optimization will not be re-released until this redesign is complete and validated.

We apologize for the disruption. If you have questions or are still experiencing issues, please contact our support team.

Resolved: This incident has been resolved. If you cannot access your account, please clear your browser cache.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: Please follow updates here.

Resolved: This incident has been resolved.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: Update #1 - 4/30/2026

On April 28 at approximately 15:00 UTC and again on April 29 at approximately 15:22 UTC, a load spike affected our internal task queue, which slowed down the mechanism Scalr uses to transition runs between pipeline stages. In some cases, the transition task failed before it could complete, leaving runs stuck in a waiting state with no automatic recovery path. Both times, the queues cleared on their own while our engineering team investigated. We have already shipped an improvement (released April 30) that makes this transition task significantly more resilient to lock contention, greatly reducing the likelihood of runs getting stuck. We are continuing to investigate the underlying cause of the queue spikes and will share further updates as the investigation progresses.

Update #2 - 5/1/2026

Further investigation points to what triggered the queue slowdowns. Issues with runs getting stuck on Apr 28/29 lined up with high bursts of new runs being created at the same time as several heavier background jobs. Those jobs share the same pool of workers, and when they spiked together, the pool fell behind, which delayed the steps that move runs from one stage to the next. The resilience improvement we shipped on April 30 already addresses that delay.

We are now planning changes to separate the heaviest background work from run-transition processing, so a busy moment in one area cannot slow the other. We will share another update as that work progresses.

Resolved: This incident has been resolved. An RCA will be posted soon.

Monitoring: We are continuing to monitor for any further issues.

Identified: The issue has been identified and a fix is being implemented.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating an issue where some runs are stuck in a pending approval state without the ability to approve them.

Postmortem: Summary

A backend optimization deployed on April 24 inadvertently triggered a database performance issue that caused a platform-wide outage. The change has been fully reverted and the platform is stable. We are addressing the underlying query design before re-attempting the optimization.

What Happened

The optimization changed how policy code is delivered during runs, moving from an inline payload to a download from blob storage. This introduced an additional authorization check on each policy download that was not present before.

That authorization check relied on an existing database query with significant hidden complexity: under normal conditions it goes unnoticed, but at production policy-check volume it joined approximately 2 million rows per call, taking around 25 seconds to complete. The increased call frequency exposed this latency, drove the database to 100% CPU utilization, and exhausted the connection pool, making the platform unavailable.

Resolution

The optimization was fully reverted, restoring normal platform behavior. No data was lost or corrupted.

What We're Doing Next

The authorization query is being redesigned to use a direct indexed lookup, which eliminates the row-scan behavior that caused the spike. The optimization will not be re-released until this redesign is complete and validated.

We apologize for the disruption. If you have questions or are still experiencing issues, please contact our support team.

Resolved: This incident has been resolved. If you cannot access your account, please clear your browser cache.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.