Resolved: Response times have returned to normal. This incident is now resolved.

Monitoring: We've increased the compute available to the secrets service, and have seen response times return to normal levels.

Investigating: We're observing increased latency on secrets endpoints for a subset of our customers. We're currently investigating and will provide status updates as they become available.

Resolved: We have completed the provisioning of additional capacity mentioned in our last update, and error rates and response times have returned to normal.

This incident is now resolved.

Monitoring: We continue to observe high latency on isolated infrastructure serving Agent API endpoints for a subset of customers. We are provisioning additional capacity to address this latency, and have informed impacted customers.

Monitoring: We've seen a small number of unrelated issues, each affecting a subset of customers. Most impact is resolved, but we are continuing to monitor impact for a small number of remaining customers. We are in touch with those customers directly.

Monitoring: We've made some changes to address the issue and are seeing signs of recovery. We continue to monitor the situation.

Investigating: We continue to experience high latency on some services. We're continuing to identify root causes.

Investigating: We're still experiencing latency issues for agent api and job dispatch. We continue to investigate and identify the root cause.

Investigating: We're seeing elevated job dispatch latency and Agent API latency across multiple shards. We're investigating.

Postmortem: # Service Impact

Between approximately 18:00 UTC and 22:50 UTC February 26, 2026, a subset of customers experienced increased latency when dispatching jobs to agents. Affected customers observed agents sitting idle for several minutes despite having matching jobs waiting in the queue. Job dispatch eventually succeeded, but with significantly elevated latency. The impact was concentrated on specific database shards but affected customers across multiple shards over the course of the incident.

Incident Summary

A database maintenance task designed to improve job ordering performance was running across all production database shards. However, this task was itself contributing significant database load, which impacted normal job dispatch and pipeline upload operations. This increased database load caused dispatch operations to queue up, resulting in the observed delays in matching jobs to agents.

The issue was compounded by a connection pooling service having several containers running on underperforming infrastructure, which reduced the available database throughput.

Contributing factors:

• The maintenance task consumed limited database resources, which conflicted with concurrent dispatch operations
• The task was running simultaneously across all database shards, amplifying the impact
• A connection pooling service had degraded capacity due to infrastructure imbalance

Changes we're making

• The maintenance task has been paused and in future will be run during low-traffic periods and on individual shards rather than all shards simultaneously
• The connection pooling service has been rebalanced to ensure consistent performance
• We are improving our monitoring and dashboards to enable faster identification of lock contention issues during incidents

Resolved: We have seen a full recovery of service, and have a good understanding of the underlying cause. We will publish a post-incident review next week.

Monitoring: We've seen recovery for the remaining subset of customers. We will continue to monitor.

Investigating: We're seeing ongoing latency impact across for a subset of customers. Some customers are seeing signs of improvement, but we are continuing to investigate the issue.

Monitoring: We're seeing signs of recovery and will continue to monitor.

Identified: Some customers are experiencing increased latency for jobs being assigned to agents. We have identified the cause and are working on mitigations.

Resolved: We have completed the provisioning of additional capacity mentioned in our last update, and error rates and response times have returned to normal.

This incident is now resolved.

Monitoring: We continue to observe high latency on isolated infrastructure serving Agent API endpoints for a subset of customers. We are provisioning additional capacity to address this latency, and have informed impacted customers.

Monitoring: We've seen a small number of unrelated issues, each affecting a subset of customers. Most impact is resolved, but we are continuing to monitor impact for a small number of remaining customers. We are in touch with those customers directly.

Monitoring: We've made some changes to address the issue and are seeing signs of recovery. We continue to monitor the situation.

Investigating: We continue to experience high latency on some services. We're continuing to identify root causes.

Investigating: We're still experiencing latency issues for agent api and job dispatch. We continue to investigate and identify the root cause.

Investigating: We're seeing elevated job dispatch latency and Agent API latency across multiple shards. We're investigating.

Resolved: Processing of test execution ingestion data has successfully caught up.

Monitoring: Processing the backlog of test executions is taking longer than expected, so elevated ingestion latency remains. We continue to monitor progress. We have also identified database schema improvements we can make that will increase throughput to avoid this scenario in future.

Monitoring: We've identified and fixed the root cause and now monitoring the issue. Test execution may be delayed for up to 30 minutes.