Buildkite Status

Resolved: We have confirmed that latency and error rates have returned to normal for impacted customers.

Monitoring: We have identified and fixed the issue with the underlying database for a subset of customers. We are now monitoring the issue.

Investigating: We're observing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

Resolved: An increase in requests has lead to the API service being temporarily saturated. We have updated rate limits to ensure this doesn't re-occur and will add further resources if necessary

Investigating: We're observing increased latency and error rates in the Agent API for a subset of our customers. We're currently investigating and will provide status updates as they become available.

Resolved: Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

Monitoring: Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

Identified: We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

Investigating: We are investigating delays with build and job notifications for a subset of customers.

Resolved: Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

Monitoring: Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

Identified: We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

Investigating: We are investigating delays with build and job notifications for a subset of customers.

Resolved: Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

Monitoring: Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

Identified: We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

Investigating: We are investigating delays with build and job notifications for a subset of customers.

Resolved: Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

Monitoring: Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

Identified: We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

Investigating: We are investigating delays with build and job notifications for a subset of customers.

Resolved: Previously elevated loads with Hosted Agents dispatch have fully recovered.

Monitoring: We have mitigated the issue causing increased Hosted Agents dispatch latency and intermittent timeout errors for a subset of customers. We identified abnormal workload activity that was placing elevated load on a supporting service, and have now blocked that activity and applied additional protections. Service metrics have returned to normal, and we are continuing to monitor closely.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We're observing increased error rates and dispatch latency for a subset of our customers. We're currently investigating and will provide status updates as they become available.