Buildkite Status

Resolved: Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.

Monitoring: We are seeing successful email deliveries and are monitoring processing of our outbound email queue.

Identified: Our on-call engineers are working on failing over to an alternative upstream email provider to restore outbound email notifications. No other Buildkite services have been impacted by this disruption.

Identified: We are experiencing delivery issues with email notifications due to an issue with an upstream provider.

Resolved: Job dispatch for Hosted Agents is no longer experiencing delays.

Monitoring: We are continuing to process through the backlog of delayed jobs and monitoring.

Identified: We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

Investigating: Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

Investigating: We're investigating an issue with job dispatch to our Hosted Agents.

Resolved: Jobs running on Hosted Agents are now being dispatched promptly.

Monitoring: We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs.

Investigating: Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating and will update again soon.

Investigating: We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

Resolved: Jobs running on Hosted Agents are now being dispatched promptly.

Monitoring: We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

Investigating: We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.

Resolved: Job dispatch for Hosted Agents is no longer experiencing delays.

Monitoring: We are continuing to process through the backlog of delayed jobs and monitoring.

Identified: We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

Investigating: Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

Investigating: We're investigating an issue with job dispatch to our Hosted Agents.

Resolved: Jobs running on Hosted Agents are now being dispatched promptly.

Monitoring: We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

Investigating: We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.

Resolved: Job dispatch for Hosted Agents is no longer experiencing delays.

Monitoring: We are continuing to process through the backlog of delayed jobs and monitoring.

Identified: We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

Investigating: Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

Investigating: We're investigating an issue with job dispatch to our Hosted Agents.

Resolved: Jobs running on Hosted Agents are now being dispatched promptly.

Monitoring: We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

Investigating: We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.

Resolved: Job dispatch for Hosted Agents is no longer experiencing delays.

Monitoring: We are continuing to process through the backlog of delayed jobs and monitoring.

Identified: We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

Investigating: Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

Investigating: We're investigating an issue with job dispatch to our Hosted Agents.

Resolved: Jobs running on Hosted Agents are now being dispatched promptly.

Monitoring: We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

Investigating: We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.