CircleCI Status

Resolved: We are concluding this incident based on a confirmed fix with our infrastructure provider and successful tests. For any customers seeing failed runs, please run again for a successful connection.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Due to continued reports of intermittent failures and a lack of sustained improvement while in monitoring status, this incident is elevated back to investigating to escalate with our infrastructure provider.

Monitoring: We are continuing to monitor. We still observe a low percentage of intermittent failures. The failure rate is trending down as we continue to monitor. We advise customers to contact CircleCI Support if they see a change in degraded services and increased failure rate.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Our team is moving this incident to Monitoring status based on significant improvement seen in our tests and low error rate. We are seeing improvement in network routing for Mac jobs and are actively engaged with our infrastructure provider. If your Mac jobs experience network-related failures or degraded service please reach out to CircleCI support.

Identified: We are seeing improvement in Mac job network routing.

We will continue to monitor for signs of recovery.

Identified: Mac jobs are still seeing intermittent failures with network routing.

We are working with our infrastructure provider to address the networking issues.

Identified: We have implemented a change to address the routing issues and are monitoring the results.

Investigating: What's happening Our engineers are investigating network routing issues for some Mac jobs.

What's impacted Mac jobs that make outbound requests, including checking out from GitHub.

Resolved: The issue where customers running Docker jobs experienced significant delays, with jobs stuck in a "preparing environment" state, has now been resolved. We thank you for your patience while our team worked on implementing a fix.

Monitoring: What's happening Our engineers have deployed a fix and we are observing the job backlog clearing. Docker jobs are now starting normally.

What's impacted Customers who submitted Docker jobs during the incident window may still see some jobs completing with elevated queue times. Newly submitted jobs should be starting at normal times.

What can you expect Docker job startup times are returning to normal. Thank you for your patience while our engineers resolved this issue.

We are monitoring the recovery and will provide a further update if the situation changes.

Identified: What's impacted Customers running Docker jobs. Affected jobs are queuing rather than starting, resulting in elevated wait times.

What can you expect Docker jobs may remain stuck in "preparing environment" for an extended period. Our engineers have identified the cause and are actively working to restore normal job startup times. Thank you for your patience while our team deploys a fix.

Next update We will provide another update as soon as more information is available

Identified: We are deploying a change to mitigate this issue.

Investigating: We are investigating reports of delays starting Docker jobs.