CircleCI Status

Resolved: The incident has been resolved. Thank you for your patience. The "infra-fail" should no longer be occurring.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are slowly returning to normal. If you had jobs that failed with "infra-fail", those jobs can be rerun. We thank you for your patience while our engineers worked to get our system back to stability.

Investigating: We're currently investigating a high number of infra failures. We'll update as soon as we know more details.

Resolved: The incident has been resolved. Thank you for your patience. The "infra-fail" should no longer be occurring.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are slowly returning to normal. If you had jobs that failed with "infra-fail", those jobs can be rerun. We thank you for your patience while our engineers worked to get our system back to stability.

Investigating: We're currently investigating a high number of infra failures. We'll update as soon as we know more details.

Resolved: This incident has been resolved.

Monitoring: All jobs should now be running normally. The backlog of jobs has cleared and jobs should be running in typical time. We will continue monitoring to ensure consistent service. Thank you for your patience.

Identified: We are continuing to see recovery, but some customers may still experience delays in jobs running on M4 executors. We appreciate your patience whilst we work through the backlog of jobs.

Identified: We have mitigated the issue and MacOS jobs which do not have a resource class specified should run normally. Customers can rerun jobs which failed an they should now run normally without any modification to their config. Thank you for your patience whillst we resolved this.

Customers using M4 machine jobs may experience minor delays in jobs starting during this period as there are a large number of re-runs occurring.

Identified: We have identified an issue in our system which mean that MacOS jobs which don't specify a resource class will fail to run with an "Invalid resource class" error. We are working to correct this, but customers can workaround this by specifying a resource class in their config.yml.

Resolved: The incident has been resolved. Thank you for your patience. The "infra-fail" should no longer be occurring.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are slowly returning to normal. If you had jobs that failed with "infra-fail", those jobs can be rerun. We thank you for your patience while our engineers worked to get our system back to stability.

Investigating: We're currently investigating a high number of infra failures. We'll update as soon as we know more details.

Resolved: The incident has been resolved. Thank you for your patience. The "infra-fail" should no longer be occurring.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are slowly returning to normal. If you had jobs that failed with "infra-fail", those jobs can be rerun. We thank you for your patience while our engineers worked to get our system back to stability.

Investigating: We're currently investigating a high number of infra failures. We'll update as soon as we know more details.

Resolved: Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC.

What was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window.

Resolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them.

The incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures

Monitoring: As of 16:32 UTC, job triggering and workflow starts have returned to normal operation across all resource classes. The impact was limited to a 12-minute window between 16:20 and 16:32 UTC.

What's impacted: All new jobs and workflows are now starting normally.

What to expect: If you have workflows or jobs that were stuck during the 16:20-16:32 UTC window, please manually rerun them.

We are continuing to investigate the root cause of this disruption and will provide an update within 30 minutes or once our investigation is complete.

Investigating: At 16:20 UTC, we began experiencing delays in job triggering and starts across all resource classes. Some workflows and jobs may appear stuck in a running state.

What’s impacted: Job triggering is experiencing delays or is stuck. This affects all resource classes and executors.

What to expect: If you have workflows that appear stuck and haven’t started, we recommend manually rerunning them.

We are actively investigating the root cause and working to restore normal processing speeds. Next update: We will provide an update within 30 minutes or earlier with our progress

Resolved: We have resolved the issues affecting job triggering, workflow starts, and API queries. Our systems have been stabilized and are operating normally.

What was impacted: Job triggering, workflow starts, API queries, and pipeline page loading experienced disruptions for some customers. This affected all resource classes and executors.

Resolution: We implemented mitigation measures to address high volume workflow queries impacting our internal systems and increased system capacity. All new jobs and workflows are now starting normally, pipeline pages are loading, and API queries are functioning as expected.

What to expect: If you have jobs that became stuck during this incident, please rerun them. If you continue to experience issues after rerunning, please contact our support team. Some customers may still see jobs stuck in a cancelling state. Engineering is aware and addressing to mitigate risk.

We will continue monitoring our systems and conducting a thorough review to identify additional preventive measures.

Investigating: We have deployed changes to mitigate the high volume of workflow queries impacting our systems. Pipeline pages that were previously failing to load are now loading successfully, and we are seeing significant reduction in API errors.

What's impacted: Some customers continue to experience jobs stuck in a not-running state from earlier in the incident. New job triggering and workflow starts are now functioning normally.

What's happening: We have implemented mitigation measures and increased system capacity. We are continuing to investigate the remaining stuck jobs for affected customers.

What to expect: If you experienced issues loading pipeline pages or querying workflow data via the API, these should now be resolved. New jobs and workflows should trigger normally. If you have jobs that appeared stuck earlier, please try rerunning them while we continue to investigate the reports of jobs that do remain stuck for a small number of customers. The data for those workflows should be available and queryable.

Next update: We will provide an update within 30 minutes. Thank you for your patience while our engineers work through this incident.

Investigating: We are currently experiencing issues affecting job triggering and workflow starts across all resource classes. Jobs may appear stuck in a not-running state, and some customers may encounter 500 errors when making API calls to check job or workflow status.

What's impacted: Job triggering, workflow starts, and API queries for job and workflow status are experiencing disruptions. This affects all resource classes and executors. Some users may also experience issues loading the pipeline page.

What to expect: We are actively working to stabilize our systems and restore normal operations. We will provide updates as we make progress toward resolution.

We thank you for your patience while we work through these issues - we will update with our progress within 30 minutes or earlier.

Investigating: We are currently investigating reports of jobs not starting. We apologize for the inconvenience.

Resolved: This incident has been resolved. Things should be back to normal.

Identified: We are seeing some issues loading the pipelines page. This is intermittent and won't affect most users. No work is being affected, just the display of pipelines. We have identified the issue and are working on a fix.

Resolved: The issues affecting the pipelines page display are related to a broader incident impacting our systems. We have opened a separate incident tracking job triggering and API status issues, which encompasses the pipelines page loading problems. Please follow https://status.circleci.com/incidents/jq4bgq2sjt1r for ongoing updates.

Investigating: We are continuing to investigate this issue.

Investigating: We are seeing some issues loading the pipelines page. This doesn't affect all users but affects the display of pipelines. We are continuing to investigate the cause of this issue.