
CircleCI Status
Real-time updates of CircleCI issues and outages
AI outages and updates, in one feedEvery AI model outage and update, in one feed
Modelpeers by StatusSight is live todayModelpeers, from the StatusSight team, is live on Product Hunt today. We'd love your support.Modelpeers by StatusSight just launchedModelpeers, from the StatusSight team, just launched on Product Hunt. We'd love your support.
CircleCI status is Operational
CircleCI Docker Jobs
CircleCI Machine Jobs
CircleCI macOS Jobs
CircleCI Windows Jobs
Lior Grossman
Maker of StatusSight
I built StatusSight. Here's what I'm building next with AI.
In Lior Builds, my free newsletter, I share one story from a product I built with AI, plus short notes on the tools and models worth using: what worked, what did not, and the numbers behind it. Every week or two.
Active Incidents
Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Postmortem: ## Summary
(All times UTC)
From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected.
The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding.
In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19.
Customers do not need to run a full analysis. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data.
The original status page can be found here: Delays and intermittent failures running Smarter Testing.
Background
Smarter Testing uses data stored for each project to decide which tests to run (test impact analysis), how to split them across parallel nodes, and how many nodes a job needs. To decide which tests to run, Smarter Testing uses impact data, a map of which files each test touches, and this was the part of the system most affected by this incident. When a job runs, Smarter Testing records updates to this data, and our system applies them to a database in the background.
During a test job, Smarter Testing contacts our backend service, which reads from that database. If that service is unavailable, test jobs using Smarter Testing cannot run their tests.
The size of a project’s impact data depends on how many tests it has and how many files each test touches. Unusually large impact data can place heavy load on the database, and our safeguards did not stop this.
What Happened
(All times UTC)
At 12:12 on November 19, a Smarter Testing analysis began producing impact data that grew steadily until it was unusually large. Between 13:15 and 14:30, our system scaled the workers that apply these updates. The additional workers added load to the database, and between 13:15 and 14:00, database CPU rose to 96-98%. At 13:20, the workers began timing out on these updates and retrying them repeatedly. From then on, impact data updates were delayed, so test selection used older data and some jobs could run more tests than necessary.
At 14:12, our engineering team was alerted to delays in processing impact data, and at 14:17 we began investigating. At 14:33, we opened an incident. At first, the impact appeared limited to delayed updates.
At 15:07, we found that the impact data being applied was far larger than normal and was driving the load on the database. By 15:17, we projected that the database would use up its disk throughput burst balance in about an hour and that free storage would run out in about two.
We took three steps to remove the source of the load. At 15:27, we blocked the analysis that was producing this data, and at 15:28, we cancelled the job running it. From 15:50, some test jobs using Smarter Testing began to fail intermittently. At 15:51, we dropped the queued updates for this data so the workers would stop retrying them.
We also considered moving the database to a larger instance type. That change requires a restart and a short outage and at that point, the impact appeared limited to delays in impact data, so we opted to remove the load first. At 16:05, we increased storage because free space was falling quickly. That change put the database into a modifying state, which blocked the instance type change until 18:18.
At 16:20, the database used up its disk throughput burst balance and stopped responding. Jobs using Smarter Testing failed. We had no way to switch Smarter Testing off for customers, so we focused on cutting load. Jobs recovered at 16:38. We scaled the workers to zero (16:41) and we deleted the remaining queued updates for this data (16:42), and a single restarted worker had cleared the backlog by 16:58.
At 17:00, database CPU rose again, to 65-82%, while the burst balance was still exhausted. To reduce load, we reduced batch sizes and ran a single worker. By 17:34, we had concluded that only an instance type without burst limits would fix this, but the storage change still had the database in a modifying state. At 17:37, the database stopped responding again, and test jobs using Smarter Testing failed. At 17:53, it failed over to its standby instance, which restored its burst balance, and test jobs were running again by 17:55.
From 16:20 to 18:00, some requests for selection, timing and parallelism data timed out. During that time, dynamic test splitting may have used older timing data, and jobs using dynamic parallelism ran on their maximum number of nodes.
When the storage change completed at 18:18, we announced emergency maintenance, and at 18:50 we moved the database to a larger instance type without burst limits, with about 45 seconds of downtime. We scaled the workers back up to clear the backlog, and we resolved the incident at 19:19.
Future Prevention and Process Improvement
Changes already in place:
A larger database. We moved the Smarter Testing database to an instance type that does not rely on burst capacity, which was the limit that caused both outages. We also increased its storage and tuned its background cleanup. Database CPU fell afterward and we are monitoring it closely.
We made test jobs resilient to Smarter Testing outages. Since the incident, we added a fallback that our engineering team can turn on if Smarter Testing becomes unavailable. With the fallback on, jobs continue to run their tests without Smarter Testing features, rather than failing.
Further improvements
Stronger safeguards against unusually large data. We are adding protections so that unusually large data is limited and we are improving how our system handles failed updates.
We are improving database performance. We are optimizing the queries our system runs against the Smarter Testing database, so it can handle more load.
Warning us sooner. We are adding earlier alerts on database capacity.
Conclusion
Since the incident, the Smarter Testing database has been running on infrastructure with substantially more resources and no burst limits, and it has been stable since the change. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve this incident.
Please reach out to our support team with any questions or concerns.
Resolved: Smarter Testing functionality is restored.
We appreciate your patience as we worked through this.
Monitoring: The maintenance was successful. Smarter Testing functionality has been restored.
We are monitoring to ensure performance remains stable.
Identified: At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail.
Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:50 UTC
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:30 UTC.
Identified: Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing.
Thank you for your patience while our engineers work to resolve this.
Identified: Since 17:09 UTC, we have been seeing delays in updating test impact analysis data.
Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
Identified: Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
Investigating: Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Recently Resolved Incidents
Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.
Resolved: We have seen recovery for IP range jobs for gen2 docker jobs. We appreciate your patience while we worked through the incident.
Monitoring: We are continuing to monitor the rollout for recovery.
Monitoring: A fix has been fully deployed, we are now monitoring for recovery.
Identified: The fix for IP range docker jobs are rolling out across the fleet. We will monitor for recovery in infra fails.
Identified: We are continuing to rollout a fix for IP range docker jobs. Customers may continue to retry jobs that use IP ranges that experience infra fails.
We will provide an update within 20 minutes on the fix.
Identified: Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.
CircleCI Outage Survival Guide
Also from the maker of StatusSight
When an outage reaches home
PrepHabit tracks major US outages across banks, payments, flights, power and internet, with the cause each company gave and what to do at home.
CircleCI Components
CircleCI Docker Jobs
Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Postmortem: ## Summary
(All times UTC)
From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected.
The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding.
In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19.
Customers do not need to run a full analysis. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data.
The original status page can be found here: Delays and intermittent failures running Smarter Testing.
Background
Smarter Testing uses data stored for each project to decide which tests to run (test impact analysis), how to split them across parallel nodes, and how many nodes a job needs. To decide which tests to run, Smarter Testing uses impact data, a map of which files each test touches, and this was the part of the system most affected by this incident. When a job runs, Smarter Testing records updates to this data, and our system applies them to a database in the background.
During a test job, Smarter Testing contacts our backend service, which reads from that database. If that service is unavailable, test jobs using Smarter Testing cannot run their tests.
The size of a project’s impact data depends on how many tests it has and how many files each test touches. Unusually large impact data can place heavy load on the database, and our safeguards did not stop this.
What Happened
(All times UTC)
At 12:12 on November 19, a Smarter Testing analysis began producing impact data that grew steadily until it was unusually large. Between 13:15 and 14:30, our system scaled the workers that apply these updates. The additional workers added load to the database, and between 13:15 and 14:00, database CPU rose to 96-98%. At 13:20, the workers began timing out on these updates and retrying them repeatedly. From then on, impact data updates were delayed, so test selection used older data and some jobs could run more tests than necessary.
At 14:12, our engineering team was alerted to delays in processing impact data, and at 14:17 we began investigating. At 14:33, we opened an incident. At first, the impact appeared limited to delayed updates.
At 15:07, we found that the impact data being applied was far larger than normal and was driving the load on the database. By 15:17, we projected that the database would use up its disk throughput burst balance in about an hour and that free storage would run out in about two.
We took three steps to remove the source of the load. At 15:27, we blocked the analysis that was producing this data, and at 15:28, we cancelled the job running it. From 15:50, some test jobs using Smarter Testing began to fail intermittently. At 15:51, we dropped the queued updates for this data so the workers would stop retrying them.
We also considered moving the database to a larger instance type. That change requires a restart and a short outage and at that point, the impact appeared limited to delays in impact data, so we opted to remove the load first. At 16:05, we increased storage because free space was falling quickly. That change put the database into a modifying state, which blocked the instance type change until 18:18.
At 16:20, the database used up its disk throughput burst balance and stopped responding. Jobs using Smarter Testing failed. We had no way to switch Smarter Testing off for customers, so we focused on cutting load. Jobs recovered at 16:38. We scaled the workers to zero (16:41) and we deleted the remaining queued updates for this data (16:42), and a single restarted worker had cleared the backlog by 16:58.
At 17:00, database CPU rose again, to 65-82%, while the burst balance was still exhausted. To reduce load, we reduced batch sizes and ran a single worker. By 17:34, we had concluded that only an instance type without burst limits would fix this, but the storage change still had the database in a modifying state. At 17:37, the database stopped responding again, and test jobs using Smarter Testing failed. At 17:53, it failed over to its standby instance, which restored its burst balance, and test jobs were running again by 17:55.
From 16:20 to 18:00, some requests for selection, timing and parallelism data timed out. During that time, dynamic test splitting may have used older timing data, and jobs using dynamic parallelism ran on their maximum number of nodes.
When the storage change completed at 18:18, we announced emergency maintenance, and at 18:50 we moved the database to a larger instance type without burst limits, with about 45 seconds of downtime. We scaled the workers back up to clear the backlog, and we resolved the incident at 19:19.
Future Prevention and Process Improvement
Changes already in place:
A larger database. We moved the Smarter Testing database to an instance type that does not rely on burst capacity, which was the limit that caused both outages. We also increased its storage and tuned its background cleanup. Database CPU fell afterward and we are monitoring it closely.
We made test jobs resilient to Smarter Testing outages. Since the incident, we added a fallback that our engineering team can turn on if Smarter Testing becomes unavailable. With the fallback on, jobs continue to run their tests without Smarter Testing features, rather than failing.
Further improvements
Stronger safeguards against unusually large data. We are adding protections so that unusually large data is limited and we are improving how our system handles failed updates.
We are improving database performance. We are optimizing the queries our system runs against the Smarter Testing database, so it can handle more load.
Warning us sooner. We are adding earlier alerts on database capacity.
Conclusion
Since the incident, the Smarter Testing database has been running on infrastructure with substantially more resources and no burst limits, and it has been stable since the change. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve this incident.
Please reach out to our support team with any questions or concerns.
Resolved: Smarter Testing functionality is restored.
We appreciate your patience as we worked through this.
Monitoring: The maintenance was successful. Smarter Testing functionality has been restored.
We are monitoring to ensure performance remains stable.
Identified: At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail.
Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:50 UTC
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:30 UTC.
Identified: Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing.
Thank you for your patience while our engineers work to resolve this.
Identified: Since 17:09 UTC, we have been seeing delays in updating test impact analysis data.
Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
Identified: Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
Investigating: Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.
Resolved: We have seen recovery for IP range jobs for gen2 docker jobs. We appreciate your patience while we worked through the incident.
Monitoring: We are continuing to monitor the rollout for recovery.
Monitoring: A fix has been fully deployed, we are now monitoring for recovery.
Identified: The fix for IP range docker jobs are rolling out across the fleet. We will monitor for recovery in infra fails.
Identified: We are continuing to rollout a fix for IP range docker jobs. Customers may continue to retry jobs that use IP ranges that experience infra fails.
We will provide an update within 20 minutes on the fix.
Identified: Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.
CircleCI Machine Jobs
Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Postmortem: ## Summary
(All times UTC)
From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected.
The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding.
In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19.
Customers do not need to run a full analysis. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data.
The original status page can be found here: Delays and intermittent failures running Smarter Testing.
Background
Smarter Testing uses data stored for each project to decide which tests to run (test impact analysis), how to split them across parallel nodes, and how many nodes a job needs. To decide which tests to run, Smarter Testing uses impact data, a map of which files each test touches, and this was the part of the system most affected by this incident. When a job runs, Smarter Testing records updates to this data, and our system applies them to a database in the background.
During a test job, Smarter Testing contacts our backend service, which reads from that database. If that service is unavailable, test jobs using Smarter Testing cannot run their tests.
The size of a project’s impact data depends on how many tests it has and how many files each test touches. Unusually large impact data can place heavy load on the database, and our safeguards did not stop this.
What Happened
(All times UTC)
At 12:12 on November 19, a Smarter Testing analysis began producing impact data that grew steadily until it was unusually large. Between 13:15 and 14:30, our system scaled the workers that apply these updates. The additional workers added load to the database, and between 13:15 and 14:00, database CPU rose to 96-98%. At 13:20, the workers began timing out on these updates and retrying them repeatedly. From then on, impact data updates were delayed, so test selection used older data and some jobs could run more tests than necessary.
At 14:12, our engineering team was alerted to delays in processing impact data, and at 14:17 we began investigating. At 14:33, we opened an incident. At first, the impact appeared limited to delayed updates.
At 15:07, we found that the impact data being applied was far larger than normal and was driving the load on the database. By 15:17, we projected that the database would use up its disk throughput burst balance in about an hour and that free storage would run out in about two.
We took three steps to remove the source of the load. At 15:27, we blocked the analysis that was producing this data, and at 15:28, we cancelled the job running it. From 15:50, some test jobs using Smarter Testing began to fail intermittently. At 15:51, we dropped the queued updates for this data so the workers would stop retrying them.
We also considered moving the database to a larger instance type. That change requires a restart and a short outage and at that point, the impact appeared limited to delays in impact data, so we opted to remove the load first. At 16:05, we increased storage because free space was falling quickly. That change put the database into a modifying state, which blocked the instance type change until 18:18.
At 16:20, the database used up its disk throughput burst balance and stopped responding. Jobs using Smarter Testing failed. We had no way to switch Smarter Testing off for customers, so we focused on cutting load. Jobs recovered at 16:38. We scaled the workers to zero (16:41) and we deleted the remaining queued updates for this data (16:42), and a single restarted worker had cleared the backlog by 16:58.
At 17:00, database CPU rose again, to 65-82%, while the burst balance was still exhausted. To reduce load, we reduced batch sizes and ran a single worker. By 17:34, we had concluded that only an instance type without burst limits would fix this, but the storage change still had the database in a modifying state. At 17:37, the database stopped responding again, and test jobs using Smarter Testing failed. At 17:53, it failed over to its standby instance, which restored its burst balance, and test jobs were running again by 17:55.
From 16:20 to 18:00, some requests for selection, timing and parallelism data timed out. During that time, dynamic test splitting may have used older timing data, and jobs using dynamic parallelism ran on their maximum number of nodes.
When the storage change completed at 18:18, we announced emergency maintenance, and at 18:50 we moved the database to a larger instance type without burst limits, with about 45 seconds of downtime. We scaled the workers back up to clear the backlog, and we resolved the incident at 19:19.
Future Prevention and Process Improvement
Changes already in place:
A larger database. We moved the Smarter Testing database to an instance type that does not rely on burst capacity, which was the limit that caused both outages. We also increased its storage and tuned its background cleanup. Database CPU fell afterward and we are monitoring it closely.
We made test jobs resilient to Smarter Testing outages. Since the incident, we added a fallback that our engineering team can turn on if Smarter Testing becomes unavailable. With the fallback on, jobs continue to run their tests without Smarter Testing features, rather than failing.
Further improvements
Stronger safeguards against unusually large data. We are adding protections so that unusually large data is limited and we are improving how our system handles failed updates.
We are improving database performance. We are optimizing the queries our system runs against the Smarter Testing database, so it can handle more load.
Warning us sooner. We are adding earlier alerts on database capacity.
Conclusion
Since the incident, the Smarter Testing database has been running on infrastructure with substantially more resources and no burst limits, and it has been stable since the change. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve this incident.
Please reach out to our support team with any questions or concerns.
Resolved: Smarter Testing functionality is restored.
We appreciate your patience as we worked through this.
Monitoring: The maintenance was successful. Smarter Testing functionality has been restored.
We are monitoring to ensure performance remains stable.
Identified: At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail.
Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:50 UTC
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:30 UTC.
Identified: Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing.
Thank you for your patience while our engineers work to resolve this.
Identified: Since 17:09 UTC, we have been seeing delays in updating test impact analysis data.
Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
Identified: Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
Investigating: Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
CircleCI macOS Jobs
Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Postmortem: ## Summary
(All times UTC)
From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected.
The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding.
In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19.
Customers do not need to run a full analysis. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data.
The original status page can be found here: Delays and intermittent failures running Smarter Testing.
Background
Smarter Testing uses data stored for each project to decide which tests to run (test impact analysis), how to split them across parallel nodes, and how many nodes a job needs. To decide which tests to run, Smarter Testing uses impact data, a map of which files each test touches, and this was the part of the system most affected by this incident. When a job runs, Smarter Testing records updates to this data, and our system applies them to a database in the background.
During a test job, Smarter Testing contacts our backend service, which reads from that database. If that service is unavailable, test jobs using Smarter Testing cannot run their tests.
The size of a project’s impact data depends on how many tests it has and how many files each test touches. Unusually large impact data can place heavy load on the database, and our safeguards did not stop this.
What Happened
(All times UTC)
At 12:12 on November 19, a Smarter Testing analysis began producing impact data that grew steadily until it was unusually large. Between 13:15 and 14:30, our system scaled the workers that apply these updates. The additional workers added load to the database, and between 13:15 and 14:00, database CPU rose to 96-98%. At 13:20, the workers began timing out on these updates and retrying them repeatedly. From then on, impact data updates were delayed, so test selection used older data and some jobs could run more tests than necessary.
At 14:12, our engineering team was alerted to delays in processing impact data, and at 14:17 we began investigating. At 14:33, we opened an incident. At first, the impact appeared limited to delayed updates.
At 15:07, we found that the impact data being applied was far larger than normal and was driving the load on the database. By 15:17, we projected that the database would use up its disk throughput burst balance in about an hour and that free storage would run out in about two.
We took three steps to remove the source of the load. At 15:27, we blocked the analysis that was producing this data, and at 15:28, we cancelled the job running it. From 15:50, some test jobs using Smarter Testing began to fail intermittently. At 15:51, we dropped the queued updates for this data so the workers would stop retrying them.
We also considered moving the database to a larger instance type. That change requires a restart and a short outage and at that point, the impact appeared limited to delays in impact data, so we opted to remove the load first. At 16:05, we increased storage because free space was falling quickly. That change put the database into a modifying state, which blocked the instance type change until 18:18.
At 16:20, the database used up its disk throughput burst balance and stopped responding. Jobs using Smarter Testing failed. We had no way to switch Smarter Testing off for customers, so we focused on cutting load. Jobs recovered at 16:38. We scaled the workers to zero (16:41) and we deleted the remaining queued updates for this data (16:42), and a single restarted worker had cleared the backlog by 16:58.
At 17:00, database CPU rose again, to 65-82%, while the burst balance was still exhausted. To reduce load, we reduced batch sizes and ran a single worker. By 17:34, we had concluded that only an instance type without burst limits would fix this, but the storage change still had the database in a modifying state. At 17:37, the database stopped responding again, and test jobs using Smarter Testing failed. At 17:53, it failed over to its standby instance, which restored its burst balance, and test jobs were running again by 17:55.
From 16:20 to 18:00, some requests for selection, timing and parallelism data timed out. During that time, dynamic test splitting may have used older timing data, and jobs using dynamic parallelism ran on their maximum number of nodes.
When the storage change completed at 18:18, we announced emergency maintenance, and at 18:50 we moved the database to a larger instance type without burst limits, with about 45 seconds of downtime. We scaled the workers back up to clear the backlog, and we resolved the incident at 19:19.
Future Prevention and Process Improvement
Changes already in place:
A larger database. We moved the Smarter Testing database to an instance type that does not rely on burst capacity, which was the limit that caused both outages. We also increased its storage and tuned its background cleanup. Database CPU fell afterward and we are monitoring it closely.
We made test jobs resilient to Smarter Testing outages. Since the incident, we added a fallback that our engineering team can turn on if Smarter Testing becomes unavailable. With the fallback on, jobs continue to run their tests without Smarter Testing features, rather than failing.
Further improvements
Stronger safeguards against unusually large data. We are adding protections so that unusually large data is limited and we are improving how our system handles failed updates.
We are improving database performance. We are optimizing the queries our system runs against the Smarter Testing database, so it can handle more load.
Warning us sooner. We are adding earlier alerts on database capacity.
Conclusion
Since the incident, the Smarter Testing database has been running on infrastructure with substantially more resources and no burst limits, and it has been stable since the change. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve this incident.
Please reach out to our support team with any questions or concerns.
Resolved: Smarter Testing functionality is restored.
We appreciate your patience as we worked through this.
Monitoring: The maintenance was successful. Smarter Testing functionality has been restored.
We are monitoring to ensure performance remains stable.
Identified: At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail.
Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:50 UTC
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:30 UTC.
Identified: Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing.
Thank you for your patience while our engineers work to resolve this.
Identified: Since 17:09 UTC, we have been seeing delays in updating test impact analysis data.
Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
Identified: Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
Investigating: Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
CircleCI Windows Jobs
Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.
Postmortem: ## Summary
(All times UTC)
From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected.
The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding.
In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19.
Customers do not need to run a full analysis. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data.
The original status page can be found here: Delays and intermittent failures running Smarter Testing.
Background
Smarter Testing uses data stored for each project to decide which tests to run (test impact analysis), how to split them across parallel nodes, and how many nodes a job needs. To decide which tests to run, Smarter Testing uses impact data, a map of which files each test touches, and this was the part of the system most affected by this incident. When a job runs, Smarter Testing records updates to this data, and our system applies them to a database in the background.
During a test job, Smarter Testing contacts our backend service, which reads from that database. If that service is unavailable, test jobs using Smarter Testing cannot run their tests.
The size of a project’s impact data depends on how many tests it has and how many files each test touches. Unusually large impact data can place heavy load on the database, and our safeguards did not stop this.
What Happened
(All times UTC)
At 12:12 on November 19, a Smarter Testing analysis began producing impact data that grew steadily until it was unusually large. Between 13:15 and 14:30, our system scaled the workers that apply these updates. The additional workers added load to the database, and between 13:15 and 14:00, database CPU rose to 96-98%. At 13:20, the workers began timing out on these updates and retrying them repeatedly. From then on, impact data updates were delayed, so test selection used older data and some jobs could run more tests than necessary.
At 14:12, our engineering team was alerted to delays in processing impact data, and at 14:17 we began investigating. At 14:33, we opened an incident. At first, the impact appeared limited to delayed updates.
At 15:07, we found that the impact data being applied was far larger than normal and was driving the load on the database. By 15:17, we projected that the database would use up its disk throughput burst balance in about an hour and that free storage would run out in about two.
We took three steps to remove the source of the load. At 15:27, we blocked the analysis that was producing this data, and at 15:28, we cancelled the job running it. From 15:50, some test jobs using Smarter Testing began to fail intermittently. At 15:51, we dropped the queued updates for this data so the workers would stop retrying them.
We also considered moving the database to a larger instance type. That change requires a restart and a short outage and at that point, the impact appeared limited to delays in impact data, so we opted to remove the load first. At 16:05, we increased storage because free space was falling quickly. That change put the database into a modifying state, which blocked the instance type change until 18:18.
At 16:20, the database used up its disk throughput burst balance and stopped responding. Jobs using Smarter Testing failed. We had no way to switch Smarter Testing off for customers, so we focused on cutting load. Jobs recovered at 16:38. We scaled the workers to zero (16:41) and we deleted the remaining queued updates for this data (16:42), and a single restarted worker had cleared the backlog by 16:58.
At 17:00, database CPU rose again, to 65-82%, while the burst balance was still exhausted. To reduce load, we reduced batch sizes and ran a single worker. By 17:34, we had concluded that only an instance type without burst limits would fix this, but the storage change still had the database in a modifying state. At 17:37, the database stopped responding again, and test jobs using Smarter Testing failed. At 17:53, it failed over to its standby instance, which restored its burst balance, and test jobs were running again by 17:55.
From 16:20 to 18:00, some requests for selection, timing and parallelism data timed out. During that time, dynamic test splitting may have used older timing data, and jobs using dynamic parallelism ran on their maximum number of nodes.
When the storage change completed at 18:18, we announced emergency maintenance, and at 18:50 we moved the database to a larger instance type without burst limits, with about 45 seconds of downtime. We scaled the workers back up to clear the backlog, and we resolved the incident at 19:19.
Future Prevention and Process Improvement
Changes already in place:
A larger database. We moved the Smarter Testing database to an instance type that does not rely on burst capacity, which was the limit that caused both outages. We also increased its storage and tuned its background cleanup. Database CPU fell afterward and we are monitoring it closely.
We made test jobs resilient to Smarter Testing outages. Since the incident, we added a fallback that our engineering team can turn on if Smarter Testing becomes unavailable. With the fallback on, jobs continue to run their tests without Smarter Testing features, rather than failing.
Further improvements
Stronger safeguards against unusually large data. We are adding protections so that unusually large data is limited and we are improving how our system handles failed updates.
We are improving database performance. We are optimizing the queries our system runs against the Smarter Testing database, so it can handle more load.
Warning us sooner. We are adding earlier alerts on database capacity.
Conclusion
Since the incident, the Smarter Testing database has been running on infrastructure with substantially more resources and no burst limits, and it has been stable since the change. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve this incident.
Please reach out to our support team with any questions or concerns.
Resolved: Smarter Testing functionality is restored.
We appreciate your patience as we worked through this.
Monitoring: The maintenance was successful. Smarter Testing functionality has been restored.
We are monitoring to ensure performance remains stable.
Identified: At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail.
Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:50 UTC
Identified: Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again.
Dynamic Test Splitting is still not running optimally.
Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes.
Thank you for your patience while our engineers work to resolve this.
We will provide another update by 18:30 UTC.
Identified: Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing.
Thank you for your patience while our engineers work to resolve this.
Identified: Since 17:09 UTC, we have been seeing delays in updating test impact analysis data.
Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
Identified: Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
Investigating: Some Smarter Testing customers may not be able to run tests.
We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.