Coveralls Status

Postmortem: ### Resolved: elevated error rates, Monday September 28, 08:00 to 09:15 PT

Timeline: Between about 08:00 and 09:12 PST on September 28, some requests to Coveralls returned HTTP 500 errors. Coverage uploads, GitHub webhook events, badges, and coverage pages were affected. Failures were intermittent at first and peaked toward the end of the window. Service was fully restored at 09:12 PST.

Cause: a capacity issue with our primary database. We have have added capacity, enabled automatic scaling, and changed our monitoring triggers to be alerted well before this kind of issue can recur.

What to do: If you were affected: re-run any CI job that uploaded coverage to Coveralls between 08:00 and 09:15 PT on September 28, or simply push a new commit. No data was lost or corrupted; those uploads simply did not land.

We are sorry for the disruption.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Operational in all regions. We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.

Monitoring: Users in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST.

Behavior: PRs opened between 2am-6am PST may have slow or missing Coveralls PR comments or status checks.

We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next ~1hr.

Postmortem: ### Resolved: elevated error rates, Monday September 28, 08:00 to 09:15 PT

Timeline: Between about 08:00 and 09:12 PST on September 28, some requests to Coveralls returned HTTP 500 errors. Coverage uploads, GitHub webhook events, badges, and coverage pages were affected. Failures were intermittent at first and peaked toward the end of the window. Service was fully restored at 09:12 PST.

Cause: a capacity issue with our primary database. We have have added capacity, enabled automatic scaling, and changed our monitoring triggers to be alerted well before this kind of issue can recur.

What to do: If you were affected: re-run any CI job that uploaded coverage to Coveralls between 08:00 and 09:15 PT on September 28, or simply push a new commit. No data was lost or corrupted; those uploads simply did not land.

We are sorry for the disruption.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Operational in all regions. We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.

Monitoring: Users in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST.

Behavior: PRs opened between 2am-6am PST may have slow or missing Coveralls PR comments or status checks.

We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next ~1hr.