Coveralls Status

Monitoring: Mitigation in place.

All systems operational.

This morning we deployed additional capacity and autoscaling measures to reduce 504 errors on coverage report uploads:

• Doubled our web server fleet (on top of the prior doubling when this issue began).
• Enabled autoscaling at the web layer, allowing the fleet to double again automatically when NGINX response times exceed thresholds.

The underlying trigger remains rare surges of upload requests from outlier repositories (750–1250 uploads per build). While we have paused processing for these repos, our HTTP servers must still handle the incoming requests until they stop.

Timezone coverage: As a small team based in Los Angeles (PDT), our ability to respond in real time is most limited overnight (10p–6a PDT). Unfortunately, the primary outlier repos are in APAC, making this the window of highest risk. With these changes, we hope to reduce the occurrence of upload 504s during this window.

We will monitor results closely and continue tuning autoscaling thresholds. Please let us know if you continue to see 504 errors on uploads.

Monitoring: All systems operational.

Earlier today (6:45–7:45 AM PDT), we received elevated reports of 504 timeout errors. We have not been able to reproduce the issue since, but if you are still experiencing errors, please contact us at [email protected].

The affected areas may include:

• Coverage Report Uploads (/api/v1/jobs)
• Add Repos Page
• Repo Page
• Source File Page

Fixes for the Add Repos, Repo, and Source File pages are scheduled to be deployed by end of day (PDT).

Monitoring: All systems operational.

We have released one of two parts of a near-term solution into production resolving a minority subset of 504 errors. We are still working on releasing part two into production.

Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: All systems operational.

Continuing to keep this open until we have released our short-term fix into production.

Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: We are still working on a near-term fix. We will post here, and here when complete: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: We’re currently seeing elevated reports of 504 Timeout errors affecting some customers on a subset of Coveralls pages, including:

• Source File pages
• Repo pages
• Add Repos pages

All systems and pages are generally operational; a subset of customers are experiencing these errors, sometimes intermittently.

There is a public tracking issue for the Source File timeout errors here: https://github.com/lemurheavy/coveralls-public/issues/1757

Fix in progress: We’re implementing a short-term fix over the next 24–48 hours, which should eliminate the timeouts.

A longer-term fix is also planned, but will roll out over several weeks, but early phases of that implementation should also reduce the request times that were originally triggering the 504 timeouts.

What you can do: If you're currently affected, we recommend following updates here, and subscribing to the public issue: https://github.com/lemurheavy/coveralls-public/issues/1757

If your issue pattern differs from above, or you suspect a different root cause, reach out to [email protected], and we'll verify for you.

Monitoring: Mitigation in place.

All systems operational.

This morning we deployed additional capacity and autoscaling measures to reduce 504 errors on coverage report uploads:

• Doubled our web server fleet (on top of the prior doubling when this issue began).
• Enabled autoscaling at the web layer, allowing the fleet to double again automatically when NGINX response times exceed thresholds.

The underlying trigger remains rare surges of upload requests from outlier repositories (750–1250 uploads per build). While we have paused processing for these repos, our HTTP servers must still handle the incoming requests until they stop.

Timezone coverage: As a small team based in Los Angeles (PDT), our ability to respond in real time is most limited overnight (10p–6a PDT). Unfortunately, the primary outlier repos are in APAC, making this the window of highest risk. With these changes, we hope to reduce the occurrence of upload 504s during this window.

We will monitor results closely and continue tuning autoscaling thresholds. Please let us know if you continue to see 504 errors on uploads.

Monitoring: All systems operational.

Earlier today (6:45–7:45 AM PDT), we received elevated reports of 504 timeout errors. We have not been able to reproduce the issue since, but if you are still experiencing errors, please contact us at [email protected].

The affected areas may include:

• Coverage Report Uploads (/api/v1/jobs)
• Add Repos Page
• Repo Page
• Source File Page

Fixes for the Add Repos, Repo, and Source File pages are scheduled to be deployed by end of day (PDT).

Monitoring: All systems operational.

We have released one of two parts of a near-term solution into production resolving a minority subset of 504 errors. We are still working on releasing part two into production.

Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: All systems operational.

Continuing to keep this open until we have released our short-term fix into production.

Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: We are still working on a near-term fix. We will post here, and here when complete: https://github.com/lemurheavy/coveralls-public/issues/1757

Monitoring: We’re currently seeing elevated reports of 504 Timeout errors affecting some customers on a subset of Coveralls pages, including:

• Source File pages
• Repo pages
• Add Repos pages

All systems and pages are generally operational; a subset of customers are experiencing these errors, sometimes intermittently.

There is a public tracking issue for the Source File timeout errors here: https://github.com/lemurheavy/coveralls-public/issues/1757

Fix in progress: We’re implementing a short-term fix over the next 24–48 hours, which should eliminate the timeouts.

A longer-term fix is also planned, but will roll out over several weeks, but early phases of that implementation should also reduce the request times that were originally triggering the 504 timeouts.

What you can do: If you're currently affected, we recommend following updates here, and subscribing to the public issue: https://github.com/lemurheavy/coveralls-public/issues/1757

If your issue pattern differs from above, or you suspect a different root cause, reach out to [email protected], and we'll verify for you.