GitHub Status

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations

Resolved: On May 27, 2025, between 09:31 UTC and 13:31 UTC, some Actions jobs experienced failures uploading to and downloading from the Actions Cache service. During the incident, 6% of all workflow runs couldn’t upload or download cache entries from the service, resulting in a non-blocking warning message in the logs and performance degradation. The disruption was caused by an infrastructure update related to the retirement of a legacy service, which unintentionally impacted Cache service availability. We resolved the incident by reverting the change and have since implemented a permanent fix to prevent recurrence.

We are improving our configuration change processes by introducing additional end-to-end tests to cover the identified gaps, and implementing deployment pipeline improvements to reduce mitigation time for similar issues in the future.

Investigating: Mitigation is applied and we’re seeing signs of recovery. We’re monitoring the situation until the mitigation is applied to all affected repositories.

Investigating: We are experiencing degradation with the GitHub Actions cache service and are working on applying the appropriate mitigations.

Investigating: We are investigating reports of degraded performance for Actions

Resolved: On May 28, 2025, from approximately 09:45 UTC to 14:45 UTC, GitHub Actions experienced delayed job starts for workflows in public repos using Ubuntu-24 standard hosted runners. This was caused by a misconfiguration in backend caching behavior after a failover, which led to duplicate job assignments and reduced available capacity. Approximately 19.7% of Ubuntu-24 hosted runner jobs on public repos were delayed. Other hosted runners, self-hosted runners, and private repo workflows were unaffected.

By 12:45 UTC, we mitigated the issue by redeploying backend components to reset state and scaling up available resources to more quickly work through the backlog of queued jobs. We are working to improve our deployment and failover resiliency and validation to reduce the likelihood of similar issues in the future.

Investigating: We are continuing to monitor the affected Actions runners to ensure a smooth recovery.

Investigating: We are observing indications of recovery with the affected Actions runners.

The team will continue monitoring systems to ensure a return to normal service.

Investigating: We're continuing to investigate delays in Actions runners for hosted Ubuntu 24.

We will provide further updates as more information becomes available.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing high wait times for obtaining standard hosted runners for ubuntu 24. Other hosted labels and self-hosted runners are not impacted.

Investigating: We are currently investigating this issue.

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations

Resolved: On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.

On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.

In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.

We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.

We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.

Investigating: We continue to see signs of recovery.

Investigating: Issues is operating normally.

Investigating: Git Operations is operating normally.

Investigating: API Requests is operating normally.

Investigating: Copilot is operating normally.

Investigating: Packages is operating normally.

Investigating: Actions is operating normally.

Investigating: Packages is experiencing degraded performance. We are continuing to investigate.

Investigating: Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: We are continuing to investigate degraded performance.

Investigating: Issues is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for API Requests and Git Operations