GitHub Status

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Actions is operating normally.

Investigating: Pull Requests is operating normally.

Investigating: Issues is operating normally.

Investigating: Webhooks is operating normally.

Investigating: Git Operations is operating normally.

Investigating: We have fully mitigated this issue and all services are operating normally.

Investigating: Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We have identified a change that was made in the Pull Request area for GitHub. Users may be unable to use certain pull request and issues features and may see some webhooks impacted. We have identified the issue, taken mitigation and are starting to see recovery but will continue to monitor and post updates as we have them.

Investigating: Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating: Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for Issues and Webhooks

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Actions is operating normally.

Investigating: Pull Requests is operating normally.

Investigating: Issues is operating normally.

Investigating: Webhooks is operating normally.

Investigating: Git Operations is operating normally.

Investigating: We have fully mitigated this issue and all services are operating normally.

Investigating: Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We have identified a change that was made in the Pull Request area for GitHub. Users may be unable to use certain pull request and issues features and may see some webhooks impacted. We have identified the issue, taken mitigation and are starting to see recovery but will continue to monitor and post updates as we have them.

Investigating: Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating: Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for Issues and Webhooks

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Actions is operating normally.

Investigating: Pull Requests is operating normally.

Investigating: Issues is operating normally.

Investigating: Webhooks is operating normally.

Investigating: Git Operations is operating normally.

Investigating: We have fully mitigated this issue and all services are operating normally.

Investigating: Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We have identified a change that was made in the Pull Request area for GitHub. Users may be unable to use certain pull request and issues features and may see some webhooks impacted. We have identified the issue, taken mitigation and are starting to see recovery but will continue to monitor and post updates as we have them.

Investigating: Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating: Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for Issues and Webhooks

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Actions is operating normally.

Investigating: Pull Requests is operating normally.

Investigating: Issues is operating normally.

Investigating: Webhooks is operating normally.

Investigating: Git Operations is operating normally.

Investigating: We have fully mitigated this issue and all services are operating normally.

Investigating: Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We have identified a change that was made in the Pull Request area for GitHub. Users may be unable to use certain pull request and issues features and may see some webhooks impacted. We have identified the issue, taken mitigation and are starting to see recovery but will continue to monitor and post updates as we have them.

Investigating: Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating: Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for Issues and Webhooks

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Pull Requests is operating normally.

Investigating: We continue to investigate issues with PRs. Impact remains limited to less than 2% of users.

Investigating: We continue to investigate issues with PRs impacting less than 2% of customers.

Investigating: We continue to investigate issues with PRs impacting less than 2% of customers.

Investigating: We're seeing issues related to PR are investigating. Less than 2% of users are impacted.

Investigating: We are investigating reports of degraded performance for Pull Requests

Resolved: At 15:33 UTC on August 5, 2025, we initiated a production database migration to drop a column from a table backing pull request functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column in a subset of pull request queries. As a result, there were elevated error rates across pushes, webhooks, notifications, and pull requests with impact peaking at approximately 4% of all web and REST API traffic.

We mitigated the issue by deploying a change that instructed the ORM to ignore the removed column. Most affected services recovered by 16:13 UTC. However, that fix was applied only to our largest production environment. An update to some of our custom and canary environments did not pick up the fix and this triggered a secondary incident affecting ~0.1% of pull request traffic, which was fully resolved by 19:45 UTC.

While migrations have protections such as progressive roll-out first targeting validation environments and acknowledge gates, this incident identified an application monitoring gap that would have prevented continued rollout when impact was observed. We will add additional automation and safeguards to prevent future incidents without requiring human intervention. We are also already working on a way to streamline some types of changes across environments, which would have prevented the second incident from occurring.

Investigating: Actions is operating normally.

Investigating: Pull Requests is operating normally.

Investigating: Issues is operating normally.

Investigating: Webhooks is operating normally.

Investigating: Git Operations is operating normally.

Investigating: We have fully mitigated this issue and all services are operating normally.

Investigating: Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We have identified a change that was made in the Pull Request area for GitHub. Users may be unable to use certain pull request and issues features and may see some webhooks impacted. We have identified the issue, taken mitigation and are starting to see recovery but will continue to monitor and post updates as we have them.

Investigating: Webhooks is experiencing degraded availability. We are continuing to investigate.

Investigating: Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating: Actions is experiencing degraded performance. We are continuing to investigate.

Investigating: Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating: We are investigating reports of degraded performance for Issues and Webhooks