Fly.io Status

Resolved: This incident has been resolved. All platform and API operations are working normally.

Monitoring: API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery.

Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully.

Sprite and MPG cluster creations are processing as normal.

Identified: A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work.

We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.

Identified: An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time.

We are continuing to work on restoring full availability.

Identified: We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands.

We have identified the cause of the issue and are continuing to work on a fix.

Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.

Identified: New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the increased latency and are working on a fix.

The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an app|machine not found error

Investigating: We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We're currently investigating issues with the Machines API. Customer deployments and the Fly dashboard may be affected.

Resolved: This incident has been resolved.

Investigating: We are currently investigating this issue. The page currently displays: "We’re having trouble loading the cost breakdown."

Resolved: This incident was caused by a failed Redis node that powers our GraphQL API. We were able to recreate the Redis node and restore service.

We are still investigating the root cause of the failure. In the mean time, all API endpoints now appear to be stable and errors have dropped to baseline level.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We're investigating elevated GraphQL errors that affect some API endpoints.

Resolved: This incident has been resolved. All platform and API operations are working normally.

Monitoring: API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery.

Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully.

Sprite and MPG cluster creations are processing as normal.

Identified: A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work.

We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.

Identified: An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time.

We are continuing to work on restoring full availability.

Identified: We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands.

We have identified the cause of the issue and are continuing to work on a fix.

Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.

Identified: New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the increased latency and are working on a fix.

The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an app|machine not found error

Investigating: We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We're currently investigating issues with the Machines API. Customer deployments and the Fly dashboard may be affected.

Resolved: This incident has been resolved.

Monitoring: We're continuing to monitor after having added more capacity to our DFW and IAD regions.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We have added additional capacity in DFW and IAD regions and are monitoring the impact.

New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.

Identified: New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.

Identified: We have identified the problem and are working on a fix.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We're continuing to monitor after having added more capacity to our DFW and IAD regions.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We have added additional capacity in DFW and IAD regions and are monitoring the impact.

New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.

Identified: New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.

Identified: We have identified the problem and are working on a fix.

Resolved: This incident has been resolved.

Monitoring: We have provisioned additional capacity in dfw and iad and are monitoring to ensure machine and builder starts are succeeding consistently.

Identified: These regions (Dallas, TX dfw and Ashburn, VA iad) are currently low on capacity. New machine creates in these regions might fail temporarily, and Depot builders may be unavailable, causing deploys to hang in "Waiting for Depot builder". If you are having issues with Depot builders, consider moving them to a different non-iad, non-dfw region in your fly.io dashboard's "Settings" page under "App builders", or try --depot=false.

Resolved: This incident has been resolved. All platform and API operations are working normally.

Monitoring: API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery.

Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully.

Sprite and MPG cluster creations are processing as normal.

Identified: A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work.

We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.

Identified: An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time.

We are continuing to work on restoring full availability.

Identified: We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands.

We have identified the cause of the issue and are continuing to work on a fix.

Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.

Identified: New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the increased latency and are working on a fix.

The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an app|machine not found error

Investigating: We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We're currently investigating issues with the Machines API. Customer deployments and the Fly dashboard may be affected.

Resolved: This incident was caused by a failed Redis node that powers our GraphQL API. We were able to recreate the Redis node and restore service.

We are still investigating the root cause of the failure. In the mean time, all API endpoints now appear to be stable and errors have dropped to baseline level.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We're investigating elevated GraphQL errors that affect some API endpoints.

Resolved: This incident has been resolved.

Monitoring: We're continuing to monitor after having added more capacity to our DFW and IAD regions.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We have added additional capacity in DFW and IAD regions and are monitoring the impact.

New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.

Identified: New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.

Identified: We have identified the problem and are working on a fix.

Resolved: This incident has been resolved. All platform and API operations are working normally.

Monitoring: API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery.

Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully.

Sprite and MPG cluster creations are processing as normal.

Identified: A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work.

We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.

Identified: An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time.

We are continuing to work on restoring full availability.

Identified: We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands.

We have identified the cause of the issue and are continuing to work on a fix.

Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.

Identified: New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the increased latency and are working on a fix.

The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an app|machine not found error

Investigating: We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.

Resolved: This incident has been resolved.

Monitoring: We're continuing to monitor after having added more capacity to our DFW and IAD regions.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We have added additional capacity in DFW and IAD regions and are monitoring the impact.

New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Identified: We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.

Identified: New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.

Identified: We have identified the problem and are working on a fix.

Resolved: This incident has been resolved. All platform and API operations are working normally.

Monitoring: API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery.

Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully.

Sprite and MPG cluster creations are processing as normal.

Identified: A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work.

We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.

Identified: An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time.

We are continuing to work on restoring full availability.

Identified: We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands.

We have identified the cause of the issue and are continuing to work on a fix.

Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.

Identified: New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the increased latency and are working on a fix.

The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an app|machine not found error

Investigating: We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.