Fly.io Status

Resolved: This issue is now resolved.

Monitoring: We have successfully run a fix to re-sync our global and regional state stores in order to bring machines back to a healthy state, and we're monitoring the situation to confirm that there are no more issues.

Identified: Machines created after the delayed machine registration incident (https://status.flyio.net/incidents/3npj6935byt4) may have incomplete networking configurations and could be unable to receive traffic. We've identified the issue and are deploying code fixes as well as updating created machines. This affects a small number of machines on all our regions.

Resolved: This incident has been resolved.

Monitoring: A fix has been deployed across all impacted hosts. We are seeing a sharp reduction in Token errors since 20:00 UTC and other metrics are recovering as well. We are continuing to monitor closely

Identified: We saw some improvement from the previous fix, however errors remained elevated on some hosts. We have identified the root cause of the remaining errors as a communication issue between the hosts and our Token database. We are preparing a fix that should resolve these.

Identified: We have rolled out an initial fix for the token issues and are monitoring for improvements.

Investigating: While Machine registration error rates have improved, we are now seeing elevated error rates verifying user tokens during some actions.

Users may see errors like "failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens" when deploying or creating machines.

We are investigating

Identified: A fix has been rolled out and most hosts are registering machines as normal. A few hosts remain with elevated error rates, we are continuing to fix these. Users who experience an error creating or deploying a new machine should re-try the operation.

Identified: We have identified elevated error rates registering new machines with our global state tracking service on some hosts. We have identified the issue and are deploying a fix.

Users may have seen elevated machine create, start, or deployment failures over the past ~20 minutes.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are seeing Machines API connectivity improve in APAC regions. We continue monitoring for full recovery.

Identified: The issue has been identified and we are seeing Machines API performance improve in most regions since ~16:20 UTC.

Machines API calls in the SYD, NRT, SIN region may continue to see 5xx errors or higher latency at this time. We are continuing to work on restoring full API performance in all regions

Investigating: We are investigating widespread Machines API issues since 16:00 UTC. You may experience 5xx errors or higher latency at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are still working on restoring the MPG clusters. Most of them should be operational already.

Identified: Affected hosts are starting to come back online. We are working on restoring affected MPG clusters.

Identified: One of our upstream providers is experiencing a major power issue in their AMS datacenter. Managed Postgres instances in AMS are experiencing an outage as our control plane for Managed Postgres is taken down by the incident.

Investigating: One of our upstream providers is performing an emergency DC maintenance. You may see degraded connectivity on some of your apps in AMS. Most apps in AMS are not affected.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are seeing private networking / certificates in SYD improving. We are continuing to monitor for full recovery.

Investigating: Private Networking (6PN) is degraded in SYD region. Communication between Machines in SYD region and Machines in other regions may fail at this time. Newly created Machines in SYD may fail to sync to other regions (may not show up in Machines API List endpoint, or state may be incorrect). We are working with our upstream providers to resolve this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating communication issues between some SYD (Sydney, Australia) region hosts and our Certificate vault.

Requests hitting SYD region edges may see issues resolving certificates at this time, especially for newly issued or not recently used certificates.

Resolved: This incident has been resolved.

Monitoring: A fix has been deployed across all impacted hosts. We are seeing a sharp reduction in Token errors since 20:00 UTC and other metrics are recovering as well. We are continuing to monitor closely

Identified: We saw some improvement from the previous fix, however errors remained elevated on some hosts. We have identified the root cause of the remaining errors as a communication issue between the hosts and our Token database. We are preparing a fix that should resolve these.

Identified: We have rolled out an initial fix for the token issues and are monitoring for improvements.

Investigating: While Machine registration error rates have improved, we are now seeing elevated error rates verifying user tokens during some actions.

Users may see errors like "failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens" when deploying or creating machines.

We are investigating

Identified: A fix has been rolled out and most hosts are registering machines as normal. A few hosts remain with elevated error rates, we are continuing to fix these. Users who experience an error creating or deploying a new machine should re-try the operation.

Identified: We have identified elevated error rates registering new machines with our global state tracking service on some hosts. We have identified the issue and are deploying a fix.

Users may have seen elevated machine create, start, or deployment failures over the past ~20 minutes.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: Some new Fly.io users may encounter an "upgrade your organization" error message when attempting to deploy apps for the first time. We're currently working with Depot to figure out what's causing the issue.

In the meantime, you should be able to work around the issue by using Fly builders with fly deploy --depot=false.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are seeing private networking / certificates in SYD improving. We are continuing to monitor for full recovery.

Investigating: Private Networking (6PN) is degraded in SYD region. Communication between Machines in SYD region and Machines in other regions may fail at this time. Newly created Machines in SYD may fail to sync to other regions (may not show up in Machines API List endpoint, or state may be incorrect). We are working with our upstream providers to resolve this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating communication issues between some SYD (Sydney, Australia) region hosts and our Certificate vault.

Requests hitting SYD region edges may see issues resolving certificates at this time, especially for newly issued or not recently used certificates.

Resolved: All MPG clusters are back to full, normal operations.

Monitoring: All MPG clusters are reachable.

Identified: We are still continuing cleanup on some clusters.

Identified: All cluster primary and pgBouncer machines are now healthy and operating normally.

We are still continuing cleanup on some clusters with lagging or degraded replicas, but this should not impact writes or reads to clusters.

Identified: We are continuing to work on restoring all clusters to full health.

Identified: With the underlying incident stabilizing (https://status.flyio.net/incidents/3npj6935byt4) we are seeing improvements amongst impacted clusters. We continue to work on restoring all clusters to full health.

Identified: A number of clusters in IAD, AMS, and SIN regions continue to see degraded replicas and PGBouncers at this time. A smaller number of clusters in these regions are also seeing disruption to their primaries. We continue to work on restoring full cluster health in all regions.

Identified: A small number of MPG clusters in the AMS and IAD region are currently in degraded states due to downstream impact from this Machines API issue: https://status.flyio.net/incidents/3npj6935byt4

Most of the impacted clusters may see a degraded replica or PG Bouncer in their statuspage. A very small number may be unable to connect to their MPG primary node, the team is working to restore connectivity as the top priority. Users may also see delays registering new clusters in these regions at this time.

Resolved: This incident has been resolved.

Monitoring: We've rolled out a fix for the remaining impacted clusters, and we're now monitoring the results.

Identified: We've rolled out a fix for some additional impacted clusters, and we're continuing to work on the remaining clusters.

Identified: We've identified the issue - some MPG clusters in IAD should be seeing improvements, and we're working on rolling out a fix for the remaining impacted clusters.

Investigating: We're currently looking into an issue with MPG clusters in the IAD region.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are seeing private networking / certificates in SYD improving. We are continuing to monitor for full recovery.

Investigating: Private Networking (6PN) is degraded in SYD region. Communication between Machines in SYD region and Machines in other regions may fail at this time. Newly created Machines in SYD may fail to sync to other regions (may not show up in Machines API List endpoint, or state may be incorrect). We are working with our upstream providers to resolve this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating communication issues between some SYD (Sydney, Australia) region hosts and our Certificate vault.

Requests hitting SYD region edges may see issues resolving certificates at this time, especially for newly issued or not recently used certificates.

Resolved: All MPG clusters are back to full, normal operations.

Monitoring: All MPG clusters are reachable.

Identified: We are still continuing cleanup on some clusters.

Identified: All cluster primary and pgBouncer machines are now healthy and operating normally.

We are still continuing cleanup on some clusters with lagging or degraded replicas, but this should not impact writes or reads to clusters.

Identified: We are continuing to work on restoring all clusters to full health.

Identified: With the underlying incident stabilizing (https://status.flyio.net/incidents/3npj6935byt4) we are seeing improvements amongst impacted clusters. We continue to work on restoring all clusters to full health.

Identified: A number of clusters in IAD, AMS, and SIN regions continue to see degraded replicas and PGBouncers at this time. A smaller number of clusters in these regions are also seeing disruption to their primaries. We continue to work on restoring full cluster health in all regions.

Identified: A small number of MPG clusters in the AMS and IAD region are currently in degraded states due to downstream impact from this Machines API issue: https://status.flyio.net/incidents/3npj6935byt4

Most of the impacted clusters may see a degraded replica or PG Bouncer in their statuspage. A very small number may be unable to connect to their MPG primary node, the team is working to restore connectivity as the top priority. Users may also see delays registering new clusters in these regions at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are still working on restoring the MPG clusters. Most of them should be operational already.

Identified: Affected hosts are starting to come back online. We are working on restoring affected MPG clusters.

Identified: One of our upstream providers is experiencing a major power issue in their AMS datacenter. Managed Postgres instances in AMS are experiencing an outage as our control plane for Managed Postgres is taken down by the incident.

Investigating: One of our upstream providers is performing an emergency DC maintenance. You may see degraded connectivity on some of your apps in AMS. Most apps in AMS are not affected.

Resolved: All MPG clusters are back to full, normal operations.

Monitoring: All MPG clusters are reachable.

Identified: We are still continuing cleanup on some clusters.

Identified: All cluster primary and pgBouncer machines are now healthy and operating normally.

We are still continuing cleanup on some clusters with lagging or degraded replicas, but this should not impact writes or reads to clusters.

Identified: We are continuing to work on restoring all clusters to full health.

Identified: With the underlying incident stabilizing (https://status.flyio.net/incidents/3npj6935byt4) we are seeing improvements amongst impacted clusters. We continue to work on restoring all clusters to full health.

Identified: A number of clusters in IAD, AMS, and SIN regions continue to see degraded replicas and PGBouncers at this time. A smaller number of clusters in these regions are also seeing disruption to their primaries. We continue to work on restoring full cluster health in all regions.

Identified: A small number of MPG clusters in the AMS and IAD region are currently in degraded states due to downstream impact from this Machines API issue: https://status.flyio.net/incidents/3npj6935byt4

Most of the impacted clusters may see a degraded replica or PG Bouncer in their statuspage. A very small number may be unable to connect to their MPG primary node, the team is working to restore connectivity as the top priority. Users may also see delays registering new clusters in these regions at this time.

Resolved: This incident has been resolved.

Monitoring: Sprite creation appears to be back to normal operation now.

Identified: We've identified the cause of the delay following creates and we're deploying a fix.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: Sprite creation generates an error that the sprite "is not assigned to compute." Eventually the sprite transitions from an unknown state to warm, so there is a delay before the sprite is usable.