Fly.io Status

Identified: We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing.

Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running.

We will provide another update within 15 minutes

Investigating: We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

Resolved: This incident has been resolved.

Monitoring: The maintenance has been completed and control plane functions should recover to normal. We're monitoring for any further complications.

Identified: We're performing an emergency maintenance on Petsem, our secrets management service. Some control plane write operations may temporarily fail, for example, creating new apps or secrets. Existing apps and machines should keep functioning without issues.

Identified: We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing.

Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running.

We will provide another update within 15 minutes

Investigating: We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

Resolved: This incident has been resolved.

Identified: Some of our edge nodes in GRU has suffered an error that crashed some of the critical services. We're currently working to bring them back online. Some traffic entering through GRU (i.e. users connecting from around GRU) may be temporarily affected: connections may see increased latency or be occasionally dropped.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Egress ips are broken in most of ORD, we are currently investigating this issue

Resolved: This incident has been resolved.

Monitoring: Our upstream paths have been fixed. We are monitoring the results.

Identified: We are working with our upstream network provider to address periodic loss of connectivity over transit in ord

Identified: We are seeing a recurrence of elevated latency across some hosts in ORD impacting the MPG control plane and a subset of clusters there. We are working to address this.

Monitoring: The network between our regions has been performing well for the majority of traffic. We're still continuing to monitor a few impacted routes in North America that may be seeing elevated latency and packet loss.

Monitoring: Impacted NA backbones have been sidestepped where possible, and we're continuing to monitor network health.

Monitoring: Managed Postgres in ORD has returned to normal operation.

We continue to see slightly elevated latencies and loss over transits in North America. We are working with our upstream network providers to improve performance.

Investigating: We are investigating elevated network instability across some hosts in ORD. Apps and managed postgres clusters on impacted hosts may see elevated latency or networking errors at this time.

Identified: We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing.

Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running.

We will provide another update within 15 minutes

Investigating: We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

Identified: We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing.

Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running.

We will provide another update within 15 minutes

Investigating: We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

Resolved: This incident has been resolved.

Monitoring: Our upstream paths have been fixed. We are monitoring the results.

Identified: We are working with our upstream network provider to address periodic loss of connectivity over transit in ord

Identified: We are seeing a recurrence of elevated latency across some hosts in ORD impacting the MPG control plane and a subset of clusters there. We are working to address this.

Monitoring: The network between our regions has been performing well for the majority of traffic. We're still continuing to monitor a few impacted routes in North America that may be seeing elevated latency and packet loss.

Monitoring: Impacted NA backbones have been sidestepped where possible, and we're continuing to monitor network health.

Monitoring: Managed Postgres in ORD has returned to normal operation.

We continue to see slightly elevated latencies and loss over transits in North America. We are working with our upstream network providers to improve performance.

Investigating: We are investigating elevated network instability across some hosts in ORD. Apps and managed postgres clusters on impacted hosts may see elevated latency or networking errors at this time.

Resolved: This incident has been resolved.

Monitoring: An initial fix has been implemented and connectivity to all impacted clusters has been restored. We are continuing to monitor to ensure stable recovery.

Identified: We are continuing to address this issue. Some clusters in IAD are unavailable at this time, some users may have seen unexpected cluter restarts. We are working on restoring normal performance for all clusters in IAD

Investigating: We are investigating MPG control plane instability in a subset of the IAD region. A small number of clusters in the region may have seen unexpected failovers or connection issues over the past 30m.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are seeing error rates for sprites in SIN normalize. We are continuing to monitor to ensure full recovery.

Investigating: We are investigating elevated 500 / internal server error rates with Sprites in the SIN region. Users may see increased errors when accessing sprites located in this region, or for requests to the Sprites API originating from SIN