Fly.io Status

Resolved: This incident has been resolved.

Monitoring: Background jobs have caught up and the API is fully operational. We are continuing to monitor service health.

Identified: A fix has been put in place, and we are no longer seeing elevated API errors. Some dashboard actions will be delayed while background processing catches up.

Identified: The cause of the errors has been identified and we are working on a fix.

Investigating: We are investigating elevated errors with our GraphQL API and background job processing

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are aware of egress IP issues in SIN and NRT and are working on a fix. Some machines in SIN and NRT using egress IPs may temporarily lose connectivity or otherwise see degraded performance.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are investigating issues with static egress IPv6 addresses in NRT region. Apps using static egress IPs may experience connectivity failures to some destinations.

Resolved: This incident has been resolved.

Monitoring: Customer workloads are now starting and we're monitoring the affected hosts. Affected Managed Postgres instances will be investigated.

Identified: Power restoration is ongoing and we're making sure the hosts are healthy before starting customer workloads to avoid issues. Customer impact remains and updates to come.

Identified: Power restoration work in a subset of ORD is still in progress and impact remains ongoing for a subset of hosts and some Managed Postgres clusters.

Identified: Our provider has advised us their facilities team is working on restoring the power, we'll provide another update as soon as we learn more.

Identified: We've identified and reported power issues with one of our upstream providers in ORD. We're waiting for an update from our upstream for a resolution. Some Managed Postgres clusters in ORD will be unavailable due to placement.

Investigating: We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time. Some Managed Postgres clusters in ORD region may be unavailable at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We've identified the issue as a networking hardware failure impacting a subset of hosts at one of our Upstream providers in ORD. We are working with our provider to restore connectivity.

Investigating: We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time.

Some Managed Postgres clusters in ORD region may be unavailable or see connectivity issues at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are aware of egress IP issues in SIN and NRT and are working on a fix. Some machines in SIN and NRT using egress IPs may temporarily lose connectivity or otherwise see degraded performance.

Resolved: This incident has been resolved.

Identified: Almost all metrics have caught up aside from a small handful in sin and syd. We're continuing to monitor and expect these to complete in the next few hours.

Identified: Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.

Identified: Backlogged metrics are still being processed and ingestion delays persist for some customers

Identified: Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.

Identified: Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.

Identified: Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.

Identified: We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.

Identified: We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.

Identified: We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.

Identified: We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.

Investigating: We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented upstream and certificates are being issued successfully. We will continue to monitor.

Identified: The issue has been identified and we are awaiting a fix.

Investigating: We are currently investigating errors when issuing new SSL certificates for hostnames.

Resolved: This incident has been resolved.

Monitoring: Customer workloads are now starting and we're monitoring the affected hosts. Affected Managed Postgres instances will be investigated.

Identified: Power restoration is ongoing and we're making sure the hosts are healthy before starting customer workloads to avoid issues. Customer impact remains and updates to come.

Identified: Power restoration work in a subset of ORD is still in progress and impact remains ongoing for a subset of hosts and some Managed Postgres clusters.

Identified: Our provider has advised us their facilities team is working on restoring the power, we'll provide another update as soon as we learn more.

Identified: We've identified and reported power issues with one of our upstream providers in ORD. We're waiting for an update from our upstream for a resolution. Some Managed Postgres clusters in ORD will be unavailable due to placement.

Investigating: We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time. Some Managed Postgres clusters in ORD region may be unavailable at this time.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We've identified the issue as a networking hardware failure impacting a subset of hosts at one of our Upstream providers in ORD. We are working with our provider to restore connectivity.

Investigating: We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time.

Some Managed Postgres clusters in ORD region may be unavailable or see connectivity issues at this time.