Fly.io Status

Resolved: This incident has been resolved.

Investigating: We’re investigating an issue where some customers see a "Something went wrong" error when adding a credit card in the Fly.io dashboard.

Resolved: This incident has been resolved.

Monitoring: All servers in EZE are operational and machines are back online. We are monitoring the service health.

Identified: The hardware failure has been resolved, and we are working to bring machines back online.

Identified: Machines in EZE remain unavailable due to equipment failure caused by the power outage. We are working with our upstream providers to restore connectivity.

Identified: We are continuing to track this issue with our upstream providers.

Identified: We are continuing to track this issue with our upstream providers.

Identified: Due to a power failure at the EZE datacenter, machines hosted in EZE are unavailable. We are tracking this issue with our upstream providers.

Investigating: We are currently investigating this issue. Apps with machines only in EZE will be unavailable.

Resolved: This incident has been resolved.

Monitoring: Traffic has been rerouted in all affected regions and we are not seeing further networking issues. We are continuing to monitor.

Investigating: We are investigating network issues across our US / North America regions. Machines hosted in these regions may experience increased latency or packet loss.

Resolved: This incident has been resolved.

Monitoring: Traffic has been rerouted in all affected regions and we are not seeing further networking issues. We are continuing to monitor.

Investigating: We are investigating network issues across our US / North America regions. Machines hosted in these regions may experience increased latency or packet loss.

Resolved: This incident has been resolved.

Monitoring: This incident has been mitigated for all but a small number of hosts in SJC region. We are waiting for those hosts to recover and are monitoring the situation.

Identified: We've identified an issue with an upstream network provider across various US regions, and are working to shift traffic to alternative routes.

Investigating: We are investigating network issues in SJC/SEA regions. Apps running in those regions may experience increased latency or packet loss.

Resolved: This incident has been resolved.

Monitoring: This incident has been mitigated for all but a small number of hosts in SJC region. We are waiting for those hosts to recover and are monitoring the situation.

Identified: We've identified an issue with an upstream network provider across various US regions, and are working to shift traffic to alternative routes.

Investigating: We are investigating network issues in SJC/SEA regions. Apps running in those regions may experience increased latency or packet loss.

Resolved: This incident has been resolved.

Monitoring: Traffic has been rerouted in all affected regions and we are not seeing further networking issues. We are continuing to monitor.

Investigating: We are investigating network issues across our US / North America regions. Machines hosted in these regions may experience increased latency or packet loss.

Resolved: This incident is now resolved, Depot builders should be working correctly now.

Identified: We're still rolling out configurations to mitigate this issue. Note that this issue only affects users with builders in some US regions. In the meanwhile, trying the recommended workarounds, or relocating your builder (Settings -> App builders) to either Dallas, Chicago or Toronto should allow your builds to complete successfully.

Identified: We've identified the root cause for this issue and are implementing mitigations.

Investigating: We are investigating sporadic build failures mentioning Docker rate limits on "unauthenticated pulls" and error 429.

In the meantime, workarounds to try in order are:

1. Use fly deploy --depot=false
2. Use fly deploy --buildkit
3. Install Docker locally and use --local-only with Fly deploy

Resolved: This incident is now resolved, Depot builders should be working correctly now.

Identified: We're still rolling out configurations to mitigate this issue. Note that this issue only affects users with builders in some US regions. In the meanwhile, trying the recommended workarounds, or relocating your builder (Settings -> App builders) to either Dallas, Chicago or Toronto should allow your builds to complete successfully.

Identified: We've identified the root cause for this issue and are implementing mitigations.

Investigating: We are investigating sporadic build failures mentioning Docker rate limits on "unauthenticated pulls" and error 429.

In the meantime, workarounds to try in order are:

1. Use fly deploy --depot=false
2. Use fly deploy --buildkit
3. Install Docker locally and use --local-only with Fly deploy

Resolved: This incident has been resolved.

Identified: We've identified the issue within Quickwit data source for logs. We're implementing a fix.

Investigating: We found out that loading old logs via Grafana is return 5xx errors. Tailing live logs via UI and CLI is working.