Fly.io Status

Postmortem: ## Impact

From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region.

What happened

During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out.

What we're changing

• We will be adding tooling to return a replica to service safely after maintenance.
• We will be adding safety checks to our upgrade process.
• We will make the upgrade procedure more resilient to errors like this one.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We have identified a bug and are working on a fix.

Investigating: We are investigating issues in GRU related to a migration of our distributed state database. Apps continue to run, but creating or updating machines in GRU region may fail. Connections to apps in GRU, or connections from clients near GRU region, may fail at this time

Resolved: This incident has been resolved

Monitoring: A fix has been implemented and we are seeing private networking error rates return to normal. We are continuing to monitor to ensure full recovery.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: Sprites API performance has largely recovered and most users should no longer see errors creating, connecting to, or managing Sprites. We're still seeing a small number of intermittent errors and are investigating them before resolving this incident.

Monitoring: We have deployed another potential fix and are monitoring results.

Identified: This issue is now occurring on regions other than SJC. We are still working on a fix.

Identified: The issue has been identified and we are rolling out mitigations.

Investigating: We are investigating intermittent API failures for users connecting near SJC.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented are we are monitoring to ensure recovery.

Investigating: We are investigating elevated errors with private networking between machines in some regions.

Resolved: This incident has been resolved.

Identified: We are continuing to work on a fix for this issue.

Identified: One of the internet transit providers upstream of the affected hosts is performing regional maintenance, which is expected to complete at 10:00AM UTC. IPv6 connectivity to and from destinations using that transit may be impacted during this time window.

Investigating: We are investigating IPv6 connectivity issues on a subset of hosts in DFW region. Machines on impacted hosts may see inbound/outbound connectivity issues over IPv6. IPv4 connectivity is not impacted.