Postmortem: # What happened

A recent upgrade to regions AU.platform.sh and AU-2.platform.sh was unable to fully purge the stale network states within our routing infrastructure.

This caused client programs within our subsystems responsible for handling deployments, console and project API having busy-looping processes retrying invalid TCP connections indefinitely and consumed all the available CPU resources.

‌

This caused the console to freeze, slow loading times when using project API functions, and delays in deployments or backups.

‌

After further investigation, the Operations team has deployed an emergency fix to eliminate those busy-looping processes.

Our team has stopped the scheduled upgrades to other regions and is still working on a permanent fix.

‌

Customer impact

From 2025-03-20 12:50 UTC to 2025-03-21 09:00 UTC, customers may experience delays when loading console, triggering deployments or taking backups for their projects in regions AU.platform.sh and AU-2.platform.sh .

‌

From 2025-03-21 07:27 UTC to 2025-03-21 07:55 UTC, there may be issues with outgoing connections in AU.platform.sh environments, including live ones, due to our emergency fix deployment.

‌

What was done to resolve the incident

The Operations team has taken out the busy-looping processes in the subsystems for deployments, console, and project API functions and deployed an emergency correction fix to the affected regions.

‌

Incident has been resolved.

Resolved: This incident has been resolved.

Monitoring: After applying an emergency fix, we can see that the services are now operational for both AU and AU-2 regions.

Our Operations team is still actively monitoring the services and implementing a permanent fix for this issue.

Identified: AU is now fully recovered.

Our engineers are still implementing a permanent fix for this issue.

Identified: The outgoing connections issues have been corrected after deploying an emergency fix.

Identified: We are continuing to work on a fix for this issue.

Identified: We noticed that some customer environments are failing to make outgoing connections. Our engineers are still actively working on the recovery.

Identified: AU-2 is now fully recovered.

Identified: We have detected an issue affecting service on the AU and AU-2 regions. Our Operations team has been notified and is currently working to resolve the issue. Projects on affected regions may experience delays when loading console, triggering deployments or taking backups. This issue does not affect availability of any environments, only the API services for the project. There is no site downtime as a result of this issue.

Postmortem: # What happened

A recent upgrade to regions AU.platform.sh and AU-2.platform.sh was unable to fully purge the stale network states within our routing infrastructure.

This caused client programs within our subsystems responsible for handling deployments, console and project API having busy-looping processes retrying invalid TCP connections indefinitely and consumed all the available CPU resources.

‌

This caused the console to freeze, slow loading times when using project API functions, and delays in deployments or backups.

‌

After further investigation, the Operations team has deployed an emergency fix to eliminate those busy-looping processes.

Our team has stopped the scheduled upgrades to other regions and is still working on a permanent fix.

‌

Customer impact

From 2025-03-20 12:50 UTC to 2025-03-21 09:00 UTC, customers may experience delays when loading console, triggering deployments or taking backups for their projects in regions AU.platform.sh and AU-2.platform.sh .

‌

From 2025-03-21 07:27 UTC to 2025-03-21 07:55 UTC, there may be issues with outgoing connections in AU.platform.sh environments, including live ones, due to our emergency fix deployment.

‌

What was done to resolve the incident

The Operations team has taken out the busy-looping processes in the subsystems for deployments, console, and project API functions and deployed an emergency correction fix to the affected regions.

‌

Incident has been resolved.

Resolved: This incident has been resolved.

Monitoring: After applying an emergency fix, we can see that the services are now operational for both AU and AU-2 regions.

Our Operations team is still actively monitoring the services and implementing a permanent fix for this issue.

Identified: AU is now fully recovered.

Our engineers are still implementing a permanent fix for this issue.

Identified: The outgoing connections issues have been corrected after deploying an emergency fix.

Identified: We are continuing to work on a fix for this issue.

Identified: We noticed that some customer environments are failing to make outgoing connections. Our engineers are still actively working on the recovery.

Identified: AU-2 is now fully recovered.

Identified: We have detected an issue affecting service on the AU and AU-2 regions. Our Operations team has been notified and is currently working to resolve the issue. Projects on affected regions may experience delays when loading console, triggering deployments or taking backups. This issue does not affect availability of any environments, only the API services for the project. There is no site downtime as a result of this issue.