Postmortem: # What happened

Failed automation prevented connections from reaching the backend subsystems responsible for serving project API/console functions in region AU-2.

‌

‌

Customer impact

Between 2025-10-29 22:01 UTC and 2025-10-30 01:48 UTC,
customers were unable to submit deployments, access the web console UI, and use project API functions for their projects in region AU-2.

Environments as well as live production sites are NOT affected by this incident.

‌

What was done to resolve the incident

Manual intervention restored the connections to those backend subsystems.
Our engineering team will be implementing a long-term fix in order to avoid this from happening in the future.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We have detected an issue affecting service on the AU-2 region. Our Operations team has been notified and is currently working to restore service. Customers might face problems using features like submitting deployments, the web console UI, and project API functions.

This issue impacts console/project API functions only. Environments as well as live production sites are NOT affected. We will update you as soon as we have further information.

Postmortem: ## What happened

An update to orchestration service introduced a fault which adversely affected coordination of services on dedicated clusters.

Customer Impact

Some customer sites experienced outages while others experienced degraded performance. Deployments on some clusters were also affected.

What was done to resolve the incident

Faulty update was rolled back and affected clusters were fixed by operations team.

Resolved: This incident has been resolved.

Monitoring: We have fixed all the identified clusters affected by this issue, and we are continuing to monitor the status.

Monitoring: Fix has been rolled out to most of the clusters. Team is working through remaining clusters.

Monitoring: We are still deploying the fix across the remaining affected clusters. We are closely monitoring system performance to ensure full resolution and detect any residual issues.

Monitoring: The deployment of the fix is currently underway across the remaining affected clusters. We are closely monitoring system performance to ensure full resolution and detect any residual issues.

Monitoring: A fix is being rolled out and we are monitoring the results

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified, and a fix is being implemented.

Symptoms include unhealthy services on the cluster, which will often result in the application returning 500 errors.

Investigating: We have detected an issue affecting orchestration services on dedicated enterprise clusters. Our Operations team has been notified and is currently working to restore service.

We will update you as soon as we have further information.

Postmortem: # What happened

Failed automation prevented connections from reaching the backend subsystems responsible for serving project API/console functions in region AU-2.

‌

‌

Customer impact

Between 2025-10-29 22:01 UTC and 2025-10-30 01:48 UTC,
customers were unable to submit deployments, access the web console UI, and use project API functions for their projects in region AU-2.

Environments as well as live production sites are NOT affected by this incident.

‌

What was done to resolve the incident

Manual intervention restored the connections to those backend subsystems.
Our engineering team will be implementing a long-term fix in order to avoid this from happening in the future.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We have detected an issue affecting service on the AU-2 region. Our Operations team has been notified and is currently working to restore service. Customers might face problems using features like submitting deployments, the web console UI, and project API functions.

This issue impacts console/project API functions only. Environments as well as live production sites are NOT affected. We will update you as soon as we have further information.