Temporal Status

Resolved: The environment has been validated and all residual issues have been cleared. This incident is now resolved.

Monitoring: A configuration change affecting namespaces in the us-central1 and asia-southeast2 regions resulted in some workflow requests not being processed as expected. The change was rolled back and workflow processing resumed. We are continuing to validate the environment and clear any residual issues, and will provide further updates as that work completes.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: While a small number of customer environments may still see some issues, mitigation has been in place and we continue to actively monitor.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: The issue has been mitigated (as of 21:55 UTC) and we are actively monitoring the previously impacted environments.

Identified: Some customers with namespaces in the us-central1 and asia-southeast2 regions may experience errors and blocked progress on their workflows. We are currently rolling out a fix.

Resolved: The environment has been validated and all residual issues have been cleared. This incident is now resolved.

Monitoring: A configuration change affecting namespaces in the us-central1 and asia-southeast2 regions resulted in some workflow requests not being processed as expected. The change was rolled back and workflow processing resumed. We are continuing to validate the environment and clear any residual issues, and will provide further updates as that work completes.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: While a small number of customer environments may still see some issues, mitigation has been in place and we continue to actively monitor.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: The issue has been mitigated (as of 21:55 UTC) and we are actively monitoring the previously impacted environments.

Identified: Some customers with namespaces in the us-central1 and asia-southeast2 regions may experience errors and blocked progress on their workflows. We are currently rolling out a fix.

Resolved: This incident has been resolved.

Monitoring: OpenMetrics incomplete responses have recovered and remain stable. We continue to monitor the situation.

Investigating: We are currently investigating a degradation in OpenMetrics endpoint causing incomplete responses for customers in APSE region starting 10:10 UTC.