Temporal Status

Resolved: A spike in workload caused resource contention in the underlying infrastructure, resulting in elevated latency for critical APIs. We scaled up the underlying resources to handle the increased load, and latencies have returned to normal. We will continue to monitor the system, but the mitigation is complete and the incident has been resolved.

Monitoring: Latencies are back to normal levels, we’re monitoring.

Investigating: The failover has been completed.

Investigating: We are failing over MRN namespaces

Investigating: Customers in AWS eu-west-1 may experience elevated API latencies.

Resolved: All MRN-enabled namespaces failed over by Temporal have been failed back. This incident has been fully mitigated and is now resolved.

Monitoring: We are seeing full recovery on our end, we are continuing to monitor.

Monitoring: Mitigation is in effect, we are seeing some recovery. Please submit a support ticket if you are seeing service degradation.

Identified: Mitigation is still in progress. We are scaling up service to help with mitigation efforts.

Identified: All MRN-enabled namespaces have been failed over as of 17:53 UTC. Mitigation continues to progress.

Identified: We are working on a fix, if your namespace are MRN enabled please fail over to your passive regions for faster recovery.

Investigating: Customers with namespaces in us-east-2 may experience errors with visibility queries. We are currently investigating.

Resolved: Errors and latencies have recovered, incident has been resolved.

Investigating: Some customers in AWS us-east-1 may experience elevated latencies and errors on their namespaces

Resolved: The latencies were caused by issues with a downstream service which resolved after we scaled up. We've been monitoring for errors and latencies and have not seen any further issues. This issue is now considered resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We have rolled out a change and have paused further investigation overnight pending validation under daytime traffic.

Identified: We are continuing to work on the investigation and remediation of the elevated API latencies in GCP us-east4.

We will provide further updates within one hour.

Identified: We are currently working to remediate the elevated API latencies that were observed in GCP us-east4.

Investigating: We are continuing to investigate the elevated API latencies in GCP us-east4 and will provide more details as they become available.

Investigating: Customers in GCP us-east4 may experience elevated API latencies.