Temporal Status

Resolved: We have continued to see normal activity in this region with latencies holding at pre-incident levels. This incident is now resolved.

Monitoring: Our engineering has implemented a fix and we are seeing normal service return to this region. We are continuing to monitor.

Identified: As of 17:53 UTC our engineering team has identified the issue and is actively working to restore normal service. We will provide another update as more information becomes available.

Investigating: We are continuing to investigate this issue.

Investigating: Some customers in AWS eu-west-1 may experience elevated latencies starting 13:47 UTC time.

Resolved: This incident has been resolved.

Monitoring: The affected infrastructure has recovered. Latencies and error rates have returned to normal, and workloads have resumed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: A fix has been implemented and we are continuing to monitor.

Identified: We have mitigated the primary impact, and service has been restored for most affected customers. Recovery efforts continue for the remaining infrastructure, with a small number of customer namespaces still affected. Our engineering team is actively working to complete restoration and will provide another update as progress is made.

Investigating: We are currently investigating this issue

Resolved: We're seeing recovery on our end and errors and latencies on Visibility APIs have dropped. We will continue to monitor this but the incident is mitigated and resolved.

Monitoring: We have identified the issue and implemented mitigation efforts. We are currently monitoring.

Investigating: We are currently investigating this issue.

Resolved: Some customers may have experienced lag in Temporal Cloud metrics (OpenMetrics - v1), starting from 12:45 UTC to 13:25 UTC. Metrics lag have since recovered back to nominal.

Resolved: A spike in workload caused resource contention in the underlying infrastructure, resulting in elevated latency for critical APIs. We scaled up the underlying resources to handle the increased load, and latencies have returned to normal. We will continue to monitor the system, but the mitigation is complete and the incident has been resolved.

Monitoring: Latencies are back to normal levels, we’re monitoring.

Investigating: The failover has been completed.

Investigating: We are failing over MRN namespaces

Investigating: Customers in AWS eu-west-1 may experience elevated API latencies.

Resolved: All MRN-enabled namespaces failed over by Temporal have been failed back. This incident has been fully mitigated and is now resolved.

Monitoring: We are seeing full recovery on our end, we are continuing to monitor.

Monitoring: Mitigation is in effect, we are seeing some recovery. Please submit a support ticket if you are seeing service degradation.

Identified: Mitigation is still in progress. We are scaling up service to help with mitigation efforts.

Identified: All MRN-enabled namespaces have been failed over as of 17:53 UTC. Mitigation continues to progress.

Identified: We are working on a fix, if your namespace are MRN enabled please fail over to your passive regions for faster recovery.

Investigating: Customers with namespaces in us-east-2 may experience errors with visibility queries. We are currently investigating.

Resolved: Errors and latencies have recovered, incident has been resolved.

Investigating: Some customers in AWS us-east-1 may experience elevated latencies and errors on their namespaces