UiPath Status

Postmortem: ## Customer impact

On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region:

• Australia: approximately 15:45 UTC to 18:35 UTC
• Europe: approximately 16:35 UTC to 18:00 UTC
• Japan: approximately 16:45 UTC to 18:30 UTC

Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated.

Root cause

We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over.

In the affected regions, the traffic switch was approved by the engineer running the upgrade before the data restore and the ingestion catch-up had completed, which is a deviation from our upgrade procedure. This was a human error. As a result, Orchestrator was reading robot logs from a cluster that did not yet contain recent data, while new logs continued to be collected but were not yet indexed on the cluster serving requests. The error was not caught because our deployment tooling relies on the operator to verify data completeness and does not enforce the order between the data restore and the traffic switch.

Detection

Our synthetic monitoring, which continuously runs test jobs and verifies that their logs are returned, detected missing logs in the affected regions and raised alerts. Our engineering team correlated the alerts with the in-progress upgrade, declared an incident and published a status page notification.

Response

Our engineering team reverted the traffic to the previous Elasticsearch clusters in all three regions, which immediately restored access to all robot logs, including those produced during the impact window. We confirmed recovery through synthetic monitoring in every affected region and continued monitoring before marking the incident resolved at approximately 18:45 UTC.

The upgrade of the new clusters continues in the background. Traffic will be moved to them only after the data restore is complete and ingestion is confirmed to be up to date.

Follow up

To prevent this from happening again, we are:

• Changing the deployment pipeline so that the traffic switch cannot run until the data restore and log ingestion on the new cluster are verified as complete.
• Adding an automated rollback step to the pipeline, so traffic can be returned to the previous cluster within minutes if a problem is detected after a switch.

Resolved: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Monitoring: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Investigating: We are investigating reports of an outage impacting [impacted functionality or user symptom] for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Resolved: This issue has been resolved. Between 15:05 and 16:39 UTC, jobs on Cloud Robots in the US region were slow to start or stayed in the Pending state. Jobs have been starting normally since 16:39 UTC, and jobs that were delayed during this time have now been processed. We apologize for the disruption.

Monitoring: After further work on the issue, jobs on Cloud Robots in the US region are now starting normally. Jobs that were delayed earlier are being processed. We're continuing to monitor the service and will post another update once the issue is resolved.

Monitoring: We've applied a fix for the issue that delayed jobs on Cloud Robots in the US region, and jobs are starting normally again. Jobs that were delayed are now being processed. We're monitoring to make sure the service stays stable and will post another update within 60 minutes.

Investigating: Since about 15:05 UTC, some customers in the US region may see jobs that run on Cloud Robots take longer than usual to start, or stay in the Pending state. Jobs on your own robots are not affected, and neither are other regions. We've found the likely cause and are working to restore normal job processing. Jobs that are already queued are not lost and will be processed once the issue is resolved. We'll post another update within 60 minutes.

Resolved: We have the issue in the East US region and error rates and latency returned to normal levels.

Monitoring: We have applied a capacity increase to the affected database infrastructure in the East US region and error rates and latency returned to normal levels. We will provide an update as soon as more information is available.

Investigating: We are investigating reports of intermittent slowness and errors affecting a subset of organizations in the East US region of Automation Cloud. Affected users may see pages in the Orchestrator UI fail to load (for example, the Jobs page returning "An error has occurred"), and robot operations such as queue transactions and session starts may intermittently time out. We will provide an update as soon as more information is available.

Resolved: This issue has been resolved. Between 15:05 and 16:39 UTC, jobs on Cloud Robots in the US region were slow to start or stayed in the Pending state. Jobs have been starting normally since 16:39 UTC, and jobs that were delayed during this time have now been processed. We apologize for the disruption.

Monitoring: After further work on the issue, jobs on Cloud Robots in the US region are now starting normally. Jobs that were delayed earlier are being processed. We're continuing to monitor the service and will post another update once the issue is resolved.

Monitoring: We've applied a fix for the issue that delayed jobs on Cloud Robots in the US region, and jobs are starting normally again. Jobs that were delayed are now being processed. We're monitoring to make sure the service stays stable and will post another update within 60 minutes.

Investigating: Since about 15:05 UTC, some customers in the US region may see jobs that run on Cloud Robots take longer than usual to start, or stay in the Pending state. Jobs on your own robots are not affected, and neither are other regions. We've found the likely cause and are working to restore normal job processing. Jobs that are already queued are not lost and will be processed once the issue is resolved. We'll post another update within 60 minutes.

Resolved: The fix for the issue impacting HITL task creation for Maestro with HITL task SLA timers configured in US has been fully rolled out. HITL task creation has returned to normal.

Monitoring: We are rolling out fix for an outage impacting HITL task creation for Maestros with HITL task SLA timers configured in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks within Maestros with HITL task SLA timers configured. Next update: Our teams are working to mitigate the issue and will share updates as available.

Monitoring: We are rolling out fix for an outage impacting HITL task creation for Maestros with HITL task SLA timers configured in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks within Maestros with HITL task SLA timers configured. Next update: Our teams are working to mitigate the issue and will share updates as available.

Identified: We have identified the cause for an outage impacting HITL task creation for Maestro in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks. Next update: Our teams are working to mitigate the issue and will share updates as available.

Identified: We have identified the cause for an outage impacting HITL task creation for Maestro in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks. Next update: Our teams are working to mitigate the issue and will share updates as available.

Identified: We have identified the cause for an outage impacting HITL task creation for Maestro in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks. Next update: Our teams are working to mitigate the issue and will share updates as available.

Identified: We have identified the cause for an outage impacting HITL task creation for Maestro in US and are working to mitigate the issue. Impact: Users may be unable to create HITL tasks. Next update: Our teams are working to mitigate the issue and will share updates as available.

Investigating: We are investigating reports of an outage impacting HITL task creation failures for Maestro in US. Impact: Users may be unable to create HITL tasks. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Postmortem: ## Customer impact

On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region:

• Australia: approximately 15:45 UTC to 18:35 UTC
• Europe: approximately 16:35 UTC to 18:00 UTC
• Japan: approximately 16:45 UTC to 18:30 UTC

Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated.

Root cause

We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over.

In the affected regions, the traffic switch was approved by the engineer running the upgrade before the data restore and the ingestion catch-up had completed, which is a deviation from our upgrade procedure. This was a human error. As a result, Orchestrator was reading robot logs from a cluster that did not yet contain recent data, while new logs continued to be collected but were not yet indexed on the cluster serving requests. The error was not caught because our deployment tooling relies on the operator to verify data completeness and does not enforce the order between the data restore and the traffic switch.

Detection

Our synthetic monitoring, which continuously runs test jobs and verifies that their logs are returned, detected missing logs in the affected regions and raised alerts. Our engineering team correlated the alerts with the in-progress upgrade, declared an incident and published a status page notification.

Response

Our engineering team reverted the traffic to the previous Elasticsearch clusters in all three regions, which immediately restored access to all robot logs, including those produced during the impact window. We confirmed recovery through synthetic monitoring in every affected region and continued monitoring before marking the incident resolved at approximately 18:45 UTC.

The upgrade of the new clusters continues in the background. Traffic will be moved to them only after the data restore is complete and ingestion is confirmed to be up to date.

Follow up

To prevent this from happening again, we are:

• Changing the deployment pipeline so that the traffic switch cannot run until the data restore and log ingestion on the new cluster are verified as complete.
• Adding an automated rollback step to the pipeline, so traffic can be returned to the previous cluster within minutes if a problem is detected after a switch.

Resolved: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Monitoring: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Investigating: We are investigating reports of an outage impacting [impacted functionality or user symptom] for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Postmortem: ## Customer impact

On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region:

• Australia: approximately 15:45 UTC to 18:35 UTC
• Europe: approximately 16:35 UTC to 18:00 UTC
• Japan: approximately 16:45 UTC to 18:30 UTC

Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated.

Root cause

We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over.

In the affected regions, the traffic switch was approved by the engineer running the upgrade before the data restore and the ingestion catch-up had completed, which is a deviation from our upgrade procedure. This was a human error. As a result, Orchestrator was reading robot logs from a cluster that did not yet contain recent data, while new logs continued to be collected but were not yet indexed on the cluster serving requests. The error was not caught because our deployment tooling relies on the operator to verify data completeness and does not enforce the order between the data restore and the traffic switch.

Detection

Our synthetic monitoring, which continuously runs test jobs and verifies that their logs are returned, detected missing logs in the affected regions and raised alerts. Our engineering team correlated the alerts with the in-progress upgrade, declared an incident and published a status page notification.

Response

Our engineering team reverted the traffic to the previous Elasticsearch clusters in all three regions, which immediately restored access to all robot logs, including those produced during the impact window. We confirmed recovery through synthetic monitoring in every affected region and continued monitoring before marking the incident resolved at approximately 18:45 UTC.

The upgrade of the new clusters continues in the background. Traffic will be moved to them only after the data restore is complete and ingestion is confirmed to be up to date.

Follow up

To prevent this from happening again, we are:

• Changing the deployment pipeline so that the traffic switch cannot run until the data restore and log ingestion on the new cluster are verified as complete.
• Adding an automated rollback step to the pipeline, so traffic can be returned to the previous cluster within minutes if a problem is detected after a switch.

Resolved: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Monitoring: We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Identified: We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Investigating: We are investigating reports of an outage impacting [impacted functionality or user symptom] for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.