UiPath Status

Resolved: A fix was implemented. Thank you for your patience while we worked to resolve this issue.

Monitoring: Our engineering team has narrowed down the scope to customer entities in Europe and has implemented a fix.

Investigating: We are currently investigating an issue impacting the login workflow for all customers who use Customer Managed Keys (CMK) with Single Sign-On (SSO) enabled. Our engineering team is actively working to identify the root cause and implement a solution.

Postmortem: ## Customer impact

On March 28, 2025, between 10:55–12:18 UTC Orchestrator service was down in the delayed Europe and U.S. regions. No traffic was routed to the Orchestrator during this timeframe. All other services that rely on Orchestrator were affected.

Root cause

Nginx is an open-source networking tool. UiPath uses it to route traffic. We were preparing to update the version of Nginx to fix a critical issue.

UiPath’s Nginx configuration is heavily customized. Unfortunately, these settings were missing due to a bug in our deployment pipeline. As a result, the deployment was successful, but Nginx was left in an unhealthy state. It could not route traffic to Orchestrator.

Detection

Our alerts immediately detected that traffic was not routing correctly. An engineer was paged and began investigating the outage.

The logs showed that Nginx was running, but it was unhealthy. This quickly pointed the engineer to the misconfiguration.

Response

The first thing we did was fix the settings by hand and start the deployment again. The update worked, but traffic was still not reaching our service.

After additional investigation, we determined that we also needed to manually delete and recreate the misconfigured objects. Once this was done, traffic began routing correctly. Customers were able to use Orchestrator again.

Follow-up

• We are still working on the full internal postmortem. As this is completed, we will identify additional action items and add them here.
• We will update the Nginx deployment to make it easier to review the changes. This should prevent the missing custom configuration.
• We will review the way that Nginx deployments are tested in non-production environments. This should ensure these sorts of mistakes are caught before they are visible to customers.
• When deploying Nginx into a region we will first send a small percent of traffic to the new version. This should allow us to quickly detect any issues that do make it to production.

Resolved: This incident has been resolved. Thank you for your patience.

Monitoring: A fix has been implemented and we are actively monitoring the situation.

Investigating: We are actively investigating the issue to restore services as quickly as possible.

Resolved: This issue has been mitigated. Thank you for your patience.

Monitoring: The issue has been mitigated. We will continue to monitor closely to ensure stability. Thank you for your patience.

Investigating: We are actively working to resolve the issue. Thank you for your patience.

Investigating: We are currently experiencing degraded performance with the Orchestrator and Document Understanding services in the East US region due to an issue with a third-party dependency.

Resolved: This issue has been mitigated. Thank you for your patience.

Monitoring: The issue has been mitigated. We will continue to monitor closely to ensure stability. Thank you for your patience.

Investigating: We are actively working to resolve the issue. Thank you for your patience.

Investigating: We are currently experiencing degraded performance with the Orchestrator and Document Understanding services in the East US region due to an issue with a third-party dependency.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.