UiPath Status

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: Subset of operations across multiple services is intermittently failing across multiple regions.

Postmortem: ## Customer Impact

Between April 16, 2026 at 17:31 UTC and April 16, 2026 at 18:32 UTC, a subset of customers in the US and EU regions experienced elevated error rates and incomplete responses when using the Document Understanding service's generative extraction features. During this period, affected users encountered server errors (HTTP 500) when submitting documents for automated data extraction, resulting in approximately 650 failed requests across both regions, with the US region accounting for the majority. The period of elevated errors lasted approximately one hour before subsiding. Some affected organizations experienced error rates exceeding 70% at the peak of the incident, while others saw lower but still disruptive failure rates.

Root Cause

The incident was caused by an unexpected, temporary change in the response format from our external AI model provider's embeddings related endpoint. This change broke the established interface contract between the provider and our Document Understanding service, preventing the service from correctly parsing embedding responses.

Specifically, our service expected a structured response object containing a required attribute, but the provider returned an empty response (HTTP 200 with an empty body). This caused a parsing error when our service attempted to process the response body. Importantly, the provider's endpoint continued to return a successful HTTP status code, meaning the failure was not detectable at the network level and only surfaced during response processing within our service.

The issue affected multiple unrelated organizations simultaneously across both regions and lasted approximately one hour, consistent with a temporary provider-side deployment or configuration change.

Detection

The incident was detected on April 16, 2026 at 17:44 UTC, when automated monitoring systems triggered alerts for increased counts of failed requests in the Document Understanding service's generative extraction features across both the US and EU regions. Analysis of service logs and error metrics revealed a sharp rise in server errors, with multiple organizations across both regions experiencing failures. The time between the onset of elevated error rates and detection was short, owing to real-time alerting and continuous monitoring of service health.

Response

Upon detection, the engineering team immediately convened via a dedicated call to investigate the root cause. A public status page was published at approximately 18:28 UTC informing customers that the team was investigating elevated error rates and incomplete responses in the generative extraction service across the US and EU regions. Initial efforts focused on isolating the affected regions and identifying patterns in failed requests. The team examined specific failed extraction requests, retrieved error artifacts for analysis, and compared them against successful reference cases to characterize the failure mode.

Our engineering team ruled out all internal causes early in the investigation. No deployments or configuration changes had been made to our systems for approximately one week prior to the incident. The team confirmed through systematic log analysis that the failures were isolated to the embedding model endpoint, corroborating the external provider as the source of the disruption. A support ticket was filed with the provider the following day to obtain a detailed explanation of the change.

Error rates began subsiding approximately one hour after onset. By 18:42 UTC, the team confirmed that the incident was no longer active, though investigation into the underlying cause continued. The public status page was updated at 19:19 UTC to reflect that error rates had returned to normal and the system was stable. The incident was declared fully resolved at 19:35 UTC, with all affected services operating normally. The status page was updated with a final resolution notice, and ongoing monitoring confirmed sustained stability.

Follow-Up

To reduce the risk of similar incidents and minimize customer impact from external dependency failures, we are implementing the following targeted improvements:

1. Fallback and redundancy strategies: We are evaluating and, where feasible, implementing fallback mechanisms to alternative models or providers when a contract break or provider outage is detected, reducing the duration and severity of customer impact.
2. Improved diagnostic logging: We are updating our systems to capture and retain raw provider responses for failed requests. This will enable faster forensic analysis during future incidents and eliminate the diagnostic gap identified during this investigation.
3. Expanded monitoring: We are updating our suite of alerts to cover this scenario. This will significantly reduce detection time for similar incidents.

These measures build on lessons learned from this and similar past events. We are committed to delivering reliable, resilient automation services and will continue to strengthen our systems to ensure that customers experience minimal disruption, even when external dependencies change unexpectedly.

Resolved: The issue affecting Document Understanding (DU) generative extraction in both the US and EU regions has been fully resolved.

All systems are operating normally. We will continue to monitor to ensure ongoing stability.

Monitoring: We have observed that error rates affecting Document Understanding (DU) generative extraction in both the US and EU regions have subsided.

The system is currently stable, and our team continues to monitor performance closely to ensure full reliability. We will share further updates if there are any changes.

Investigating: We are currently investigating an increase in failed requests affecting Document Understanding (DU) generative extraction capabilities in both the US and EU regions.

Our team has identified the issue and is actively working on mitigation. During this time, users may experience higher error rates or incomplete responses when using generative extraction features.

We will provide updates as we make progress toward full resolution.

Postmortem: ## Customer Impact

Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows.

Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required.

Root Cause

A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface.

The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects.

Detection

Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface.

Response

Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally.

Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved.

Follow-up

Short-term

• Enhance testing to automatically test for behaviour of legacy IXP projects
• Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements

Resolved: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

Monitoring: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

Identified: A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects.

During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required.

We will confirm once the rollout is complete.

Investigating: We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change.

This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI.

Importantly, IXP ingestion is not affected, and no action is required from customers at this time.

Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.

Identified: We are validating the reversion of the improper configuration in a lower environment, and will begin deploying it to production regions shortly

Identified: We have identified a defect in a recent deployment, which is causing the issue. We are preparing to roll back the change, which will return Agents to full functionality for all customers.

Investigating: Further investigation has revealed that this issue is affecting all regions, so customers may experience Agents failing to start across all UiPath Automation Cloud regions.

We are continuing to investigate the root cause of the issue and will be working to mitigate it promptly.

Investigating: We are continuing to investigate the cause of the issues, we will provide an update as soon as we begin working on a mitigation

Investigating: We are investigating reports of an outage impacting Agents in US and Europe Regions. Impact: Users may be unable to start Agents. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: We have identified an issue affecting customers who have a VPN gateway configured and are navigating through the UI. Other customer scenarios remain unaffected. Based on current observations, the overall impact is low.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: Subset of operations across multiple services is intermittently failing across multiple regions.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: We have identified an issue affecting customers who have a VPN gateway configured and are navigating through the UI. Other customer scenarios remain unaffected. Based on current observations, the overall impact is low.

Postmortem: ## Customer Impact

Between April 16, 2026 at 17:31 UTC and April 16, 2026 at 18:32 UTC, a subset of customers in the US and EU regions experienced elevated error rates and incomplete responses when using the Document Understanding service's generative extraction features. During this period, affected users encountered server errors (HTTP 500) when submitting documents for automated data extraction, resulting in approximately 650 failed requests across both regions, with the US region accounting for the majority. The period of elevated errors lasted approximately one hour before subsiding. Some affected organizations experienced error rates exceeding 70% at the peak of the incident, while others saw lower but still disruptive failure rates.

Root Cause

The incident was caused by an unexpected, temporary change in the response format from our external AI model provider's embeddings related endpoint. This change broke the established interface contract between the provider and our Document Understanding service, preventing the service from correctly parsing embedding responses.

Specifically, our service expected a structured response object containing a required attribute, but the provider returned an empty response (HTTP 200 with an empty body). This caused a parsing error when our service attempted to process the response body. Importantly, the provider's endpoint continued to return a successful HTTP status code, meaning the failure was not detectable at the network level and only surfaced during response processing within our service.

The issue affected multiple unrelated organizations simultaneously across both regions and lasted approximately one hour, consistent with a temporary provider-side deployment or configuration change.

Detection

The incident was detected on April 16, 2026 at 17:44 UTC, when automated monitoring systems triggered alerts for increased counts of failed requests in the Document Understanding service's generative extraction features across both the US and EU regions. Analysis of service logs and error metrics revealed a sharp rise in server errors, with multiple organizations across both regions experiencing failures. The time between the onset of elevated error rates and detection was short, owing to real-time alerting and continuous monitoring of service health.

Response

Upon detection, the engineering team immediately convened via a dedicated call to investigate the root cause. A public status page was published at approximately 18:28 UTC informing customers that the team was investigating elevated error rates and incomplete responses in the generative extraction service across the US and EU regions. Initial efforts focused on isolating the affected regions and identifying patterns in failed requests. The team examined specific failed extraction requests, retrieved error artifacts for analysis, and compared them against successful reference cases to characterize the failure mode.

Our engineering team ruled out all internal causes early in the investigation. No deployments or configuration changes had been made to our systems for approximately one week prior to the incident. The team confirmed through systematic log analysis that the failures were isolated to the embedding model endpoint, corroborating the external provider as the source of the disruption. A support ticket was filed with the provider the following day to obtain a detailed explanation of the change.

Error rates began subsiding approximately one hour after onset. By 18:42 UTC, the team confirmed that the incident was no longer active, though investigation into the underlying cause continued. The public status page was updated at 19:19 UTC to reflect that error rates had returned to normal and the system was stable. The incident was declared fully resolved at 19:35 UTC, with all affected services operating normally. The status page was updated with a final resolution notice, and ongoing monitoring confirmed sustained stability.

Follow-Up

To reduce the risk of similar incidents and minimize customer impact from external dependency failures, we are implementing the following targeted improvements:

1. Fallback and redundancy strategies: We are evaluating and, where feasible, implementing fallback mechanisms to alternative models or providers when a contract break or provider outage is detected, reducing the duration and severity of customer impact.
2. Improved diagnostic logging: We are updating our systems to capture and retain raw provider responses for failed requests. This will enable faster forensic analysis during future incidents and eliminate the diagnostic gap identified during this investigation.
3. Expanded monitoring: We are updating our suite of alerts to cover this scenario. This will significantly reduce detection time for similar incidents.

These measures build on lessons learned from this and similar past events. We are committed to delivering reliable, resilient automation services and will continue to strengthen our systems to ensure that customers experience minimal disruption, even when external dependencies change unexpectedly.

Resolved: The issue affecting Document Understanding (DU) generative extraction in both the US and EU regions has been fully resolved.

All systems are operating normally. We will continue to monitor to ensure ongoing stability.

Monitoring: We have observed that error rates affecting Document Understanding (DU) generative extraction in both the US and EU regions have subsided.

The system is currently stable, and our team continues to monitor performance closely to ensure full reliability. We will share further updates if there are any changes.

Investigating: We are currently investigating an increase in failed requests affecting Document Understanding (DU) generative extraction capabilities in both the US and EU regions.

Our team has identified the issue and is actively working on mitigation. During this time, users may experience higher error rates or incomplete responses when using generative extraction features.

We will provide updates as we make progress toward full resolution.

Postmortem: ## Customer Impact

Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows.

Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required.

Root Cause

A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface.

The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects.

Detection

Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface.

Response

Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally.

Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved.

Follow-up

Short-term

• Enhance testing to automatically test for behaviour of legacy IXP projects
• Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements

Resolved: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

Monitoring: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

Identified: A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects.

During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required.

We will confirm once the rollout is complete.

Investigating: We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change.

This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI.

Importantly, IXP ingestion is not affected, and no action is required from customers at this time.

Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.

Identified: We are validating the reversion of the improper configuration in a lower environment, and will begin deploying it to production regions shortly

Identified: We have identified a defect in a recent deployment, which is causing the issue. We are preparing to roll back the change, which will return Agents to full functionality for all customers.

Investigating: Further investigation has revealed that this issue is affecting all regions, so customers may experience Agents failing to start across all UiPath Automation Cloud regions.

We are continuing to investigate the root cause of the issue and will be working to mitigate it promptly.

Investigating: We are continuing to investigate the cause of the issues, we will provide an update as soon as we begin working on a mitigation

Investigating: We are investigating reports of an outage impacting Agents in US and Europe Regions. Impact: Users may be unable to start Agents. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: We have identified an issue affecting customers who have a VPN gateway configured and are navigating through the UI. Other customer scenarios remain unaffected. Based on current observations, the overall impact is low.

Postmortem: ## Customer Impact

Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows.

Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required.

Root Cause

A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface.

The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects.

Detection

Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface.

Response

Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally.

Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved.

Follow-up

Short-term

• Enhance testing to automatically test for behaviour of legacy IXP projects
• Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements

Resolved: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

Monitoring: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

Identified: A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects.

During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required.

We will confirm once the rollout is complete.

Investigating: We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change.

This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI.

Importantly, IXP ingestion is not affected, and no action is required from customers at this time.

Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.

Identified: We are validating the reversion of the improper configuration in a lower environment, and will begin deploying it to production regions shortly

Identified: We have identified a defect in a recent deployment, which is causing the issue. We are preparing to roll back the change, which will return Agents to full functionality for all customers.

Investigating: Further investigation has revealed that this issue is affecting all regions, so customers may experience Agents failing to start across all UiPath Automation Cloud regions.

We are continuing to investigate the root cause of the issue and will be working to mitigate it promptly.

Investigating: We are continuing to investigate the cause of the issues, we will provide an update as soon as we begin working on a mitigation

Investigating: We are investigating reports of an outage impacting Agents in US and Europe Regions. Impact: Users may be unable to start Agents. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: We have identified an issue affecting customers who have a VPN gateway configured and are navigating through the UI. Other customer scenarios remain unaffected. Based on current observations, the overall impact is low.

Postmortem: ## Customer Impact

Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows.

Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required.

Root Cause

A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface.

The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects.

Detection

Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface.

Response

Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally.

Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved.

Follow-up

Short-term

• Enhance testing to automatically test for behaviour of legacy IXP projects
• Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements

Resolved: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

Monitoring: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

Identified: A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects.

During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required.

We will confirm once the rollout is complete.

Investigating: We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change.

This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI.

Importantly, IXP ingestion is not affected, and no action is required from customers at this time.

Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.

Identified: We are validating the reversion of the improper configuration in a lower environment, and will begin deploying it to production regions shortly

Identified: We have identified a defect in a recent deployment, which is causing the issue. We are preparing to roll back the change, which will return Agents to full functionality for all customers.

Investigating: Further investigation has revealed that this issue is affecting all regions, so customers may experience Agents failing to start across all UiPath Automation Cloud regions.

We are continuing to investigate the root cause of the issue and will be working to mitigate it promptly.

Investigating: We are continuing to investigate the cause of the issues, we will provide an update as soon as we begin working on a mitigation

Investigating: We are investigating reports of an outage impacting Agents in US and Europe Regions. Impact: Users may be unable to start Agents. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Resolved: A fix has been deployed, and system is stable.

Monitoring: A fix has been deployed, and we are actively monitoring the system.

Identified: A fix has been identified and is currently being implemented.

Investigating: We have identified an issue affecting customers who have a VPN gateway configured and are navigating through the UI. Other customer scenarios remain unaffected. Based on current observations, the overall impact is low.

Postmortem: ## Customer Impact

Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows.

Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required.

Root Cause

A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface.

The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects.

Detection

Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface.

Response

Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally.

Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved.

Follow-up

Short-term

• Enhance testing to automatically test for behaviour of legacy IXP projects
• Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements

Resolved: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

Monitoring: A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

Identified: A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects.

During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required.

We will confirm once the rollout is complete.

Investigating: We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change.

This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI.

Importantly, IXP ingestion is not affected, and no action is required from customers at this time.

Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.