Harness Status

Postmortem: ### Summary

Between approximately 11:37 AM and 3:01 PM PDT (18:37–22:01 UTC) on September 29, 2026, customers using Harness CI self-hosted Linux cloud runners in prod1/prod2 (us-west1) experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code.

Root Cause

This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there.

Impact

• Customers running self-hosted Linux CI builds in us-west1 saw intermittent build/pipeline failures, not a full outage.
• Observed customer-facing errors included Git fetch timeouts, e.g. Failed to connect to github.com port 443: Connection timed out

Remediation

• Immediate: Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 (partially limited by central1 capacity); restarted the affected Cloud NAT gateways, which stabilized pipeline networking.

Action Items

1. Secure additional NAT gateway/IP capacity (In progress)
2. Improve NAT saturation detection and alerting ahead of customer impact.
3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Identified: We identified an issue that applies only to SP-initiated login through the published Harness Okta application. After a recent update to that application, those login attempts returned an HTTP 400 error when the configured SSO URL was missing from Additional ACS URLs. General SAML 2.0 configurations in Okta are not affected.

Immediate Solutions: Use IDP-initiated login. Start the login from the Harness application tile in Okta. Or Add your Harness SSO URL to Additional ACS URLs in the published Harness Okta application.

Identified: We identified an issue that applies only to SP-initiated login through the published Harness Okta application. After a recent update to that application, those login attempts returned an HTTP 400 error when the configured SSO URL was missing from Additional ACS URLs. General SAML 2.0 configurations in Okta are not affected.

Immediate Solutions: Use IDP-initiated login. Start the login from the Harness application tile in Okta. Or Add your Harness SSO URL to Additional ACS URLs in the published Harness Okta application.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: Summary

Customers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. This was caused by an unexpected traffic spike to one of our backend systems. 

Service was restored by scaling out the affected platform component. No data loss or corruption occurred.

‌

Customer Impact

Customers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally.

‌

Root Cause

A large automated provisioning run created a significant number of persistent CV monitoring workers in a short window, which overwhelmed a shared platform component.

The sudden increase in concurrent outbound calls exhausted the available network connections on each service instance.  

‌

Mitigation

The maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out was completed.

‌

Next Steps

To prevent recurrence, Harness will:

• Increase capacity headroom and add provisioning guardrails: Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services.
• Improve Resiliency: Add exponential backoff and jitter for worker retries, and use a reusable connection pool to reduce overhead and prevent transient failures from compounding under high concurrency.
• Improve monitoring and alerting: Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ### Summary

Between approximately 11:37 AM and 3:01 PM PDT (18:37–22:01 UTC) on September 29, 2026, customers using Harness CI self-hosted Linux cloud runners in prod1/prod2 (us-west1) experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code.

Root Cause

This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there.

Impact

• Customers running self-hosted Linux CI builds in us-west1 saw intermittent build/pipeline failures, not a full outage.
• Observed customer-facing errors included Git fetch timeouts, e.g. Failed to connect to github.com port 443: Connection timed out

Remediation

• Immediate: Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 (partially limited by central1 capacity); restarted the affected Cloud NAT gateways, which stabilized pipeline networking.

Action Items

1. Secure additional NAT gateway/IP capacity (In progress)
2. Improve NAT saturation detection and alerting ahead of customer impact.
3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Identified: We identified an issue that applies only to SP-initiated login through the published Harness Okta application. After a recent update to that application, those login attempts returned an HTTP 400 error when the configured SSO URL was missing from Additional ACS URLs. General SAML 2.0 configurations in Okta are not affected.

Immediate Solutions: Use IDP-initiated login. Start the login from the Harness application tile in Okta. Or Add your Harness SSO URL to Additional ACS URLs in the published Harness Okta application.

Identified: We identified an issue that applies only to SP-initiated login through the published Harness Okta application. After a recent update to that application, those login attempts returned an HTTP 400 error when the configured SSO URL was missing from Additional ACS URLs. General SAML 2.0 configurations in Okta are not affected.

Immediate Solutions: Use IDP-initiated login. Start the login from the Harness application tile in Okta. Or Add your Harness SSO URL to Additional ACS URLs in the published Harness Okta application.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.