Harness Status

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating on the issue

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue. We are currently experiencing an issue affecting Helm-based deployments that are already in progress via Continuous Deployment (CD) pipelines. New Helm executions are not impacted and continue to run successfully.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Postmortem: ## Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups

Resolved: This incident has been resolved.

Monitoring: We have identified and mitigated the issue, and are currently monitoring.

Monitoring: A fix has been implemented and we are monitoring the results.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating on the issue

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating on the issue