Harness Status

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

Between 8:30 AM and 11:30 AM PST on December 3rd, 2025, customers in the Prod-2 environment experienced issues where some CI builds remained stuck in the Queued state despite being under their concurrency limits. New builds continued running, but previously queued executions did not transition to Running until a fix was deployed.

Root Cause

A recently introduced change in the concurrency queuing logic caused stale messages remain in the Redis queue.These stale messages could not be processed because their corresponding executions had already been aborted and their metadata removed.he dequeue poller repeatedly attempted to acquire a lock for these messages and failed, placing them back at the head of the queue, preventing newer queued executions from being processed.

The missing fallback logic to clean up stale messages resulted in older builds remaining perpetually queued.

Impact

‌

• New builds continued to run normally, but older queued executions did not start until the fix was applied.
• The issue was isolated to a subset of executions for a single customer; other customers and modules did not experience service degradation.

Remediation

‌

Immediate:

• Increased concurrency limits for the affected customer to ensure no new builds were queued.
• Deployed a hotfix that cleaned up stale messages and allowed queued builds to transition to Running.

‌

Permanent:

• Updated the locking flow to detect when execution metadata is missing and safely clean up stale messages

Action Items

1. Add fallback logic to safely delete stale queue messages when lock acquisition fails due to missing execution metadata.
2. Add retries and improved cleanup handling when ACK operations fail.
3. Enhance monitoring to detect queue buildup or repeated unack patterns earlier.
4. Expand testing coverage for concurrency queue edge cases, including aborted pipelines and metadata cleanup races.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

Between 8:30 AM and 11:30 AM PST on December 3rd, 2025, customers in the Prod-2 environment experienced issues where some CI builds remained stuck in the Queued state despite being under their concurrency limits. New builds continued running, but previously queued executions did not transition to Running until a fix was deployed.

Root Cause

A recently introduced change in the concurrency queuing logic caused stale messages remain in the Redis queue.These stale messages could not be processed because their corresponding executions had already been aborted and their metadata removed.he dequeue poller repeatedly attempted to acquire a lock for these messages and failed, placing them back at the head of the queue, preventing newer queued executions from being processed.

The missing fallback logic to clean up stale messages resulted in older builds remaining perpetually queued.

Impact

‌

• New builds continued to run normally, but older queued executions did not start until the fix was applied.
• The issue was isolated to a subset of executions for a single customer; other customers and modules did not experience service degradation.

Remediation

‌

Immediate:

• Increased concurrency limits for the affected customer to ensure no new builds were queued.
• Deployed a hotfix that cleaned up stale messages and allowed queued builds to transition to Running.

‌

Permanent:

• Updated the locking flow to detect when execution metadata is missing and safely clean up stale messages

Action Items

1. Add fallback logic to safely delete stale queue messages when lock acquisition fails due to missing execution metadata.
2. Add retries and improved cleanup handling when ACK operations fail.
3. Enhance monitoring to detect queue buildup or repeated unack patterns earlier.
4. Expand testing coverage for concurrency queue edge cases, including aborted pipelines and metadata cleanup races.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We have implemented a fix and are monitoring.

Investigating: We have are investigating the issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Postmortem: ## Summary

On 12/02/2025 between 9:12:30 AM PDT to 9:26 AM PDT, Harness Gateway service experienced intermittent restarts, which caused temporary disruption in request processing for some customers.

Customer Impact: Harness UI was either unavailable or slow , No running pipelines were affected.

Root Cause

During this period, traffic volume doubled compared to normal levels. This unexpected surge in traffic overwhelmed the database because a key index was missing, which in turn caused the Gateway service to become temporarily unstable.

Mitigation Actions

• Immediate Response

  • We rate limited the traffic surge by implementing filters based on traffic pattern and fingerprints.

• Resolution

  • The missing database index has been added, and the Gateway service is now operating normally and handling increased traffic without issues

Next Steps and Improvements

• To prevent such issues from happening and reducing the impact, additional detection measures to block traffic surges is being implemented.
• We are proactively optimizing the system and adding additional testing and monitoring to ensure that, in the event of unexpected traffic spikes, our services remain performant and reliable

Resolved: This incident has been resolved. We will share the RCA as soon as the analysis is completed.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.