Harness Status: check for Harness outages and issues

Harness status is Operational

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery (CD) - FirstGen - EOS

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

See all 106 Harness components

Last updated 1 minute ago from official status page. Learn more

Stay ahead of Harness outages

Active Incidents

Stale Data Observed for Custom Dashboards in Prod1

Started 26 Mar 2025 03:28:24 (6 days ago), resolved 26 Mar 2025 04:26:08 (6 days ago)

Minor Incident

Postmortem

Custom Dashboards

We are currently investigating this issue.

Postmortem: ### Summary

On March 25, 2025, for 2 hours and 22 minutes, customers in the prod-1 production environment observed stale data on the following custom dashboards: pipeline executions, stage executions, and step executions.

What was the issue?

The metadata state tables managing the ETL process were corrupted during a version upgrade, requiring fixes to this table. No data was lost during this process.

Resolution

The metadata state was reset to trigger data mart updates.

‌

Time(UTC)	Event
26 Mar 2:04 AM	We identified the ETL process that timed out after the upgrade.
26 Mar 3:18 PM	Redeployed the ETL process, applied the plan, and recreated the views.
26 Mar 4:22 AM	The metadata schema was rebuilt, and all data quality checks were confirmed to be passing.
26 Mar 4:25 AM	The incident was resolved.

‌

RCA

‌

Plan application errors were due to an upgrade of ETL process timing out after running for two hours. This resulted in metadata corruption, requiring data fixing. While no data loss was experienced, data staleness was observed because the data marts were not updated with the latest ETL intervals during the metadata recreation.

‌

Action Items

Update the ETL framework frequently to avoid significant version number jumps**.**
Set up a regular cadence for testing new updates and deploying them into production.

1 Apr 2025 10:43:39 (2 hours ago)

Resolved: This incident has been resolved. Thanks for your patience.

26 Mar 2025 04:26:08 (6 days ago)

Identified: We are working towards testing a fix in our dev environment.

26 Mar 2025 04:13:13 (6 days ago)

Identified: We are continuing to work on a fix for the issue.

26 Mar 2025 03:28:24 (6 days ago)

Identified: We are continuing to work on a fix for the issue.

26 Mar 2025 02:42:58 (6 days ago)

Identified: We are working on a fix. We have identified that only Unified Dashboards for pipeline, stage, and steps are currently impacted.

26 Mar 2025 02:05:47 (6 days ago)

Investigating: We are currently investigating this issue.

26 Mar 2025 02:04:03 (6 days ago)

See all updates

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Custom dashboards are not loading in Prod1, Prod2 and Prod3 because of our Managed service - Looker facing an outage currently. We will monitor this outage and update once we have a status update from Looker.

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

Identified: We are observing a gradual recovery of the Looker service. Some dashboards are now rendering correctly. However, a partial outage remains in effect. We will continue to monitor the situation and provide updates as they become available.

26 Mar 2025 00:56:13 (6 days ago)

Identified: Custom dashboards are not loading in Prod1, Prod2 and Prod3 because of our Managed service - Looker facing an outage currently. We will monitor this outage and update once we have a status update from Looker.

26 Mar 2025 00:17:25 (7 days ago)

See all updates

Recently Resolved Incidents

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Started 31 Mar 2025 18:54:00 (18 hours ago), resolved 31 Mar 2025 20:22:37 (16 hours ago)

Minor Incident

Resolved

Continuous Delivery - Next Generation (CDNG)

We are currently investigating login failures in prod4.

See all updates

Started 31 Mar 2025 18:54:09 (18 hours ago), resolved 31 Mar 2025 19:09:41 (17 hours ago)

Minor Incident

Resolved

Continuous Delivery - Next Generation (CDNG)

We are currently investigating login failures in prod4.

See all updates

Harness Outage Survival Guide

A step-by-step guide to help you survive a Harness outage

NaN%

Harness Components

Harness

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Prod 0

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Prod 1

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery (CD) - FirstGen - EOS

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

26 Mar 2025 00:56:13 (6 days ago)

26 Mar 2025 00:17:25 (7 days ago)

See all updates

Stale Data Observed for Custom Dashboards in Prod1

Started 26 Mar 2025 03:28:24 (6 days ago), resolved 26 Mar 2025 04:26:08 (6 days ago)

Minor Incident

Postmortem

Custom Dashboards

We are currently investigating this issue.

Postmortem: ### Summary

What was the issue?

The metadata state tables managing the ETL process were corrupted during a version upgrade, requiring fixes to this table. No data was lost during this process.

Resolution

The metadata state was reset to trigger data mart updates.

‌

Time(UTC)	Event
26 Mar 2:04 AM	We identified the ETL process that timed out after the upgrade.
26 Mar 3:18 PM	Redeployed the ETL process, applied the plan, and recreated the views.
26 Mar 4:22 AM	The metadata schema was rebuilt, and all data quality checks were confirmed to be passing.
26 Mar 4:25 AM	The incident was resolved.

‌

RCA

‌

Action Items

Update the ETL framework frequently to avoid significant version number jumps**.**
Set up a regular cadence for testing new updates and deploying them into production.

1 Apr 2025 10:43:39 (2 hours ago)

Resolved: This incident has been resolved. Thanks for your patience.

26 Mar 2025 04:26:08 (6 days ago)

Identified: We are working towards testing a fix in our dev environment.

26 Mar 2025 04:13:13 (6 days ago)

Identified: We are continuing to work on a fix for the issue.

26 Mar 2025 03:28:24 (6 days ago)

Identified: We are continuing to work on a fix for the issue.

26 Mar 2025 02:42:58 (6 days ago)

Identified: We are working on a fix. We have identified that only Unified Dashboards for pipeline, stage, and steps are currently impacted.

26 Mar 2025 02:05:47 (6 days ago)

Investigating: We are currently investigating this issue.

26 Mar 2025 02:04:03 (6 days ago)

See all updates

Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Supply Chain Assurance (SSCA)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Engineering Insights (SEI)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Code Repository

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Prod 2

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery (CD) - FirstGen - EOS

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

26 Mar 2025 00:56:13 (6 days ago)

26 Mar 2025 00:17:25 (7 days ago)

See all updates

Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Supply Chain Assurance (SSCA)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Engineering Insights (SEI)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Code Repository

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Prod 3

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery (CD) - FirstGen - EOS

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

26 Mar 2025 00:56:13 (6 days ago)

26 Mar 2025 00:17:25 (7 days ago)

See all updates

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Software Supply Chain Assurance (SSCA)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Software Engineering Insights (SEI)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Code Repository

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

PROD3: UI not loading

Started 1 Apr 2025 07:05:32 (5 hours ago), resolved 1 Apr 2025 07:41:49 (5 hours ago)

Incident without Impact

Resolved

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

More...

A fix has been implemented and we are monitoring.

See all updates

Harness Prod 4

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Started 31 Mar 2025 18:54:00 (18 hours ago), resolved 31 Mar 2025 20:22:37 (16 hours ago)

Minor Incident

Resolved

Continuous Delivery - Next Generation (CDNG)

We are currently investigating login failures in prod4.

See all updates

Started 31 Mar 2025 18:54:09 (18 hours ago), resolved 31 Mar 2025 19:09:41 (17 hours ago)

Minor Incident

Resolved

Continuous Delivery - Next Generation (CDNG)

We are currently investigating login failures in prod4.

See all updates

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

26 Mar 2025 00:56:13 (6 days ago)

26 Mar 2025 00:17:25 (7 days ago)

See all updates

Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Code Repository

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Prod Eu1

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Delivery - Next Generation (CDNG)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Continuous Integration Enterprise(CIE) - Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Cloud Cost Management (CCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Error Tracking (CET)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Chaos Engineering

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom Dashboards

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Started 26 Mar 2025 00:17:25 (7 days ago), resolved 26 Mar 2025 01:29:57 (6 days ago)

Critical Incident

Postmortem

Custom Dashboards

Postmortem: ### Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

31 Mar 2025 22:24:04 (14 hours ago)

Resolved: The dashboards are now rendering correctly with the recovery of Looker Service.

26 Mar 2025 01:29:57 (6 days ago)

Monitoring: The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

26 Mar 2025 01:23:04 (6 days ago)

26 Mar 2025 00:56:13 (6 days ago)

26 Mar 2025 00:17:25 (7 days ago)

See all updates

Feature Flags (FF)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Security Testing Orchestration (STO)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Service Reliability Management (SRM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Internal Developer Portal (IDP)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Infrastructure as Code Management (IaCM)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Engineering Insights (SEI)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Service Reliability Management - Error Tracking FirstGen (fka OverOps)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Software Engineering Insights FirstGen (fka Propelo)

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Engineering Insights FirstGen (fka Propelo) - EU

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Software Engineering Insights FirstGen (fka Propelo) - US

Tue 25

Wed 26

Thu 27

Fri 28

Sat 29

Sun 30

Mon 31

now

Harness Status

Real-time updates of Harness issues and outages

Harness status is Operational

Custom Dashboards

Custom Dashboards

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Active Incidents

What was the issue?

Resolution

‌

RCA

Action Items

What was the issue?

RCA

Action Items

Recently Resolved Incidents

Harness Outage Survival Guide

Harness Components

Harness

Harness Prod 0

Harness Prod 1

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Chaos Engineering

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Custom Dashboards

What was the issue?

RCA

Action Items

What was the issue?

Resolution

‌

RCA

Action Items

Feature Flags (FF)

Security Testing Orchestration (STO)

Service Reliability Management (SRM)

Internal Developer Portal (IDP)

Infrastructure as Code Management (IaCM)

Software Supply Chain Assurance (SSCA)

Software Engineering Insights (SEI)

Code Repository

Harness Prod 2

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Chaos Engineering

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Continuous Integration Enterprise(CIE) - Mac Cloud Builds

Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Continuous Integration Enterprise(CIE) - Linux Cloud Builds

Custom Dashboards

What was the issue?

RCA

Action Items

Feature Flags (FF)

Security Testing Orchestration (STO)

Service Reliability Management (SRM)

Internal Developer Portal (IDP)

Infrastructure as Code Management (IaCM)

Software Supply Chain Assurance (SSCA)

Software Engineering Insights (SEI)

Code Repository

Harness Prod 3

Continuous Delivery (CD) - FirstGen - EOS

Continuous Delivery - Next Generation (CDNG)

Cloud Cost Management (CCM)

Continuous Error Tracking (CET)

Continuous Integration Enterprise(CIE) - Self Hosted Runners

Continuous Integration Enterprise(CIE) - Mac Cloud Builds