Cornerstone Status

Postmortem: Incident Summary:
On July 28, 2025, clients reported an issue in the LAX PRD and PIL SL4 environment affecting access to the platform.

Impact:
Clients were unable to log in to the portal during the incident window

Root Cause Analysis (RCA):
The disruption was caused by a failure in a backend database service, which broke application connectivity and interrupted login functionality.
The underlying issue was traced to contention and buildup of audit logs in the database. This led to resource contention within the database engine, which degraded performance and eventually caused the service to become unresponsive. The failure was further compounded by a delay in the automated recovery mechanism, which extended the outage duration.

Corrective Actions:
The engineering team took immediate steps to restore functionality, including clearing impacted service components, restoring access, and validating system stability. Full access to the platform was reinstated following these actions.

Preventive Measures:
To prevent recurrence, the following measures have been implemented:
o System housekeeping routines have been reviewed and reinforced to ensure consistent execution.
o Failover mechanisms are being enhanced to improve responsiveness and recovery time in the event of future disruptions.

Resolved: The CSOD Technology Team observed a service disruption on this swimlane. The problem began at 1:25 PM Pacific Time and service was restored at 1:45 PM Pacific Time. During this time, clients with portals on this swimlane were not able to access the application. After a period of monitoring we are marking this incident resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Customers on LAX SL4 may encounters errors and issues trying to log into Production and Pilot. We are actively investigating the issue as our top priority. We will share regular updates as they become available.

Monitoring: Email delivery continues to flow successfully, and we are closely monitoring activity to ensure sustained stability.

Monitoring: Emails are flowing successfully again. Microsoft lifted the throttling limitation and the queues have cleared. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to work closely with Microsoft to address the email delivery delays. We will continue to share regular updates until this is resolved. This is our top priority. We appreciate your continued patience.

Identified: We are continuing to work closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Identified: While the overall email delivery service for the UK region's production, pilot, and stage swimlanes customers remain unimpacted, customers sending emails specifically to the Microsoft Email Domains, may experience some issues with the email deliveries. We are closely working with the Microsoft Teams to address the email delivery delay issues.

Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Email delivery is now restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in email delivery failures. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: Email delivery continues to flow successfully, and we are closely monitoring activity to ensure sustained stability.

Monitoring: Emails are flowing successfully again. Microsoft lifted the throttling limitation and the queues have cleared. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to work closely with Microsoft to address the email delivery delays. We will continue to share regular updates until this is resolved. This is our top priority. We appreciate your continued patience.

Identified: We are continuing to work closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Identified: While the overall email delivery service for the UK region's production, pilot, and stage swimlanes customers remain unimpacted, customers sending emails specifically to the Microsoft Email Domains, may experience some issues with the email deliveries. We are closely working with the Microsoft Teams to address the email delivery delay issues.

Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Email delivery is now restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in email delivery failures. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: Email delivery continues to flow successfully, and we are closely monitoring activity to ensure sustained stability.

Monitoring: Emails are flowing successfully again. Microsoft lifted the throttling limitation and the queues have cleared. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to work closely with Microsoft to address the email delivery delays. We will continue to share regular updates until this is resolved. This is our top priority. We appreciate your continued patience.

Identified: We are continuing to work closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Identified: While the overall email delivery service for the UK region's production, pilot, and stage swimlanes customers remain unimpacted, customers sending emails specifically to the Microsoft Email Domains, may experience some issues with the email deliveries. We are closely working with the Microsoft Teams to address the email delivery delay issues.

Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Email delivery is now restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in email delivery failures. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: Email delivery continues to flow successfully, and we are closely monitoring activity to ensure sustained stability.

Monitoring: Emails are flowing successfully again. Microsoft lifted the throttling limitation and the queues have cleared. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to work closely with Microsoft to address the email delivery delays. We will continue to share regular updates until this is resolved. This is our top priority. We appreciate your continued patience.

Identified: We are continuing to work closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Identified: While the overall email delivery service for the UK region's production, pilot, and stage swimlanes customers remain unimpacted, customers sending emails specifically to the Microsoft Email Domains, may experience some issues with the email deliveries. We are closely working with the Microsoft Teams to address the email delivery delay issues.

Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Email delivery is now restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.

Identified: The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in email delivery failures. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.