Close Status

Monitoring: We had an issue with our backend search infrastructure. The issue affected Lead search, Inbox, Opportunities, Leads, and Reporting.

The engineering team deployed a fix, and the items are being reindexed. We are monitoring the situation.

Postmortem: Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

Impact

The Close Web and Desktop applications were intermittently failing to load for from June 18th 22:32 UTC to June 19th 05:10 UTC. The issue started with only a few errors with the frequency gradually increasing until it was resolved.

Root Cause and Resolution

A misconfigured version of the application was partially deployed due to mishandling of failures of a re-run CI/CD job. The issue was resolved by deploying a freshly built version of the application. We are improving our alerting systems to detect such incidents sooner and implementing safeguards into our build process to prevent misconfigured versions from occurring in the first place.

Timeline

• June 18th 22:32 UTC - an already deployed version of the application image gets misconfigured due to an issue in our build system
• as time goes on and our infrastructure cycles resources which causes more and more traffic to get served by the misconfigured version, but not often enough to trigger our alerts
• June 19th 04:17 UTC - Close Engineering team realizes there’s a problem, begins an investigation and starts taking mitigation steps
• June 19th 05:10 UTC - a new version of the application is deployed resolving the issue

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Application is loading again, we're monitoring the situation.

Investigating: We're continuing to investigate the issue.

Investigating: Users are currently unable to load the application UI due to an issue with the serving infrastructure.

Monitoring: We had an issue with our backend search infrastructure. The issue affected Lead search, Inbox, Opportunities, Leads, and Reporting.

The engineering team deployed a fix, and the items are being reindexed. We are monitoring the situation.