ModernLoop Status

Postmortem: Task search outage, August 26, 2026

On August 26 between 12:10 and 1:00 PM Pacific, task pages across ModernLoop were slow or failed to load entirely. This included the dashboard most users see right after signing in. We fixed it by 1:08 PM. No data was lost or exposed, and no scheduled interviews were affected.

What happened

At 12:09 PM we shipped an update that included a fix for a rare bug where task search could return stale entries. The fix had a flaw: it removed a limit on how much of the search index one query could read. After the deploy, every task search read far more of the index than it needed, and each extra read triggered its own database check. Search traffic roughly quadrupled within a minute. Our search infrastructure saturated, requests started running past their time limits, and the pages waiting on them failed.

The frustrating part is that the change was small and looked safe. It passed review and tests. What it didn't pass was a large index under real traffic, because nothing in our pipeline exercised that.

Timeline (Pacific)

• 12:09 PM. Update deployed.
• 12:10 PM. Task search slowed down.
• 12:15 PM. Timeouts reached customers. Monitoring and support tickets flagged it within minutes.
• 12:23 PM. We traced it to the deploy and started on a rollback and a corrected fix in parallel.
• 1:00 PM. We rolled the search service back to the previous version. Timeouts stopped.
• 1:08 PM. Error rates and response times confirmed normal.

‌

‌

What we're changing

We restored the previous version to end the incident, and we rewrote the original fix so it keeps the data-consistency improvement but caps the work a single search can do. The gap that let this ship is the one described above: no load test covers search changes at realistic index sizes, so we're adding one. Rolling back also took longer than it should have because it required someone at a terminal with production access; each service now gets a one-click rollback. And we've added an alert on search-cluster load, so a regression like this pages us at 12:11, not 12:15.

Sorry for the disruption. The affected pages recovered on their own, so there's nothing you need to do.

Resolved: This incident has been resolved.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating this issue.