ModernLoop Status

Resolved: The frontend build causing this issue has been reverted.

Resolved: This incident has been resolved.

Monitoring: # Elevated error rates on some read operations — August 3, 2026

Status: Resolved Duration: 16:43 – 17:38 UTC (approximately 55 minutes)

Summary

On August 3, 2026, between 16:43 and 17:38 UTC, some customers experienced intermittent errors and slow responses on a subset of read operations, including loading team member directories, viewing scheduling suggestions, and loading metrics dashboards. Scheduling data and all previously saved information were unaffected, and no data was lost. Retrying the affected pages generally succeeded once the incident resolved.

What happened

A routine database maintenance operation performed during a scheduled deployment placed unusually high load on our database infrastructure. This temporarily delayed our read-replica databases, causing a portion of read queries to time out or be cancelled while the replicas caught up. The initial recovery at 17:15 UTC was followed by a second period of elevated errors while database read traffic rebalanced; error rates fully returned to normal at approximately 17:38 UTC.

Impact

• A subset of read requests failed or responded slowly during the incident window. Affected requests returned errors and could be retried.
• Some customers were temporarily unable to create new schedules during the incident window; those attempts succeeded once the incident resolved. Previously created schedules, tasks, and candidate data were unaffected.
• No data was lost or corrupted.

What we're doing to prevent this

• Scheduling large database maintenance operations outside of peak usage hours.
• Adding automatic retries for the class of transient read errors seen during this incident, so brief replication delays no longer surface as user-facing errors.
• Adding earlier alerting on database replication delay so we can respond before customers are impacted.
• Optimizing the specific queries that were most affected so they are more resilient to brief periods of database pressure.

We apologize for the disruption. If you continue to see errors, please contact [email protected].

Monitoring: A fix has been implemented and we are monitoring the results.

Resolved: # Elevated error rates on some read operations — August 3, 2026

Status: Resolved Duration: 16:43 – 17:15 UTC (approximately 32 minutes)

Summary

On August 3, 2026, between 16:43 and 17:15 UTC, some customers experienced intermittent errors and slow responses on a subset of read operations, including loading team member directories, viewing scheduling suggestions, and loading metrics dashboards. Scheduling data and all previously saved information were unaffected, and no data was lost. Retrying the affected pages generally succeeded once the incident resolved.

What happened

A routine scheduled deployment placed unusually high load on our database infrastructure. This temporarily delayed our read-replica databases, causing a portion of read queries to time out or be cancelled while the replicas caught up. Error rates returned to normal at approximately 17:15 UTC once the maintenance operation completed and replication fully recovered.

Impact

• A subset of read requests failed or responded slowly during the incident window. Affected requests returned errors and could be retried.
• Write operations (creating and updating schedules, tasks, and candidate data) were not materially affected.
• No data was lost or corrupted.

What we're doing to prevent this

• Adding automatic retries for the class of transient read errors seen during this incident, so brief replication delays no longer surface as user-facing errors.
• Adding earlier alerting on database replication delay so we can respond before customers are impacted.
• Optimizing the specific queries that were most affected so they are more resilient to brief periods of database pressure.

We apologize for the disruption. If you continue to see errors, please contact [email protected].

Identified: The issue has been identified and a fix is being implemented.