Planning Center Status

Postmortem: # Post-Mortem: Service Disruption Caused by Excessive Outbound API Requests to Multitracks

Date of Incident: November 9, 2025
Duration: ~1 hour
Impacted: MultiTracks integration; indirect impact to Services and Check-Ins

Overview

On November 9, Planning Center experienced a disruption in our MultiTracks integration caused by an internal change to our caching logic. This change unintentionally triggered a large spike in outbound API requests to MultiTracks, which led to their service throttling our traffic. The resulting failures also caused secondary errors in Planning Center services that rely on responses from that API.

This issue was rooted in Planning Center’s system behavior, and we take full responsibility for the impact.

What Happened

A caching pattern change dramatically increased the number of API requests we sent to MultiTracks. Their service began throttling our traffic, and our retry behavior amplified the load on both systems. This led to elevated errors in the MultiTracks integration and related Planning Center processes.

The impact to customers was somewhat minimized because Music Stand apps use local caching, but some experienced errors, slow responses, or missing data.

Resolution

We quickly identified the regression and deployed a revert of the caching change. Once reverted, our outbound traffic returned to normal and MultiTracks’ throttling stopped, allowing dependent systems to recover.

Root Cause

• A caching change that generated excessive outbound API requests
• Retry and timeout behaviors that worsened cascading failures

Preventive Actions

1. Review and tune third-party timeouts to avoid cascading failures
2. Improve graceful degradation strategies for external dependencies
3. Enhance pre-deployment checks for changes affecting outbound request volume

Current Status

The MultiTracks integration is stable. We are implementing the above improvements to prevent similar issues and ensure our systems remain resilient even when dependencies become unstable.

Resolved: We experienced a service disruption caused by an unexpected spike from outbound API requests from our system to Multitracks. This led to instability in both their API and our own services that rely on it.

A fix has been implemented on our side, and all services should be back to normal.

Identified: The issue has been identified and our team is working on a fix.

Investigating: Users may experience errors or degraded performance when accessing services reliant on the Multitracks API. We are currently investigating the issue to restore full functionality.

Resolved: The Multitracks Team has implemented a fix that fixed an issue with Multitracks searches being down as well as RehearsalMix files returning an error message. Things should be back to normal.

Investigating: Our team is investigating users experiencing issues accessing Rehearsal Mix search features and files not playing correctly or display licensing errors.