Planning Center Status

Postmortem: On Sunday morning, November 17th, 2024, Check-Ins experienced a significant slowdown in performance between 6:55 AM and 7:42 AM PST, causing delays and in some cases, an inability to check people in. We understand that you count on Check-Ins being fast and reliable on Sunday mornings, and we’re deeply sorry for the disruption this caused to you and your church community.

What Happened? Recently, we began a change aimed at removing an unnecessary network request from the printing process, in an effort to both make things faster and make stations more resilient to different network conditions. However, this new solution introduced a new query that was not covered by our database indexes, and as a result was not efficient.

When Sunday’s higher traffic hit, this slow query backed up our servers, slowing performance to an unusable state for all stations.

Timeline of Events:

• 6:55 AM PST: We received alerts about increased latency in Check-Ins.
• 7:21 AM PST: We traced the problem to a change made earlier in the week.
• 7:38 AM PST: We reverted the change and began monitoring for recovery.
• 7:42 AM PST: The revert was complete, load decreased, and response times returned to normal.

How Are We Fixing This? Check-Ins is a vital part of the Sunday morning experience that many churches rely on, and this last weekend’s issues are not up to the standards we adhere to. We’re committed to doing everything we can to prevent this from happening again including changes to both incident responses and monitoring changes. Here’s what we’re doing:

• Adding better alerting to monitor performance changes throughout the week.
• Keeping older application versions available for quicker rollbacks.
• Reviewing and improving our incident communication processes.

We sincerely apologize for the frustration this may have caused. Our top priority is ensuring Check-Ins remains a seamless part of your Sunday morning experience.

Thank you for allowing us to serve your church. We’re here for you, and we’re taking every step to make sure your next experience is fast, smooth, and stress-free. Please feel free to reach out to support@planningcenter.com if you have further questions. We’re happy to help!

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented, and we are monitoring the results.

Identified: The issue has been identified, and we are working on getting a fix in place.

Investigating: Check-Ins is loading very slowly or not at all for many customers. Our team is investigating.