Northpass Status

Postmortem: On September 11, 2025, our platform experienced intermittent disruptions impacting our AWS-hosted customers. These disruptions affected our end-users' ability to interact with courses. The issue was traced to an automated deployment process that unintentionally updated certain backend services to incompatible versions. This created temporary mismatches between services and led to periodic failures. Azure-hosted customers were not affected.

Impact

• Duration:  2 hours (16:24 - 18:23 EDT).
• Pattern: Cyclic issues every few minutes with brief automatic recovery periods.
• Symptoms: Users experienced difficulties accessing courses, completing activities, and general course interactions.
• Affected users: All AWS-hosted customers during problematic service version combinations.

Root Cause

Our deployment automation tool was querying the docker registry for the latest images, but received inconsistent results due to the large number of images in our registry (suspected registry API limitation). This caused the automation to cycle through different microservice image combinations approximately every 10 minutes, creating incompatible service versions that disrupted the interdependent functionality required for course interactions.

Resolution

Our team cleaned up the docker registry by removing old images, significantly reducing the total number of images. This stabilized our deployment automation process and eliminated the cycling behavior.

Next steps to preventing recurrence

Implement automated docker image cleanup policy to maintain registry hygiene.

Resolved: This incident have now been resolved, we apologize for the service interruption. Our team is hard at work on delivering on our promised transparency, we will provide the postmortem of this issue within 48 hours.

Monitoring: A Fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on resolving this issue.

Identified: We've identified the issue and are working towards a resolution.

Investigating: End users are experiencing issues when interacting with courses

Postmortem: # Service Disruption on September 9–10, 2025

Summary

On September 9, 2025, Northpass experienced a service disruption affecting all AWS-hosted customers. Users reported difficulties logging in, accessing courses, and completing activities. Azure-hosted customers were not affected.

Impact

• Duration: ~2 hours of major disruption, followed by slower performance for several more hours
• Affected customers: AWS-hosted environments only
• Symptoms: login errors, slow page loads, delayed certificates, and issues with integrations (e.g., Zoom sessions)

Root Cause

The disruption was caused by our primary database running out of available input/output capacity (IOPS) in AWS. This slowed down critical operations and caused delays across the platform.

Resolution

Our engineering team took immediate action to stabilize the system, including expanding capacity and reducing system load. Once traffic normalized, performance returned to expected levels.

Next Steps (Preventing Recurrence)

We are implementing the following permanent improvements:

• Upgrading our AWS database storage to a higher-performance type with more IOPS capacity
• Improving monitoring and alerting to detect database pressure earlier
• Optimizing how we process background tasks to reduce load during peak usage
• Optimizing database queries to reduce impact on performance and improve reliability

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for issues. Customers and Learners may experience slower page loads as services recover.

Monitoring: We are continuing to monitor for issues. Customers and Learners may experience slower page loads as services recover.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Applications are stable but we are still monitoring and checking for slowness.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue. Admins and end users are experiencing errors when logging in

Investigating: We're experiencing an elevated level of API errors and are currently looking into the issue.