Pusher Status

Resolved: This incident has been resolved.

Monitoring: All services are now fully operational. Our team continues to monitor the system closely to ensure ongoing stability and performance.

Identified: Our team has identified the root cause of the issue and implemented a fix. The affected cluster is now in the process of recovering, and performance is gradually returning to normal. We will continue to closely monitor the situation until full recovery is confirmed.

Identified: We are currently experiencing a major outage in US3 cluster. Our engineering team is actively working to restore full functionality as quickly as possible. In the meantime, we recommend customers temporarily switch to an alternative cluster to minimize service disruption.

Identified: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We're seeing an increased latency and a number of 503 errors on us3 cluster. The team is investigating.

Resolved: This incident has been resolved.

Investigating: Our cloud provider is currently experiencing an incident that's causing increased latency on Pusher Channels main cluster mt1 . This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Postmortem: ## Root Cause Analysis: Elevated API Errors and Outage in AP4 Cluster

Incident Date: October 20, 2025

Status: Resolved

Summary

Between October 20 and October 21, 2025, customers using the AP4 cluster experienced elevated API errors, latency, and message publishing failures. The issue primarily affected the Channels API, preventing customers from publishing new messages and leading to degraded real-time functionality for end-users.

During system recovery and while implementing mitigations from a previous incident on Oct 18th, a misconfigured Redis container in the AP4 cluster failed to start correctly, preventing caching operations needed for API requests. This misconfiguration went undetected by proactive monitoring, delaying full recovery until October 21 at 17:43 UTC.

Impact

Throughout the incident period, customers in the AP4 cluster experienced:

• High API error rates when attempting to publish messages through the Channels API
• Failed or delayed message delivery for connected clients
• Temporary downtime for end-customer applications relying on real-time messages

Other clusters remained operational, though some minor latency was observed in isolated regions due to dependencies on shared services.

Root Cause

This incident resulted from a chain of events involving both external and internal factors:

1. **Major AWS Outage (October 20)**A large-scale AWS outage in the US-East region disrupted multiple dependent systems, impacting several Pusher clusters.
2. Misconfigured Redis Container (October 21) As systems in the AP4 cluster attempted to scale during recovery, one of the backend Redis cache containers failed to start due to a misconfigured environment variable. This prevented Redis from initializing properly, resulting in API operations failing or timing out.
3. Monitoring Gap Existing monitoring did not capture the Redis startup failure because the specific failure mode occurred after initialization checks had passed. This delayed internal detection until API error rates increased and customer impact was observed.
4. Delayed Customer Communication Initial updates to customers were delayed while the team triaged the issue and verified the failure pattern, prolonging the time before external notification.

Detection and Response

The issue was detected through a combination of monitoring alerts showing elevated error rates and customer reports of publishing failures.

Timeline of Events:

• October 20 – AWS outage began, affecting multiple Pusher clusters leading to increased delays and errors. October 20, evening UTC – Pusher clusters began recovery as AWS services were restored.
• October 21, 15:06 UTC – Internal monitoring detected elevated API errors in AP4; engineers began investigation.  Incident was unrelated to prior AWS outage.
• October 21, 15:09 UTC – Root cause identified as a failed Redis caching container.
• October 21, 15:27 UTC – Restoration of Redis connections underway.
• October 21, 15:29 UTC – Fix implemented; cluster began gradual recovery.
• October 21, 17:39 UTC – Full stabilization confirmed across AP4 nodes.
• October 21, 17:43 UTC – Incident marked resolved after sustained recovery.

Resolution

To restore full functionality, the engineering team:

• Corrected the Redis container configuration preventing startup
• Restarted and validated cache services across all AP4 nodes
• Confirmed API endpoints were fully operational and message publishing resumed
• Monitored latency and error metrics to confirm sustained stability

Preventative Actions

To reduce recurrence risk and improve detection and response, Pusher is implementing the following:

• Enhanced Redis Monitoring: Extending monitoring coverage to detect Redis startup and post-init failures.
• Customer Communication Enhancements: Improving internal escalation and communication processes to ensure faster external updates.

Next Steps and Commitment

We recognize the importance of reliable API performance for our customers. Our teams are conducting a full review of caching dependencies and configuration management across all clusters to prevent similar incidents.

We sincerely apologize for the disruption caused by this event and appreciate your patience as we worked through a complex multi-day recovery scenario. Pusher remains committed to transparency, reliability, and continuous improvement in service resilience.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are currently investigating an issue affecting the Channels API. A large number of customers may be unable to publish new messages through the API in the AP4 cluster.

Monitoring: A fix has been implemented and we are monitoring. The cluster will take a few more minutes to fully stabilize across all nodes.

Identified: The team continues to make progress restoring the cache services. We anticipate full resolution in the next 10-15 minutes.

Identified: The team has identified the issue with one of our backend caching servers. The team is working to restore the connections to the server.

Investigating: We're experiencing an elevated level of API errors and latency in AP4 and are currently looking into the issue.

Resolved: This incident has been resolved.

Monitoring: All services are now fully operational. Our team continues to monitor the system closely to ensure ongoing stability and performance.

Identified: Our team has identified the root cause of the issue and implemented a fix. The affected cluster is now in the process of recovering, and performance is gradually returning to normal. We will continue to closely monitor the situation until full recovery is confirmed.

Identified: We are currently experiencing a major outage in US3 cluster. Our engineering team is actively working to restore full functionality as quickly as possible. In the meantime, we recommend customers temporarily switch to an alternative cluster to minimize service disruption.

Identified: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We're seeing an increased latency and a number of 503 errors on us3 cluster. The team is investigating.

Resolved: This incident has been resolved.

Investigating: Our cloud provider is currently experiencing an incident that's causing increased latency on Pusher Channels main cluster mt1 . This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Postmortem: ## Root Cause Analysis: Increased Latency and Message Delivery Failures – US2 Cluster

Incident Date: October 19, 2025

Duration: 00:44 UTC – 08:17 UTC

Status: Resolved

Summary

On October 19, 2025, customers using Pusher experienced increased latency and message delivery failures. These issues primarily affected the US2 cluster, with intermittent impact also observed in the MT1 and US3 clusters.  The incident resulted in delayed or undelivered messages for many applications.

Latency stabilized at 08:17 UTC after mitigation actions were completed.

Impact

Between 00:44 UTC and 08:17 UTC, multiple customers experienced:

• Delayed or failed message delivery across affected clusters
• Degraded performance in connection establishment and publishing

The most significant and prolonged impact occurred in the US2 cluster, while MT1 and US3 clusters saw elevated latency for a shorter period before stabilizing.

Root Cause

The primary cause of the incident was IP address saturation within the subnet assigned to the public Pusher clusters.

When traffic levels increased, the US2 cluster was unable to scale out further because the available IP addresses in its subnet were fully utilized. This IP scaling limitation prevented the creation of additional instances needed to handle the load.

Secondary factors included temporary network saturation and capacity limits at our cloud provider, which amplified the latency in the early stages of the incident.

Detection and Response

The issue was first detected through a combination of customer reports and internal monitoring alerts showing elevated response times and connection errors.

The timeline of actions was as follows:

• 00:44 UTC – Monitoring alerted the team to increased latency across MT1, US2, and US3 clusters.
• 03:11 UTC – Engineers identified subnet capacity as a contributing factor; mitigations began.
• 04:03 UTC – All clusters were scaled out to distribute traffic; latency began to improve in MT1 and US3 clusters.
• 08:17 UTC – Manual intervention allowed US2 to scale successfully, restoring normal latency.

Resolution

To restore service, the engineering team:

• Scaled out MT1 and US3 clusters to handle increased traffic loads
• Monitored all clusters to confirm sustained stability

Once additional capacity was provisioned and high loads normalized, latency levels returned to normal and remained stable.

Preventative Actions

To prevent recurrence, Pusher has initiated the following actions:

• Rate Limits: We will re-evaluate how rate limits are implemented to better mitigate content from neighboring customers on the shared clusters.
• Subnet Expansion: Re-evaluating and increasing the size of subnets assigned to shared clusters to ensure sufficient IP availability for future scaling events.
• Load Balancer Enhancements: We will implement load balance sharding in order to better distribute connections.
• Capacity Planning Improvements: Enhancing internal monitoring and alerting for subnet and IP utilization thresholds.

Next Steps and Commitment

We recognize that message latency and delivery reliability are critical to our customers’ applications. Our team is continuing a full review of cluster capacity management and provider configuration to improve resilience under high traffic conditions.

We apologize for the disruption this incident caused and appreciate your patience while we worked to resolve it. Ensuring reliability and transparency remains our highest priority.

Resolved: Latency has remained stable over the past three hours. Our investigation identified multiple contributing factors to the earlier latency increase, including network saturation and capacity limits at our cloud providers affecting the US2 cluster. The US3 cluster issue was resolved quickly through autoscaling, while the US2 cluster experienced scaling delays that required manual intervention. We are implementing corrective measures to prevent recurrence and improve system resilience.

Identified: We have scaled all three clusters out to handle the increased traffic. We anticipate the latency to slowly improve.

Identified: The issue has been identified and we are applying mitigations. We are seeing improved latency in main and us3 clusters. us2 still has elevated latency.

Investigating: Increased latency and degraded performance on clusters main, us2 and us3.

Postmortem: ## Root Cause Analysis: Elevated API Errors and Outage in AP4 Cluster

Incident Date: October 20, 2025

Status: Resolved

Summary

Between October 20 and October 21, 2025, customers using the AP4 cluster experienced elevated API errors, latency, and message publishing failures. The issue primarily affected the Channels API, preventing customers from publishing new messages and leading to degraded real-time functionality for end-users.

During system recovery and while implementing mitigations from a previous incident on Oct 18th, a misconfigured Redis container in the AP4 cluster failed to start correctly, preventing caching operations needed for API requests. This misconfiguration went undetected by proactive monitoring, delaying full recovery until October 21 at 17:43 UTC.

Impact

Throughout the incident period, customers in the AP4 cluster experienced:

• High API error rates when attempting to publish messages through the Channels API
• Failed or delayed message delivery for connected clients
• Temporary downtime for end-customer applications relying on real-time messages

Other clusters remained operational, though some minor latency was observed in isolated regions due to dependencies on shared services.

Root Cause

This incident resulted from a chain of events involving both external and internal factors:

1. **Major AWS Outage (October 20)**A large-scale AWS outage in the US-East region disrupted multiple dependent systems, impacting several Pusher clusters.
2. Misconfigured Redis Container (October 21) As systems in the AP4 cluster attempted to scale during recovery, one of the backend Redis cache containers failed to start due to a misconfigured environment variable. This prevented Redis from initializing properly, resulting in API operations failing or timing out.
3. Monitoring Gap Existing monitoring did not capture the Redis startup failure because the specific failure mode occurred after initialization checks had passed. This delayed internal detection until API error rates increased and customer impact was observed.
4. Delayed Customer Communication Initial updates to customers were delayed while the team triaged the issue and verified the failure pattern, prolonging the time before external notification.

Detection and Response

The issue was detected through a combination of monitoring alerts showing elevated error rates and customer reports of publishing failures.

Timeline of Events:

• October 20 – AWS outage began, affecting multiple Pusher clusters leading to increased delays and errors. October 20, evening UTC – Pusher clusters began recovery as AWS services were restored.
• October 21, 15:06 UTC – Internal monitoring detected elevated API errors in AP4; engineers began investigation.  Incident was unrelated to prior AWS outage.
• October 21, 15:09 UTC – Root cause identified as a failed Redis caching container.
• October 21, 15:27 UTC – Restoration of Redis connections underway.
• October 21, 15:29 UTC – Fix implemented; cluster began gradual recovery.
• October 21, 17:39 UTC – Full stabilization confirmed across AP4 nodes.
• October 21, 17:43 UTC – Incident marked resolved after sustained recovery.

Resolution

To restore full functionality, the engineering team:

• Corrected the Redis container configuration preventing startup
• Restarted and validated cache services across all AP4 nodes
• Confirmed API endpoints were fully operational and message publishing resumed
• Monitored latency and error metrics to confirm sustained stability

Preventative Actions

To reduce recurrence risk and improve detection and response, Pusher is implementing the following:

• Enhanced Redis Monitoring: Extending monitoring coverage to detect Redis startup and post-init failures.
• Customer Communication Enhancements: Improving internal escalation and communication processes to ensure faster external updates.

Next Steps and Commitment

We recognize the importance of reliable API performance for our customers. Our teams are conducting a full review of caching dependencies and configuration management across all clusters to prevent similar incidents.

We sincerely apologize for the disruption caused by this event and appreciate your patience as we worked through a complex multi-day recovery scenario. Pusher remains committed to transparency, reliability, and continuous improvement in service resilience.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are currently investigating an issue affecting the Channels API. A large number of customers may be unable to publish new messages through the API in the AP4 cluster.

Monitoring: A fix has been implemented and we are monitoring. The cluster will take a few more minutes to fully stabilize across all nodes.

Identified: The team continues to make progress restoring the cache services. We anticipate full resolution in the next 10-15 minutes.

Identified: The team has identified the issue with one of our backend caching servers. The team is working to restore the connections to the server.

Investigating: We're experiencing an elevated level of API errors and latency in AP4 and are currently looking into the issue.

Resolved: This incident has been resolved.

Investigating: Our cloud provider is currently experiencing an incident that's causing increased latency on Pusher Channels main cluster mt1 . This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Postmortem: ## Root Cause Analysis: Elevated API Errors and Outage in AP4 Cluster

Incident Date: October 20, 2025

Status: Resolved

Summary

Between October 20 and October 21, 2025, customers using the AP4 cluster experienced elevated API errors, latency, and message publishing failures. The issue primarily affected the Channels API, preventing customers from publishing new messages and leading to degraded real-time functionality for end-users.

During system recovery and while implementing mitigations from a previous incident on Oct 18th, a misconfigured Redis container in the AP4 cluster failed to start correctly, preventing caching operations needed for API requests. This misconfiguration went undetected by proactive monitoring, delaying full recovery until October 21 at 17:43 UTC.

Impact

Throughout the incident period, customers in the AP4 cluster experienced:

• High API error rates when attempting to publish messages through the Channels API
• Failed or delayed message delivery for connected clients
• Temporary downtime for end-customer applications relying on real-time messages

Other clusters remained operational, though some minor latency was observed in isolated regions due to dependencies on shared services.

Root Cause

This incident resulted from a chain of events involving both external and internal factors:

1. **Major AWS Outage (October 20)**A large-scale AWS outage in the US-East region disrupted multiple dependent systems, impacting several Pusher clusters.
2. Misconfigured Redis Container (October 21) As systems in the AP4 cluster attempted to scale during recovery, one of the backend Redis cache containers failed to start due to a misconfigured environment variable. This prevented Redis from initializing properly, resulting in API operations failing or timing out.
3. Monitoring Gap Existing monitoring did not capture the Redis startup failure because the specific failure mode occurred after initialization checks had passed. This delayed internal detection until API error rates increased and customer impact was observed.
4. Delayed Customer Communication Initial updates to customers were delayed while the team triaged the issue and verified the failure pattern, prolonging the time before external notification.

Detection and Response

The issue was detected through a combination of monitoring alerts showing elevated error rates and customer reports of publishing failures.

Timeline of Events:

• October 20 – AWS outage began, affecting multiple Pusher clusters leading to increased delays and errors. October 20, evening UTC – Pusher clusters began recovery as AWS services were restored.
• October 21, 15:06 UTC – Internal monitoring detected elevated API errors in AP4; engineers began investigation.  Incident was unrelated to prior AWS outage.
• October 21, 15:09 UTC – Root cause identified as a failed Redis caching container.
• October 21, 15:27 UTC – Restoration of Redis connections underway.
• October 21, 15:29 UTC – Fix implemented; cluster began gradual recovery.
• October 21, 17:39 UTC – Full stabilization confirmed across AP4 nodes.
• October 21, 17:43 UTC – Incident marked resolved after sustained recovery.

Resolution

To restore full functionality, the engineering team:

• Corrected the Redis container configuration preventing startup
• Restarted and validated cache services across all AP4 nodes
• Confirmed API endpoints were fully operational and message publishing resumed
• Monitored latency and error metrics to confirm sustained stability

Preventative Actions

To reduce recurrence risk and improve detection and response, Pusher is implementing the following:

• Enhanced Redis Monitoring: Extending monitoring coverage to detect Redis startup and post-init failures.
• Customer Communication Enhancements: Improving internal escalation and communication processes to ensure faster external updates.

Next Steps and Commitment

We recognize the importance of reliable API performance for our customers. Our teams are conducting a full review of caching dependencies and configuration management across all clusters to prevent similar incidents.

We sincerely apologize for the disruption caused by this event and appreciate your patience as we worked through a complex multi-day recovery scenario. Pusher remains committed to transparency, reliability, and continuous improvement in service resilience.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are currently investigating an issue affecting the Channels API. A large number of customers may be unable to publish new messages through the API in the AP4 cluster.

Monitoring: A fix has been implemented and we are monitoring. The cluster will take a few more minutes to fully stabilize across all nodes.

Identified: The team continues to make progress restoring the cache services. We anticipate full resolution in the next 10-15 minutes.

Identified: The team has identified the issue with one of our backend caching servers. The team is working to restore the connections to the server.

Investigating: We're experiencing an elevated level of API errors and latency in AP4 and are currently looking into the issue.

Resolved: This incident has been resolved.

Monitoring: All services are now fully operational. Our team continues to monitor the system closely to ensure ongoing stability and performance.

Identified: Our team has identified the root cause of the issue and implemented a fix. The affected cluster is now in the process of recovering, and performance is gradually returning to normal. We will continue to closely monitor the situation until full recovery is confirmed.

Identified: We are currently experiencing a major outage in US3 cluster. Our engineering team is actively working to restore full functionality as quickly as possible. In the meantime, we recommend customers temporarily switch to an alternative cluster to minimize service disruption.

Identified: We are continuing to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We're seeing an increased latency and a number of 503 errors on us3 cluster. The team is investigating.

Resolved: This incident has been resolved.

Investigating: Our cloud provider is currently experiencing an incident that's causing increased latency on Pusher Channels main cluster mt1 . This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Postmortem: ## Root Cause Analysis: Increased Latency and Message Delivery Failures – US2 Cluster

Incident Date: October 19, 2025

Duration: 00:44 UTC – 08:17 UTC

Status: Resolved

Summary

On October 19, 2025, customers using Pusher experienced increased latency and message delivery failures. These issues primarily affected the US2 cluster, with intermittent impact also observed in the MT1 and US3 clusters.  The incident resulted in delayed or undelivered messages for many applications.

Latency stabilized at 08:17 UTC after mitigation actions were completed.

Impact

Between 00:44 UTC and 08:17 UTC, multiple customers experienced:

• Delayed or failed message delivery across affected clusters
• Degraded performance in connection establishment and publishing

The most significant and prolonged impact occurred in the US2 cluster, while MT1 and US3 clusters saw elevated latency for a shorter period before stabilizing.

Root Cause

The primary cause of the incident was IP address saturation within the subnet assigned to the public Pusher clusters.

When traffic levels increased, the US2 cluster was unable to scale out further because the available IP addresses in its subnet were fully utilized. This IP scaling limitation prevented the creation of additional instances needed to handle the load.

Secondary factors included temporary network saturation and capacity limits at our cloud provider, which amplified the latency in the early stages of the incident.

Detection and Response

The issue was first detected through a combination of customer reports and internal monitoring alerts showing elevated response times and connection errors.

The timeline of actions was as follows:

• 00:44 UTC – Monitoring alerted the team to increased latency across MT1, US2, and US3 clusters.
• 03:11 UTC – Engineers identified subnet capacity as a contributing factor; mitigations began.
• 04:03 UTC – All clusters were scaled out to distribute traffic; latency began to improve in MT1 and US3 clusters.
• 08:17 UTC – Manual intervention allowed US2 to scale successfully, restoring normal latency.

Resolution

To restore service, the engineering team:

• Scaled out MT1 and US3 clusters to handle increased traffic loads
• Monitored all clusters to confirm sustained stability

Once additional capacity was provisioned and high loads normalized, latency levels returned to normal and remained stable.

Preventative Actions

To prevent recurrence, Pusher has initiated the following actions:

• Rate Limits: We will re-evaluate how rate limits are implemented to better mitigate content from neighboring customers on the shared clusters.
• Subnet Expansion: Re-evaluating and increasing the size of subnets assigned to shared clusters to ensure sufficient IP availability for future scaling events.
• Load Balancer Enhancements: We will implement load balance sharding in order to better distribute connections.
• Capacity Planning Improvements: Enhancing internal monitoring and alerting for subnet and IP utilization thresholds.

Next Steps and Commitment

We recognize that message latency and delivery reliability are critical to our customers’ applications. Our team is continuing a full review of cluster capacity management and provider configuration to improve resilience under high traffic conditions.

We apologize for the disruption this incident caused and appreciate your patience while we worked to resolve it. Ensuring reliability and transparency remains our highest priority.

Resolved: Latency has remained stable over the past three hours. Our investigation identified multiple contributing factors to the earlier latency increase, including network saturation and capacity limits at our cloud providers affecting the US2 cluster. The US3 cluster issue was resolved quickly through autoscaling, while the US2 cluster experienced scaling delays that required manual intervention. We are implementing corrective measures to prevent recurrence and improve system resilience.

Identified: We have scaled all three clusters out to handle the increased traffic. We anticipate the latency to slowly improve.

Identified: The issue has been identified and we are applying mitigations. We are seeing improved latency in main and us3 clusters. us2 still has elevated latency.

Investigating: Increased latency and degraded performance on clusters main, us2 and us3.

Resolved: This incident has been resolved.

Monitoring: Our cloud provider is reporting recovery across most affected services. On our side, we haven’t noticed any issues in the past hour, and all services are operating normally. We continue to monitor the situation closely and will provide updates if anything changes.

Monitoring: The impact on our services is decreasing and we have not observed any additional issues with other services. We continue to monitor the situation closely.

Investigating: Our cloud provider is currently experiencing an incident that's causing increased latency and webhook delivery failures. This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Resolved: This incident has been resolved.

Investigating: Our cloud provider is currently experiencing an incident that's affecting Pusher Beams operations. This issue is external to our systems and is not related to our infrastructure. Our team is actively investigating to determine if there are any additional impacts and working to minimize downstream effects while our provider implements a fix.

Resolved: Our cloud provider is reporting recovery across most affected services. On our side, we haven’t noticed any issues in the past hour, and all services are operating normally. We continue to monitor the situation closely and will provide updates if anything changes.

Monitoring: The impact on our services is decreasing and we have not observed any additional issues with other services. We continue to monitor the situation closely.

Investigating: Our cloud provider is currently experiencing an incident affecting webhook deliveries in Beams. This issue is external to our infrastructure and is limited to webhooks. Other Beams functionality remains operational. Our team is actively investigating to identify any additional impacts while monitoring the provider's resolution progress.