Viirtue Status

Postmortem: ## Incident Report: NJ Data Center Network Event & Related Service Impact

Date of Incident: July 14, 2025

Prepared by: Alberto Fernandez (Director of Platform Engineering)

Start Time: 10:53 AM EST

Partial Restoration Time 11:10 AM EST

Full Resolution Time: 2:09 PM EST

Services Impacted: Apollo services across NJ, FL, and Las Vegas data centers

Customer Impact: Potential voice degradation, call setup delays, or dropped sessions for some users during routing transitions.

Executive Summary

At 10:53 AM EST, our monitoring systems detected a sharp and unusual increase in SBUS queue activity within our NJ data center. This triggered an immediate alert to our Network Operations and Engineering teams, who launched a rapid response investigation.

As a precautionary and proactive measure, we rerouted all real-time traffic away from NJ to our geographically redundant data centers in Las Vegas and Florida. This action ensured continuity of service while we investigated the root cause of the anomaly in NJ.

During the transition period, we identified two additional, unrelated issues affecting the Vegas and Florida clusters. Each was diagnosed and mitigated independently. All systems are now fully operational and stable across all locations. Further details on these issues will be provided in the final RCA.

Potential Root Cause

The incident was triggered by a transient physical network cut in our NJ data center’s upstream routing path. While the disruption lasted approximately 5 minutes, it had downstream effects on signaling queues (SBUS) and session routing.

The system behaved as designed by rerouting traffic through redundant sites, but the rapid load redistribution temporarily stressed configuration limits which temporarily impacted service quality.

Initial Remediation

• NJ Network Cut Mitigation: Root cause confirmed with upstream provider. Cable rerouted and physical safeguards enhanced.

• SBUS Monitoring: Sensitivity tuned for improved queue behavior detection.

Customer Communication

We recognize that service interruptions, even brief ones, can be disruptive to your operations. We remain committed to transparency, reliability, and continuous improvement.

Next Steps

• Perform full internal post-incident review.

• Continue monitoring all three data centers at increased granularity for the next 72 hours.

• Deliver a final Root Cause Analysis (RCA) document within 7 business days.

If you have any further questions or would like to discuss the incident in more detail, please contact our support team.

Thank you for your continued trust.

Resolved: All services have been fully restored and are now operational. An incident report will be shared later today, with a full root cause analysis to follow.

Monitoring: We’ve pinpointed the root cause and are currently validating the fix.

Monitoring: We’re continuing to work through service stabilization efforts across both regions. Teams remain engaged and progress is ongoing.

Monitoring: The 503 has been removed from core2-nj, and voice traffic has been redirected to this core.

Investigating: We are aware of the audio issue affecting a subset of calls on the Las Vegas core and are investigating the issue.

Investigating: All voice and web portal traffic directed to the core2-nj core has been redirected to our other cores. Service should improve while we work on finding the root cause.

Investigating: Due to ongoing connectivity issues at our NJ data center, we are proactively returning HTTP 503 responses to divert traffic away as a preventative measure. Services remain operational via our redundant infrastructure.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating an issue causing the manager portal to load slowly. Updates to follow.

Resolved: All services have been restored and are operational.

Monitoring: All services have been fully restored in our NJ data center. We are currently in the process of normalizing the environment, including migrating DNS records back to NJ.

Monitoring: starting maintenance

Monitoring: We will be performing scheduled maintenance on our NJ infrastructure today at 5 PM EST. All services will remain fully operational and will continue running from our Florida data center during this time. The customer portal will also remain accessible and will continue to be served from our FL data center.

Monitoring: All services are fully operational in our secondary data center.

Identified: Our engineers, in collaboration with on-site data center staff, are working to ensure full service restoration. All possible services have been migrated to our Florida data center, and certificate restoration for white-label partners is underway. All other services remain operational. The data center provider has provided an estimated time of resolution for NJ normalization by 1 PM EST.”

Identified: We are continuing to work on a fix for this issue.

Identified: Our engineers have traced the issue to the data center and are coordinating with on-site staff for deeper diagnostics and resolution. The data center reported ongoing preventative UPS maintenance, though it’s not yet confirmed to be related. All services, including the portal, remain fully operational through our redundant Florida data center.

Investigating: Our monitoring systems have identified an issue impacting the NJ data center. Our engineering team is actively investigating and working to resolve the incident.

Postmortem: ## Incident Report: NJ Data Center Network Event & Related Service Impact

Date of Incident: July 14, 2025

Prepared by: Alberto Fernandez (Director of Platform Engineering)

Start Time: 10:53 AM EST

Partial Restoration Time 11:10 AM EST

Full Resolution Time: 2:09 PM EST

Services Impacted: Apollo services across NJ, FL, and Las Vegas data centers

Customer Impact: Potential voice degradation, call setup delays, or dropped sessions for some users during routing transitions.

Executive Summary

At 10:53 AM EST, our monitoring systems detected a sharp and unusual increase in SBUS queue activity within our NJ data center. This triggered an immediate alert to our Network Operations and Engineering teams, who launched a rapid response investigation.

As a precautionary and proactive measure, we rerouted all real-time traffic away from NJ to our geographically redundant data centers in Las Vegas and Florida. This action ensured continuity of service while we investigated the root cause of the anomaly in NJ.

During the transition period, we identified two additional, unrelated issues affecting the Vegas and Florida clusters. Each was diagnosed and mitigated independently. All systems are now fully operational and stable across all locations. Further details on these issues will be provided in the final RCA.

Potential Root Cause

The incident was triggered by a transient physical network cut in our NJ data center’s upstream routing path. While the disruption lasted approximately 5 minutes, it had downstream effects on signaling queues (SBUS) and session routing.

The system behaved as designed by rerouting traffic through redundant sites, but the rapid load redistribution temporarily stressed configuration limits which temporarily impacted service quality.

Initial Remediation

• NJ Network Cut Mitigation: Root cause confirmed with upstream provider. Cable rerouted and physical safeguards enhanced.

• SBUS Monitoring: Sensitivity tuned for improved queue behavior detection.

Customer Communication

We recognize that service interruptions, even brief ones, can be disruptive to your operations. We remain committed to transparency, reliability, and continuous improvement.

Next Steps

• Perform full internal post-incident review.

• Continue monitoring all three data centers at increased granularity for the next 72 hours.

• Deliver a final Root Cause Analysis (RCA) document within 7 business days.

If you have any further questions or would like to discuss the incident in more detail, please contact our support team.

Thank you for your continued trust.

Resolved: All services have been fully restored and are now operational. An incident report will be shared later today, with a full root cause analysis to follow.

Monitoring: We’ve pinpointed the root cause and are currently validating the fix.

Monitoring: We’re continuing to work through service stabilization efforts across both regions. Teams remain engaged and progress is ongoing.

Monitoring: The 503 has been removed from core2-nj, and voice traffic has been redirected to this core.

Investigating: We are aware of the audio issue affecting a subset of calls on the Las Vegas core and are investigating the issue.

Investigating: All voice and web portal traffic directed to the core2-nj core has been redirected to our other cores. Service should improve while we work on finding the root cause.

Investigating: Due to ongoing connectivity issues at our NJ data center, we are proactively returning HTTP 503 responses to divert traffic away as a preventative measure. Services remain operational via our redundant infrastructure.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating an issue causing the manager portal to load slowly. Updates to follow.

Resolved: All services have been restored and are operational.

Monitoring: All services have been fully restored in our NJ data center. We are currently in the process of normalizing the environment, including migrating DNS records back to NJ.

Monitoring: starting maintenance

Monitoring: We will be performing scheduled maintenance on our NJ infrastructure today at 5 PM EST. All services will remain fully operational and will continue running from our Florida data center during this time. The customer portal will also remain accessible and will continue to be served from our FL data center.

Monitoring: All services are fully operational in our secondary data center.

Identified: Our engineers, in collaboration with on-site data center staff, are working to ensure full service restoration. All possible services have been migrated to our Florida data center, and certificate restoration for white-label partners is underway. All other services remain operational. The data center provider has provided an estimated time of resolution for NJ normalization by 1 PM EST.”

Identified: We are continuing to work on a fix for this issue.

Identified: Our engineers have traced the issue to the data center and are coordinating with on-site staff for deeper diagnostics and resolution. The data center reported ongoing preventative UPS maintenance, though it’s not yet confirmed to be related. All services, including the portal, remain fully operational through our redundant Florida data center.

Investigating: Our monitoring systems have identified an issue impacting the NJ data center. Our engineering team is actively investigating and working to resolve the incident.

Postmortem: ## Incident Report: NJ Data Center Network Event & Related Service Impact

Date of Incident: July 14, 2025

Prepared by: Alberto Fernandez (Director of Platform Engineering)

Start Time: 10:53 AM EST

Partial Restoration Time 11:10 AM EST

Full Resolution Time: 2:09 PM EST

Services Impacted: Apollo services across NJ, FL, and Las Vegas data centers

Customer Impact: Potential voice degradation, call setup delays, or dropped sessions for some users during routing transitions.

Executive Summary

At 10:53 AM EST, our monitoring systems detected a sharp and unusual increase in SBUS queue activity within our NJ data center. This triggered an immediate alert to our Network Operations and Engineering teams, who launched a rapid response investigation.

As a precautionary and proactive measure, we rerouted all real-time traffic away from NJ to our geographically redundant data centers in Las Vegas and Florida. This action ensured continuity of service while we investigated the root cause of the anomaly in NJ.

During the transition period, we identified two additional, unrelated issues affecting the Vegas and Florida clusters. Each was diagnosed and mitigated independently. All systems are now fully operational and stable across all locations. Further details on these issues will be provided in the final RCA.

Potential Root Cause

The incident was triggered by a transient physical network cut in our NJ data center’s upstream routing path. While the disruption lasted approximately 5 minutes, it had downstream effects on signaling queues (SBUS) and session routing.

The system behaved as designed by rerouting traffic through redundant sites, but the rapid load redistribution temporarily stressed configuration limits which temporarily impacted service quality.

Initial Remediation

• NJ Network Cut Mitigation: Root cause confirmed with upstream provider. Cable rerouted and physical safeguards enhanced.

• SBUS Monitoring: Sensitivity tuned for improved queue behavior detection.

Customer Communication

We recognize that service interruptions, even brief ones, can be disruptive to your operations. We remain committed to transparency, reliability, and continuous improvement.

Next Steps

• Perform full internal post-incident review.

• Continue monitoring all three data centers at increased granularity for the next 72 hours.

• Deliver a final Root Cause Analysis (RCA) document within 7 business days.

If you have any further questions or would like to discuss the incident in more detail, please contact our support team.

Thank you for your continued trust.

Resolved: All services have been fully restored and are now operational. An incident report will be shared later today, with a full root cause analysis to follow.

Monitoring: We’ve pinpointed the root cause and are currently validating the fix.

Monitoring: We’re continuing to work through service stabilization efforts across both regions. Teams remain engaged and progress is ongoing.

Monitoring: The 503 has been removed from core2-nj, and voice traffic has been redirected to this core.

Investigating: We are aware of the audio issue affecting a subset of calls on the Las Vegas core and are investigating the issue.

Investigating: All voice and web portal traffic directed to the core2-nj core has been redirected to our other cores. Service should improve while we work on finding the root cause.

Investigating: Due to ongoing connectivity issues at our NJ data center, we are proactively returning HTTP 503 responses to divert traffic away as a preventative measure. Services remain operational via our redundant infrastructure.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently investigating an issue causing the manager portal to load slowly. Updates to follow.