Viirtue Status

Monitoring: Incident Report – Interim RCA Date: April 8, 2025 Prepared by: Alberto Fernandez, Director Platform Engineering

Overview:

On Monday, April 7, 2025, we identified an issue impacting our New Jersey (NJ) data center, which was experiencing unusually high bandwidth utilization. Upon discovery, our engineering team took immediate action to expand the circuits in the NJ facility and initiated an in-depth investigation to determine the source of the increased traffic.

Initial Response:

We promptly performed sanity checks on the existing circuits to rule out any physical degradation or faults and proceeded to upgrade the circuits. Despite these mitigations, the bandwidth consumption remained abnormally high even with seemingly lower usage levels.

Root Cause Analysis – Ongoing:

Further investigation revealed that a network interface located in our Las Vegas (Vegas) data center was introducing packet drops. These packet losses negatively affected the SBus messaging system, which facilitates communication between all three of our cores. As a result, the SBus queue began to grow excessively, causing retransmissions and creating systemic side effects, including degraded performance in data centers that were otherwise healthy.

Corrective Action:

To prevent the Vegas site from continuing to destabilize the broader infrastructure, we made the strategic decision to take the Vegas core offline. Prior to this, we carefully migrated customers out of the Vegas environment to avoid service disruption. During the migration, however, we observed that some customers were manually reverting their configurations to route traffic back through the Vegas data center. These actions inadvertently nullified our remediation efforts.

To preserve the integrity of the infrastructure and accelerate recovery, we made the decision to disable the portal feature allowing customers to manually select their data center location. This control will remain disabled while we restore all environments to a healthy and fully optimized state.

Next Steps:

We are actively developing a remediation and validation plan to address the networking issues within the Vegas data center. This plan includes a comprehensive round of testing to ensure long-term stability before reintegrating the Vegas site back into our production ecosystem.

We strongly recommend that all customers whitelist the IP addresses of all three data centers within their firewall configurations. This ensures seamless service continuity in the event that we need to rehome services to a different location as part of our platform’s stability and automated recovery processes.

Additionally, we advise against manually configuring phones, as this can lead to inconsistencies and potential service disruptions. Instead, we encourage the use of our automated provisioning systems and DNS SRV records, which are designed with built-in self-healing capabilities to enhance reliability and minimize downtime. Transparency and Forward Commitment:

This document serves as an interim incident report and is not intended to serve as the final Root Cause Analysis (RCA). A formal RCA will be published once all issues are fully resolved and the affected infrastructure is confirmed to be stable.

Our commitment to service excellence includes maintaining transparency during incidents. We will continue to provide timely updates and documentation to ensure our partners and customers are fully informed throughout the process.

We appreciate your patience and understanding as we work diligently to restore full operational integrity across our environment.

Identified: As part of our continued commitment to enhancing the stability, reliability, and overall performance of our platform, we are implementing an important change that will affect how data center locations are assigned for your domains.

To reduce latency, minimize inter-region and geo-redundant calls, and streamline backend operations, we will be removing the option to manually select the data center location for your domains through the portal. Going forward, all domains will be automatically assigned to the optimal data center based on a combination of system performance metrics, geographic proximity, and redundancy best practices. This change is being made with your service experience in mind. By centralizing and automating data center assignment, we can significantly: • Improve overall service reliability and uptime • Reduce the risk of cross-region routing issues • Enhance failover efficiency and response times • Simplify management and reduce potential for configuration conflicts

We understand that some users may have previously utilized manual data center selection to align with specific internal preferences. However, our new intelligent assignment model ensures a more consistent and high-performing experience for all customers without requiring manual input.

This update will take effect today. There will be no impact to your existing domain functionality, and you will continue to receive the same level of service and support you expect from us.

If you have any questions or concerns about this update, please don’t hesitate to reach out to our support team at [support contact info]. Thank you for your understanding and continued trust in our platform. We’re confident this improvement will contribute to an even more stable and seamless experience moving forward.

Identified: Engineers are relocating devices away from the Las Vegas core to safely troubleshoot the network issues affecting this node. Updates will follow.

Identified: Service has been restored to the Las Vegas core.

Identified: Due to network issues at the Las Vegas data center, the Las Vegas core will remain offline. The server will be brought back online once the network has been tested and validated.

Identified: Service has been restored in the New Jersey data center.

Identified: We will be temporarily redirecting traffic from the New Jersey data center to perform network tests. Updates to follow.

Identified: The New Jersey data center experienced a network event earlier this morning that has since cleared. This event could have caused delays with the Xima integration. Updates to follow.

Identified: The network issue has been identified and corrected. Services in Las Vegas have been fully restored and are now operating normally.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the network issue and are working on a solution. Updates to follow.

Investigating: Due to network connectivity issues between the New Jersey and Las Vegas data centers, we have decided to 503 the Las Vegas core. All traffic directed to the Las Vegas core will be redirected to the Florida core. If you have any questions or concerns, please reach out to [email protected] or call 1-833-VIIRTUE

Monitoring: Incident Report – Interim RCA Date: April 8, 2025 Prepared by: Alberto Fernandez, Director Platform Engineering

Overview:

On Monday, April 7, 2025, we identified an issue impacting our New Jersey (NJ) data center, which was experiencing unusually high bandwidth utilization. Upon discovery, our engineering team took immediate action to expand the circuits in the NJ facility and initiated an in-depth investigation to determine the source of the increased traffic.

Initial Response:

We promptly performed sanity checks on the existing circuits to rule out any physical degradation or faults and proceeded to upgrade the circuits. Despite these mitigations, the bandwidth consumption remained abnormally high even with seemingly lower usage levels.

Root Cause Analysis – Ongoing:

Further investigation revealed that a network interface located in our Las Vegas (Vegas) data center was introducing packet drops. These packet losses negatively affected the SBus messaging system, which facilitates communication between all three of our cores. As a result, the SBus queue began to grow excessively, causing retransmissions and creating systemic side effects, including degraded performance in data centers that were otherwise healthy.

Corrective Action:

To prevent the Vegas site from continuing to destabilize the broader infrastructure, we made the strategic decision to take the Vegas core offline. Prior to this, we carefully migrated customers out of the Vegas environment to avoid service disruption. During the migration, however, we observed that some customers were manually reverting their configurations to route traffic back through the Vegas data center. These actions inadvertently nullified our remediation efforts.

To preserve the integrity of the infrastructure and accelerate recovery, we made the decision to disable the portal feature allowing customers to manually select their data center location. This control will remain disabled while we restore all environments to a healthy and fully optimized state.

Next Steps:

We are actively developing a remediation and validation plan to address the networking issues within the Vegas data center. This plan includes a comprehensive round of testing to ensure long-term stability before reintegrating the Vegas site back into our production ecosystem.

We strongly recommend that all customers whitelist the IP addresses of all three data centers within their firewall configurations. This ensures seamless service continuity in the event that we need to rehome services to a different location as part of our platform’s stability and automated recovery processes.

Additionally, we advise against manually configuring phones, as this can lead to inconsistencies and potential service disruptions. Instead, we encourage the use of our automated provisioning systems and DNS SRV records, which are designed with built-in self-healing capabilities to enhance reliability and minimize downtime. Transparency and Forward Commitment:

This document serves as an interim incident report and is not intended to serve as the final Root Cause Analysis (RCA). A formal RCA will be published once all issues are fully resolved and the affected infrastructure is confirmed to be stable.

Our commitment to service excellence includes maintaining transparency during incidents. We will continue to provide timely updates and documentation to ensure our partners and customers are fully informed throughout the process.

We appreciate your patience and understanding as we work diligently to restore full operational integrity across our environment.

Identified: As part of our continued commitment to enhancing the stability, reliability, and overall performance of our platform, we are implementing an important change that will affect how data center locations are assigned for your domains.

To reduce latency, minimize inter-region and geo-redundant calls, and streamline backend operations, we will be removing the option to manually select the data center location for your domains through the portal. Going forward, all domains will be automatically assigned to the optimal data center based on a combination of system performance metrics, geographic proximity, and redundancy best practices. This change is being made with your service experience in mind. By centralizing and automating data center assignment, we can significantly: • Improve overall service reliability and uptime • Reduce the risk of cross-region routing issues • Enhance failover efficiency and response times • Simplify management and reduce potential for configuration conflicts

We understand that some users may have previously utilized manual data center selection to align with specific internal preferences. However, our new intelligent assignment model ensures a more consistent and high-performing experience for all customers without requiring manual input.

This update will take effect today. There will be no impact to your existing domain functionality, and you will continue to receive the same level of service and support you expect from us.

If you have any questions or concerns about this update, please don’t hesitate to reach out to our support team at [support contact info]. Thank you for your understanding and continued trust in our platform. We’re confident this improvement will contribute to an even more stable and seamless experience moving forward.

Identified: Engineers are relocating devices away from the Las Vegas core to safely troubleshoot the network issues affecting this node. Updates will follow.

Identified: Service has been restored to the Las Vegas core.

Identified: Due to network issues at the Las Vegas data center, the Las Vegas core will remain offline. The server will be brought back online once the network has been tested and validated.

Identified: Service has been restored in the New Jersey data center.

Identified: We will be temporarily redirecting traffic from the New Jersey data center to perform network tests. Updates to follow.

Identified: The New Jersey data center experienced a network event earlier this morning that has since cleared. This event could have caused delays with the Xima integration. Updates to follow.

Identified: The network issue has been identified and corrected. Services in Las Vegas have been fully restored and are now operating normally.

Identified: We are continuing to work on a fix for this issue.

Identified: We have identified the cause of the network issue and are working on a solution. Updates to follow.

Investigating: Due to network connectivity issues between the New Jersey and Las Vegas data centers, we have decided to 503 the Las Vegas core. All traffic directed to the Las Vegas core will be redirected to the Florida core. If you have any questions or concerns, please reach out to [email protected] or call 1-833-VIIRTUE