IONOS Cloud Status

Postmortem: This is a preliminary post mortem. We will regularly update it as new information becomes available.

What happened:

On 14.11.2025 between 13:54 and 14:47 UTC we observed a network connectivity issue in our TXL datacenter, affecting multiple services and leading to increased latency and packet loss. 

This happened because: 

A datacenter cluster experienced network instability due to a broken failover in the control plane of our InfiniBand fabrics. The core issue was a constant switching of Master/Standby roles between the redundant InfiniBand Subnet Managers, leading to inconsistencies in the fabric. The initial trigger for this failover loop is still under investigation and we will provide an update here once the root cause is pinpointed. After the Subnet Roll Manager election process was stabilized the situation automatically recovered.

What we are doing to prevent this from happening again:

The exact root cause of the unexpected switching behavior is still under investigation and we will update this RCA and the action item list as new insights are obtained:

1. Further investigation into the Subnet Manager Log Messages to identify the trigger of the unexpected switching behavior (ongoing)
2. Hardening and improvement of the Subnet Manager High Availability (HA) configuration

We understand that the incident has had a negative impact on our customers and partners. We commit to a thorough and transparent investigation of the incident and continuous improvement to provide you with a stable service.

Resolved: All pending systems have reported in as healthy. We will mark this incident as resolved. We will publish the Root Cause Analysis (RCA) here as soon as it becomes available. Thank you for your patience.

Monitoring: We are seeing further improvement across the environment. We are currently actively monitoring the recovery and are confirming the Root Cause of the Incident and will publish the result here.

Identified: We are observing some improvements in our monitoring metrics after our recent intervention, and some previously affected systems are reporting back as healthy. We will continue to monitor the environment closely as we look for any other potential contributing factors. We will provide our next update no later than 15:30 UTC

Identified: We have identified and disabled a defective switch that we believe was a potential culprit. We are now closely monitoring the environment to assess the impact of this action, as we suspect there may be additional contributing factors. Our investigation continues, and we will provide the next update by 15:30 UTC at the latest.

Investigating: We are currently investigating instances of Border Gateway Protocol session flapping in the TXL datacenter. Our network experts are currently pinpointing the root cause of the issue. You can expect the next update here at 15:00 UTC.

Investigating: We are currently investigating reports related to connectivity issues in our TXL datacenter. We will update you as soon as possible.

Resolved: We want to share the following Root Cause Analysis:

What happened: On the 6th of November, our Monitoring Team alerted us to increased error rates for requests to S3 Cloudian in DE/FRA. The increased error rate was caused by a network-level issue related to MTU settings lower than standard, causing intermittent issues for some connections.

What we are doing to prevent this from happening again: On 11.11.2025, a fix will be rolled out to the affected network, which will correct the MTU settings permanently.

Monitoring: Error rate has dropped. We are closely monitoring the situation. We are compiling a root cause analysis.

Investigating: We are currently investigating an increased error rate for requests to S3 Cloudian in DE/FRA.