IONOS Cloud Status

Resolved: This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state. A Root Cause Analysis (RCA) is underway and will be published here once finalized.

Monitoring: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.

Identified: We are marking DBaaS as recovered. Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.

Identified: We are marking the Container Registry Service as recovered.

Identified: We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.

Identified: We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.

Identified: Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.

Identified: We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.

Identified: The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes. Our teams are investigating and working on a resolution.

Postmortem: What happened?

A network control-plane failure in the TXL data center caused progressive service degradation and a partial outage for one cluster. The incident resulted in intermittent connectivity loss for a subset of customers, with traffic impact ranging from 5% to 20% during three distinct intervals on 11.03.2026:

• 05:20 – 05:25 UTC
• 11:58 – 12:10 UT
• 17:47 – 18:08 UTC

The issue resolved automatically once the affected network devices were rebooted sequentially and additional configuration changes have been applied. The incidents prompted a series of emergency maintenance to stabilize the cluster.

How was this possible? (Root Cause)

The underlying cause was the exhaustion of multicast forwarding resources on the switches serving the affected topology. This trigger is technically identical to the previous incident that occurred on 03.03.2026.

When the forwarding table reaches its maximum capacity, the network control plane cannot program required updates into the forwarding tables of the switches. This continuous failure to push updates overwhelms the system, resulting in severe CPU overload and an out-of-memory (OOM) crash of the control-plane process. Without this process, the fabric is unable to maintain stable forwarding, ultimately leading to BGP session flaps and the observed loss of connectivity.

What are we doing to prevent recurrence?

As part of the measures established following the 03.03.2026 incident, we had already rolled out configuration changes to one of the two topologies in the affected cluster. These measures aim to significantly reduce the load of the control-plane by optimizing resweep and failover times. Due to existing instabilities in the second topology, these changes had not yet been applied there.

In response to the degradations observed on 11.03.2026, we accelerated the rollout of these optimizations via Emergency Maintenance across all TXL topologies and several other data centers yesterday. Post-implementation, we observed a significant positive impact, with OpenSM loads being substantially reduced. Furthermore we’ve reconfigured management services to significantly reduce the multicast forwarding entries resulting in a significantly lower base load of the control-plane process.

Immediate Technical Actions:

• Resource Management: Adjusted control-plane configurations to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table. (DONE)
• Multicast Load Reduction: Implemented measures to decrease the number of multicast group memberships, further reducing pressure on the forwarding table. (DONE)
• Proactive Monitoring: Established continuous monitoring of forwarding table utilization and control-plane memory to trigger early alerts before capacity limits are reached. (DONE)

Long-term Structural Improvements:

• Control-Plane Isolation: We are migrating our control-plane to dedicated, high-performance servers. This work, which began in Q4 2025, ensures the separation of network management from data-plane traffic to eliminate resource competition. We are also performing a deep-dive audit of IPv6 configurations and IPoIB driver settings. (Ongoing, ETA: Q2 2026)
• Network Modernization: Our interconnect fabric is undergoing a strategic modernization program—including switches, gateways, and drivers—to increase both resiliency and performance. (Ongoing, ETA: Q3 2026)

With these measures now in place, we are confident the immediate cause of the cluster instability has been mitigated. Our long-term strategy will further enhance network stability and scalability across all data centers while addressing the identified bottlenecks.

We recognize that these disruptions have impacted our customers and partners, and we sincerely appreciate your patience regarding the short-notice emergency maintenance announced yesterday. We are continuing to monitor the cluster closely to ensure all deployed fixes remain effective.

Resolved: We are marking this incident as resolved. Our Network Team will publish an RCA here, once it is compiled.

Monitoring: We are placing the incident in a monitoring state. Our Network Team is closely monitoring the cluster and working to restore full redundancy.

Identified: The deployed change has had positive effect. We are downgrading the impact level while continuing to monitor the cluster closely.

Identified: In response to monitoring alerts, our network team deployed a change to stabilize the network in the affected cluster. We will post another update at 19:00 UTC—or as soon as new information becomes available.

Investigating: We are currently investigating network connectivity issues in our TXL datacenter.

Resolved: This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state. A Root Cause Analysis (RCA) is underway and will be published here once finalized.

Monitoring: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.

Identified: We are marking DBaaS as recovered. Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.

Identified: We are marking the Container Registry Service as recovered.

Identified: We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.

Identified: We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.

Identified: Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.

Identified: We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.

Identified: The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes. Our teams are investigating and working on a resolution.

Resolved: We are marking this incident as resolved. The incident was caused by capacity constraints following a hardware failure. While capacity has been restored, we still see some usage‑specific constraints with the Llama 3.1 405B Instruct model. Our AI ModelHub team will deploy optimizations to the model to increase performance and reliability. We recommend that users still experiencing issues with the model check GPT‑OSS 120B as a potential (temporary) replacement.

Monitoring: Our AI Model Hub Team has mitigated the incident. While the underlying root cause is not yet fully established or resolved, the model service should be stable. We are monitoring the situation while the investigation is ongoing

Identified: The team has identified the root cause: hardware degradation affecting this model's hosting environment is causing backend instability. We are currently implementing a fix.

Investigating: Our Model Hub Team is currently working on resolving errors related to an instance running the llama 405b model.

Resolved: To improve visibility and streamline communication, we are merging this incident into the MK8s incident (tracked here: https://status.ionos.cloud/incidents/h9x5s66m4r28). Consequently, we will close this specific entry and provide all future updates via the referenced incident link.

Identified: The team has traced the root cause to the ongoing Kubernetes incident (link). Both teams are currently working to restore service. (https://status.ionos.cloud/incidents/h9x5s66m4r28)

Investigating: Our AI Model Hub Team is currently investigating increased error rates in the Embeddings functionality in the AI Model Hub.

Resolved: This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state. A Root Cause Analysis (RCA) is underway and will be published here once finalized.

Monitoring: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.

Identified: We are marking DBaaS as recovered. Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.

Identified: We are marking the Container Registry Service as recovered.

Identified: We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.

Identified: We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.

Identified: Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.

Identified: We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.

Identified: The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes. Our teams are investigating and working on a resolution.

Resolved: This incident is now marked resolved, as all affected control planes have returned to and stayed in a stable state. A Root Cause Analysis (RCA) is underway and will be published here once finalized.

Monitoring: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: Final changes have been applied to all afffected clusters resolving the issue. We are now monitoring the progress

Identified: The changes applied to some control plane clusters have had positive effects. The team is continuing the rollout to other affected clusters.

Identified: We are marking DBaaS as recovered. Our Kubernetes Team is currently working on stabilizing the Kubernetes Control Plane. We are focusing on mitigating recurring load spikes influencing stability.

Identified: We are marking the Container Registry Service as recovered.

Identified: We are closing the incident for the AI Model Hub. All metrics have recovered and the service should be up and running again normally.

Identified: We are adding the Container Registry as an affected Service. Customers may currently experience issues pulling and pushing images from the Registry.

Identified: Our Kubernetes Team has deployed a fix for the affected AI Model Hub Database Services. We currently see metrics improving and monitoring the situation closely.

Identified: We are expanding the scope of this incident to include DBaaS and AI Model Hub. We have observed an increased error count originating from PostgresDB on Kubernetes. Additionally, to improve transparency, the previously reported separate incident regarding the AI Model Hub (https://status.ionos.cloud/incidents/rmgs845klm32) is being merged into this primary incident.

Identified: The team has identified the root cause as a resource constraint within the etcd database. Mitigation efforts are currently underway.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes. Our teams are investigating and working on a resolution.