IONOS Cloud Status

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Resolved: We have marked this incident as resolved. Several underlying VMs entered an unhealthy state, which increased the CPU load on the remaining healthy VMs supporting the service. This cascaded into the intermittent availability issues. All systems are now operating normally.

Identified: We have observed another spike and are setting the incident back to active. Our MK8s Team is informed and are actively working the incident.

Monitoring: Control Planes are reachable again. Our MK8s Team is currently investigating the root cause for the service interruption.

Investigating: We are currently investigating issues with the MK8s control plane

Identified: We were able to reactivate the MK8s maintenances in all clusters in our TXL datacenter

Investigating: Due to exceptionally high demand in two clusters in our TXL and FRA datacenters, we have temporarily paused Managed Kubernetes Automated Maintenance.

To ensure that clusters remain operational without interruption during updates, Automated Maintenance requires temporary cluster "cloning", which increases resource consumption. As most maintenance windows are scheduled to run over the weekend, resource demand peaks sharply during these periods.

To protect cluster stability and prevent service degradation, we have paused maintenance for the affected clusters. Any maintenance scheduled from today onward will not run until we resume.

We know automated Kubernetes maintenance is an important feature of our platform, and we did not take this decision lightly. We are expanding capacity in the affected clusters as a high priority and will update this page as soon as maintenance resumes.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Resolved: We are marking this incident as resolved, as no further abnormalities occurred.

Monitoring: Error rate has dropped. We are monitoring the situation further.

Identified: We are reducing the impact of the incident, as our Object Storage Team has traced back the increased error rates to mainly SlowDown codes, which affect only heavily loaded buckets.

Investigating: We are currently investigating increased error rate for S3 in our FRA DC

Investigating: MongoDB Playground and Business Edition clusters are currently not available for provisioning in de/fra/2 (Frankfurt East). This is due to a capacity limitation in that location that prevents these cluster types from being created reliably, and it is expected to persist for the time being. We will update this status page as soon as this limitation is lifted.

As an alternative, you can:

• Provision your Playground or Business Edition cluster in another available location, or
• Use MongoDB Enterprise Edition, which remains available in Frankfurt East.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We were able to reactivate the MK8s maintenances in all clusters in our TXL datacenter

Investigating: Due to exceptionally high demand in two clusters in our TXL and FRA datacenters, we have temporarily paused Managed Kubernetes Automated Maintenance.

To ensure that clusters remain operational without interruption during updates, Automated Maintenance requires temporary cluster "cloning", which increases resource consumption. As most maintenance windows are scheduled to run over the weekend, resource demand peaks sharply during these periods.

To protect cluster stability and prevent service degradation, we have paused maintenance for the affected clusters. Any maintenance scheduled from today onward will not run until we resume.

We know automated Kubernetes maintenance is an important feature of our platform, and we did not take this decision lightly. We are expanding capacity in the affected clusters as a high priority and will update this page as soon as maintenance resumes.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Identified: We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one.

What You Can Do

• Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available.
• Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success.
• If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server.

Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible.

We sincerely appreciate your patience and understanding.

Investigating: MongoDB Playground and Business Edition clusters are currently not available for provisioning in de/fra/2 (Frankfurt East). This is due to a capacity limitation in that location that prevents these cluster types from being created reliably, and it is expected to persist for the time being. We will update this status page as soon as this limitation is lifted.

As an alternative, you can:

• Provision your Playground or Business Edition cluster in another available location, or
• Use MongoDB Enterprise Edition, which remains available in Frankfurt East.

Identified: The AI Model Hub is currently experiencing high demand. As a shared infrastructure environment, this temporary surge in utilization may result in increased latency and varied performance across the platform.

Current Impacts

• Increased Latency (TTFT): High GPU utilization may cause delayed responses (Time to First Token) when initiating model requests. We are continuously managing platform resources to maximize availability across all concurrent workloads.

• Llama 405B Constraints: Due to its scale and high compute requirements, the Llama 405B model is particularly sensitive to traffic spikes, leading to potential timeouts or higher latency—we recommend switching to alternative models for time-critical workloads requiring faster response times.

• Collections: Collections can also be influenced negatively by high demand and utilization. Please see the notes on new deployments in our documentation: https://docs.ionos.com/cloud/ai/ai-model-hub/how-tos/document-collections

Recommendations:

• To mitigate temporarily performance or availability issues, we encourage our customers and partners to implement (exponential backoff) retry mechanisms in their projects and pipelines. Usage peaks that lead in timeouts and degraded performance are usually transient.
• We kindly ask our customers to not create additional Support Tickets for questions and reports related to the performance of models in the Model Hub. Our Product and Tech Teams are aware and are actively monitoring and working on improving the service to meet demand.
• We ask Llama 405B users that face issues to consider a switch to GTP-OSS-120B instead. In most real world use cases, the model is more cost efficient and delivers a more robust performance

Outlook

The following measures are currently underway to improve the performance of the AI Model Hub:

• Various (model specific) optimizations are rolled out on a regular basis
• Improved cross-GPU load balancing (ETA July)
• Adjustments to error codes (replacing HTTP 5xx with HTTP 429) to allow clients to better understand and react to situations

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the service.

Investigating: We are continuing to investigate this issue.

Investigating: Due to an internal network issue that is currently being investigated, all models are currently unavailable.