IONOS Cloud Status

Resolved: As redundancy has been fully restored, we are resolving this incident. A RCA is being prepared and we estimate that it will be shared here today or tomorrow.

Monitoring: Our team has recovered the remaining VMs. It’s important to note that storage redundancy is not yet fully restored. Due to necessary data synchronization processes and the volume of data involved, full recovery will take several additional hours.

Important notice for affected customers:

• Load: To ensure stability of the recovered VMs, please try to avoid excessive storage load where possible.
• Provisioning: Provisioning jobs against affected VMs will not be possible until redundancy is fully restored. Consequently, making state changes will not be possible during this time.
• VM States: During the incident, the state of affected VMs may have changed; some may currently be in "shutoff" state.
• Manual Intervention: To avoid unintended impact on customers for whom "shutoff" is the desired state, we cannot automatically restart all VMs without approval.

Due to these constraints, we kindly ask affected customers to carefully check their resources and request state changes via support ticket ([email protected]) where necessary. This is only needed until redundancy is restored.

We are moving this incident to Monitoring status while work to restore full redundancy continues. We will provide an update here once that work is complete. A detailed Root Cause Analysis (RCA) will be published subsequently, explaining the cause of the incident, the measures taken to restore service, and the steps implemented to prevent recurrence.

We thank our customers and partners for their patience and cooperation in ensuring the full recovery of affected services and workloads.

Identified: We have successfully recovered additional VMs. However, because full storage redundancy has not yet been reached for these systems, Storage Provisioning Jobs will currently fail. We ask affected customers to refrain from making configuration changes or placing excessive load on these servers until redundancy is fully restored.

Identified: The team has succeeded in restoring the first VMs during the initial run. We are currently working on rolling out the restoration process further and will update on the progress.

Identified: Our development team is working on re-establishing connectivity sessions of the affected VMs so that the recovery effort can continue. The first runs are currently starting.

Identified: The team has encountered a blocker while restarting the remaining VMs due to connectivity issues between the replaced hosts and the storage servers. This has significantly increased the complexity of the recovery effort. We are currently collaborating with a specialized development team to bring the remaining systems back online. We will provide the next update at 20:00 UTC or before if there are meaningful updates.

Identified: We have made progress regarding the recovery, and are currently in the process of restarting affected Virtual Machines.

Identified: The spares have been made available. They are currently brought online to allow recovery to continue. We estimate that we will have another update on the progress within the hour.

Identified: We found relevant hardware to be defective, this has to be replaced first in the data center. After the physical connection is back, we need to sync and update the storage servers to get the VMs back online. We are working as fast as we can to get this fixed, however, this might take some hours.

Identified: Our storage team is currently working on a firmware restore and upgrade on the affected storage servers and will then attempt to bring them back into service.

Investigating: Unfortunately, we ran into further issues with our Storages in Karlsruhe, so the team keeps on repairing. Affected VMs (and VDCs) are not available and have to be restarted once they get back online. We will keep you updated asap on any new information.

Monitoring: All VMs and Storages and VDCs are back online and available. We keep monitoring the situation.

Identified: Our developers restored the storage availability, we are now restarting the affected VMs to bring them back online.

Investigating: We are currently experiencing restrictions regarding our Storage service in one cluster in our Karlsruhe region. We are working to restore regularly operation of our services as quickly as possible.

Resolved: Redundancy has been fully restored and no limitations related to provisioning should remain.

Identified: As documented in the updates to the recent Storage Incident in our Karlsruhe DC, several Virtual Data Centers (VDCs) are currently restricted from issuing new provisioning jobs while redundancy is being restored for affected VMs. Original Incident: https://status.ionos.cloud/incidents/p6pjqxzgkh1g

With this announcement we want to increase awareness about this current limitation.

Impact Scope: Only customers and VDCs impacted by the original storage incident are affected by these provisioning limitations. Symptoms: Users may encounter error code VDC-5-12 when attempting to create or modify resources. Existing Workloads: All currently running VMs remain operational

We will close this announcement once redundancy, as well as provisioning functionality, is fully restored.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are currently investigating service degradation for Control Planes of some Kubernetes clusters in the Frankfurt region (de/fra). This could impact K8s workload changes for several customers. Our teams are working on a fix.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are writing to inform you that we have been experiencing sporadic connection issues and substantial delays on packet delivery.

Network technicians have started working on the occurrence immediately after detection and will isolate the problem and solve the issue as quick as possible. However, it is possible that there will be a certain degradation in connection quality affecting individual virtual resources.

We will inform you as soon as the functionality has been restored.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some customers may experience connection problems to the control plane and degraded functionality of kubernetes

Resolved: This incident has been resolved, and all affected customers have been contacted directly via email.

Identified: We can confirm that these emails were sent out in error.

There are no customer 'to-do's / calls-to-action, and we will send out an informational update to all affected customers as soon as possible.

Investigating: We are aware that, starting this morning, multiple customers have received an automated email from our Accounting Team advising that their Umsatzsteuer- / VAT ID numbers could not be verified.

We are currently investigating this issue, and expect to send out follow-up emails to the affected customers once we know more.