Scaleway Status

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Resolved: This incident has been resolved.

Monitoring: We are currently experiencing issues launching actions on Instances in the pl-waw-1 region. This may result in actions not being launched and 500 errors on the API. We are monitoring the situation closely and will provide updates as soon as possible.

Investigating: We are currently experiencing issues launching actions on Instances in the pl-waw-1 region. This may result in actions not being launched and 500 errors on the API.

Resolved: The network switch has been changed and the network is operational again.

Investigating: Network connectivity issues are currently affecting Elastic Metal and Dedibox services in the fr-par-2 region, in room 7 rack A69. We are currently investigating this issue.

Investigating: Customers may experience issues when attempting to install Elastic Metal and Dedibox. An error occurs during the download of the preseed file, preventing the installation from completing.

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Resolved: This incident has been resolved.

Investigating: We experienced 503 errors and connection interruptions with Serverless Containers/Functions in the PL-WAW region from 10:17:40 UTC to 10:18:22 UTC. This was caused by an internal maintenance issue on our side. The issue has been identified and resolved.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Resolved: This incident has been resolved.

Identified: Our teams have found the source of the problem and are actively working on a fix.

Investigating: Every attempt to create a Redis HA / Cluster instance fails.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Resolved: Savings plans is now fixed properly Impacted clients on November invoice have been compensated accordingly

Investigating: An investigation has revealed that PLAY2 instances were not included in the compute savings plan scope for November. This issue is currently being looked into.

Resolved: The Llama 3.3 70b Instruct model is experiencing slowness, resulting in 504 timeout errors for most requests. We have identified and resolved the issue. The service is now operating normally.

Investigating: Llama 3.3 70b Instruct model is experiencing slowness, resulting in 504 timeout errors for most requests. We are actively investigating the root cause.

Resolved: The Llama 3.3 70b Instruct model is experiencing slowness, resulting in 504 timeout errors for most requests. We have identified and resolved the issue. The service is now operating normally.

Investigating: Llama 3.3 70b Instruct model is experiencing slowness, resulting in 504 timeout errors for most requests. We are actively investigating the root cause.

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Monitoring: The situation is now stable, we have been able to workaround all impacts and have means to identify and fix issues quickly. We also have been able to reproduce the issues in our testing environments and now are digging in fixing the root causes. We will provide more details in the coming days. Thanks for your patience.

Monitoring: Here is a checkpoint following issues within VPC networks in Region FR-PAR that occurred in the last few days.

From Tuesday 25th, a few issues were raised to our customer support involving connectivity issues in VPC networks. Some cases were self solved by restarting impacted nodes and/or updated IPs/MACs information. At the time, it did not appear to be out of ordinary issues.

On Wednesday 26th, more cases were raised and escalated, raising awareness of a potential more widespread bug that could disrupt some edge cases customers, engineers were involved to collect data and investigate cases. At 12:45 CET, one of VPC Edge routers crashed. A priority 2 incident was opened to engage more resources and analysis.

Huge levels of BUM (Broadcast/Unicast/Multicast) traffic cross-AZ and cross-products were also identified. In an attempt to solve these issues, the impacted VPC Edge router was fixed and re-added to the production pool to ensure enough computing resources were available to handle the BUM traffic. This reduced the global level of BUM traffic in all impacted VPC, leading us to believe that the situation was under control.

On Wednesday afternoon, more cases of latency and connectivity issues were raised to customer support. Investigations kept going on and pinpointed some desynchronization issues between VPC Edge routers and information systems that could explain higher BUM level than usual. Decision to restart VPC Edge routers services in fr-par-2 was made in an attempt to clear inconsistencies and freshen data on the routers. After a few perturbations, the situation was stable by Wednesday evening. We kept monitoring BUM levels and decision was made to accelerate a deployment of new devices that was already being prepared (more CPU & memory to avoid saturation of VPC Edge routers in case of failover scenarios).

Thursday 27th, new devices were added in the morning in an attempt to better handle BUM traffic. While this showed useful (reducing impacts), it did not significantly help globally in reducing BUM levels. In the afternoon, a workaround was deployed to protect our VPC Edge routers from these excessive BUM traffic. A rate-limiter was applied, per customer network, to ensure a no saturation scenario on VPC Edge routers and make sure legit BUM traffic was handled with the right priority in the routers. This actually caused a high reduction of BUM traffic, as we hoped (stopping what we believe was a snowball effect).

On Thursday evening, deployment of the rate-limiter was globally done and operations were stopped to monitor the situation. One conclusion of the day is that our maintenance model for VPC Edge routers seems to create ripple effects on customer networks (while it was designed to be fully transparent). Friday 28th, investigations are still ongoing to understand the root causes of the issues, high levels of BUM traffic are supposed to be handled transparently, and desynchronization issues are not an expected behavior. Data collecting on one of the VPC Edge routers caused a crash around 11:00 CET, causing ripple effects, as we understood the day before. Data collection and analysis kept going on during the day, including customer debug sessions to better understand some specific cases. During the afternoon, a human error was made by 15:50 CET causing another VPC Edge router unavailability, but we were able to restore it faster and with fewer ripple effects this time.

So, the current situation is that we were able to significantly reduce BUM levels globally, and we believe these are related to desynchronization issues of our VPC Edge routers (causing “unknown MAC and/or IP” issues, so a broadcast behavior). These desynchronizations are probably the cause of our maintenance model issues (traffic is not being treated equally/fairly by redundant VPC Edge routers).

Our teams are working on reducing desynchronization issues (short term) and fixing the underlying condition once identified (mid-term). Decision was made to stop any action on VPC Edge routers for the weekend, so all the investigation work will be done off the routers with the currently collected data to avoid creating more perturbations.

We believe the situation to be stable for almost all customers, but our VPC infrastructure in FR-PAR region cannot be considered in a fully stable condition, and we deeply apologize for this.

We want to assure you that all of our teams are focused on fixing the issues. We encourage customers to share all collected data with our customer support team if you still encounter issues.

Monitoring: From 10:26 CET to 11:11 CET a VPC equipment triggered errors. You may have seen packet loss during this time.

Monitoring: Our teams have deployed a fix on FR-PAR-1. The situation is now under monitoring and stable.

Identified: Our teams have deployed a fix on FR-PAR-2 and are continuing to work on deploying one on FR-PAR-1 as well.

Identified: We are still deploying a fix, during this deployement you may see some problems with DNS resolution inside kapsule clusters.

Identified: We are deploying a fix to stabilize the broadcast traffic.

Identified: We are still occuring high level of broadcast traffic causing some perturbations. We have a fix being tested at the moment and we will start the deployment in the next hour

Identified: In order to stabilize broadcast traffic, we are adding more computing ressources to better handle the load, some maintenances will occur today

Monitoring: The maintenance announced in the previous message was not necessary and has been canceled. We are monitoring the situation.

Investigating: From 18:00 CET we are restarting a few VPC services on fr-par-1 to fix some edges cases

Monitoring: Maintenance completed successfully.

Investigating: From 16:30 CET we will proceed with a restart of some internal VPC services to fix some edge cases, a few perturbation can occur during the restart but should not last.

Investigating: We are continuing to investigate this issue.

Investigating: A few customers are still impacted by a few perturbations, the VPC team is working to fix these isolated cases

Monitoring: We have noticed a degradation in the performance of the product link to VPC, at 13:58 CET, A fix has been made.

We are currently monitoring the situation.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing a disruption in the ingestion of logs for custom data sources in the NL-AMS region. This issue affects the availability of log data for affected services.

Resolved: This incident has been resolved.

Investigating: We experienced 503 errors and connection interruptions with Serverless Containers/Functions in the PL-WAW region from 10:17:40 UTC to 10:18:22 UTC. This was caused by an internal maintenance issue on our side. The issue has been identified and resolved.

Investigating: We are currently experiencing issues with launching tasks in the Instance Orchestration service in the pl-waw-1 region. This may result in actions not being launched and 500 errors on the API.

Resolved: This incident has been resolved.

Investigating: We experienced 503 errors and connection interruptions with Serverless Containers/Functions in the PL-WAW region from 10:17:40 UTC to 10:18:22 UTC. This was caused by an internal maintenance issue on our side. The issue has been identified and resolved.

Resolved: This incident has been resolved.

Investigating: We experienced 503 errors and connection interruptions with Serverless Containers/Functions in the PL-WAW region from 10:17:40 UTC to 10:18:22 UTC. This was caused by an internal maintenance issue on our side. The issue has been identified and resolved.

Resolved: Since 2025 Dec 2 09:18 UTC the switch is up. Redundancy is temporarily lost, but the impact on the client side should be resolved.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are investigating the issue

Investigating: We are continuing to investigate this issue.

Investigating: A switch in nl-ams-1 is down, causing a loss of public connectivity for part of a rack. Dedibox dedicated servers in this rack may experience connectivity issues.

Resolved: The network switch has been changed and the network is operational again.

Investigating: Network connectivity issues are currently affecting Elastic Metal and Dedibox services in the fr-par-2 region, in room 7 rack A69. We are currently investigating this issue.

Resolved: This incident has been resolved.

Investigating: We have detected a switch down in DC5 Room 1 Rack D14. Servers in that rack currently have no private network access and are unreachable. 04/12/2025 02H40 UTC Issue has been forwarded to our team for resolution.

Resolved: This incident has been resolved.

Identified: A technician is on site checking network connectivity.

Investigating: Some servers have lost their public connectivity.

Identified: Our service is currently experiencing disruption due to blacklisting by Microsoft. We are actively working with Microsoft to resolve this issue as soon as possible.

Investigating: Following the implementation of new email security measures, it is no longer possible to send messages via SMTP using a sender address that is different from the one used for authentication.

This change is intended to enhance security and improve the deliverability of your emails by preventing identity theft.

If you encounter a 550 ‘Authenticated sender mismatch’ error, please check that the email address configured as the sender in your email client matches the one used to connect to the SMTP server.

Resolved: This incident has been resolved.

Investigating: We have detected a switch down in DC5 Room 1 Rack D14. Servers in that rack currently have no private network access and are unreachable. 04/12/2025 02H40 UTC Issue has been forwarded to our team for resolution.

Resolved: This incident has been resolved.

Investigating: A network switch in one of our data centers is currently experiencing connectivity issues, affecting public and private network connectivity for a portion of the rack. We are investigating the cause and will provide updates as soon as possible. Datacenter: DC2, Room: 101 101, Rack: G28, Block: G

Resolved: A network switch in our fr-par-1 region experienced an unexpected reboot, causing a temporary loss of public connectivity for Dedibox dedicated servers in the affected rack. The issue has been resolved, and we are monitoring the situation to prevent any recurrence.

Resolved: The network switch has been changed and the network is operational again.

Investigating: Network connectivity issues are currently affecting Elastic Metal and Dedibox services in the fr-par-2 region, in room 7 rack A69. We are currently investigating this issue.

Resolved: This incident has been resolved.

Investigating: We have detected a switch down in DC5 Room 1 Rack D14. Servers in that rack currently have no private network access and are unreachable. 04/12/2025 02H40 UTC Issue has been forwarded to our team for resolution.

Investigating: The RPN switch in rack S1/D14 experienced a crash and reboot, causing a 5-minute unavailability of the RPN service in this rack. The service should be unavailable during the replacement.

Resolved: This incident has been resolved.

Identified: A technician is on site checking network connectivity.

Investigating: Some servers have lost their public connectivity.

Resolved: This incident has been resolved.

Investigating: The switch h6-6b46.rpn.ams1 crashed and rebooted, causing RPN in this rack to be unavailable for 5 minutes.