Scaleway Status

Identified: We did discover that the trouble is broader than we thought. As a first mitigation we do recommend that users wait before upgrading from aws v1 (we do recommand any version strictly prior to aws-cli 1.37.0 or boto3 < 1.36.0). If aws v2 (that we do not recommend for the moment) is mandatory, we do recommend using aws-cli < 2.23.0 We are working actively to support those new versions, and plan to release fixes for the end of next week. The details of the bug are : support of CRC64NVME does not work properly Tranfer-Encoding: chunked header does not work

Identified: Everyone using aws-cli versions versions v1: >=1.37.0 and v2: >=2.23.0 received a 400 errors on PUT and POST requests.

Investigating: aws-cli now enforces a CRC64NVME integrity checksum on all PUT and POST requests. This concerns the versions v1: >=1.37.0 and v2: >=2.23.0. We do not currently support this checksum, we are working on how best to handle it on our end. In the meantime, you can choose one of these options to keep your aws-cli request working:

• Use the --checksum-algorithm option with one of our supported checksums:
  • SHA1
  • SHA256
  • CRC32
  • CRC32C
• Use an older version of aws-cli, this way it will not enforce the CRC64NVME checksum

Identified: - Current progress on this incident:

Most of the disk attach are fixed, we still have some timeouts remaining Detach will no longer fail on those half-attached volumes Disks which were not attached properly during the incident wont be fixed automatically, you can detach/attach them Root cause not known

• Next actions on Instance side:

We will deploy a second release this morning with an even higher timeout value We'll wait 1-2 hours to ensure everything is OK before passing the status to monitoring as we don't know the root cause

Investigating: Since Nov 6th at 10 PM (FR), some databases have failed to start, and users are unable to connect to their databases. The issue is linked to the incident detailed at https://status.scaleway.com/incidents/n7k8t96f4cjy.

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: Starting at 18h00 UTC block storage API response times increased until it became really significant at 19h02~ UTC.

This impacted the API with slow response times. Some calls have failed (Internal error) due to the database load. No impact on Data plane (block volumes) performances.

Our team found a SQL optimization issue linked to the call ListVolumes and fixed it at 21h44 UTC. Since then, response times came back to normal and error rate dropped as well.

Investigating: We are spotting an increase of API latency on the Block storage API with a degradation of the P99 latencies on some API methods.

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Investigating: We are continuing to investigate this issue.

Investigating: Starting on 2024/11/14 17:47 UTC, clients might experience issues pushing/pulling images from rg.fr-par.scw.cloud due to a certificate issue.

As a result, users from Serverless Functions/Containers using an image hosted on rg.fr-par.scw.cloud might also get impacted, as their functions/containers won't be able to start. Impacts for Serverless Functions/Containers are:

• instances that cannot start (scale up might be impossible)
• requests timeouting or 5xx errors if no instances are available.

We are currently investigating this issue, and apologize for the inconvenience.

Resolved: This incident has been resolved.

Monitoring: A fix has been deployed. We will continue to monitor the situation.

Investigating: Since 8:00 PM, we have noticed an increase in API call errors affecting some customers. Our team is currently investigating the root cause.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Identified: - Current progress on this incident:

Most of the disk attach are fixed, we still have some timeouts remaining Detach will no longer fail on those half-attached volumes Disks which were not attached properly during the incident wont be fixed automatically, you can detach/attach them Root cause not known

• Next actions on Instance side:

We will deploy a second release this morning with an even higher timeout value We'll wait 1-2 hours to ensure everything is OK before passing the status to monitoring as we don't know the root cause

Investigating: Since Nov 6th at 10 PM (FR), some databases have failed to start, and users are unable to connect to their databases. The issue is linked to the incident detailed at https://status.scaleway.com/incidents/n7k8t96f4cjy.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing an issue affecting the instances infrastructure across all availability zones in FR-PAR.

Impact:

• Starting and stopping instances with local storage may experience significant delays
• Snapshot tasks are currently stuck and unable to progress

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing an issue affecting the instances infrastructure across all availability zones in FR-PAR.

Impact:

• Starting and stopping instances with local storage may experience significant delays
• Snapshot tasks are currently stuck and unable to progress

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing an issue affecting the instances infrastructure across all availability zones in FR-PAR.

Impact:

• Starting and stopping instances with local storage may experience significant delays
• Snapshot tasks are currently stuck and unable to progress

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Identified: A fix has been implemented and is being validated.

Identified: We have temporarily disabled the Plan and Apply API endpoints for SBS migrations due to issues encountered for a small number of migrations.

We are actively working on a resolution and expect to restore functionality within the next few days.

Resolved: This incident has been resolved.

Monitoring: Unable to use the Account APIs and also to browse the console Started : 17h16 CEST Ending: 17h27 CEST

Resolved: This incident has been resolved.

Monitoring: A faulty device was replaced. Service is UP since 13:35 CEST. Monitoring in progress.

Investigating: PUB switch unreachable since 2025/04/14 09:31 CEST. Our network team is already working to solve this issue.

Resolved: The problem was on the switch side which affected several servers and a downtime that lasted from 22h35 until 1am Paris Time UTC+2 The switch has been restarted and fixed. It is now working correctly.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Investigating: We are currently experiencing technical difficulties with our email service.

⚠️ Impact: Connection to the email on webmail show an error on storage server

We will provide updates on our progress as new information becomes available.

Resolved: This incident has been resolved.

Monitoring: A faulty device was replaced. Service is UP since 13:35 CEST. Monitoring in progress.

Investigating: PUB switch unreachable since 2025/04/14 09:31 CEST. Our network team is already working to solve this issue.