Wix Status

Postmortem: On September 15th at 8:27 AM UTC, some users in Europe experienced errors when accessing Wix.com, logging in, and viewing live sites, as well as issues using site features such as Bookings, Stores and Payments. All functionality was fully restored within approximately 30 minutes.

The root cause was an infrastructure change intended to remove unused resources in one of our European data center regions. Those resources were still serving live traffic.

We regularly decommission and clean up infrastructure that is no longer needed. In this case, the change was made in a region that had already been largely retired, and the resources targeted for removal were believed to be unused. That assumption was not fully verified before the change was applied, and configuration that was still routing European traffic was removed. Requests arriving in that region could not reach the services behind them, which led to errors across a number of Wix products.

Our monitoring detected the errors within minutes, our on-call team was engaged immediately, but, because the change had been made in a region that was being retired, it was not an obvious candidate for causing live traffic impact, and identifying it as the cause took longer than it should have. Once identified, we mitigated by shifting European traffic away from the affected region and serving it from other active regions, and restored the removed configuration in parallel. Once service was fully restored, traffic was returned to the affected region without further disruption.

Following this incident, we are introducing automated validation that verifies whether a network resource is still in use before it can be removed, blocking the change from being merged if it is. We are also strengthening review requirements for infrastructure changes, and improving high availability and automatic failover between regions, including a faster way to move traffic out of a region when needed, so that similar issues can be contained before it affects users.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are aware that users are experiencing multiple issues with Wix Services and we are currently investigating it. Further updates to follow shortly.

Postmortem: On September 15th at 8:27 AM UTC, some users in Europe experienced errors when accessing Wix.com, logging in, and viewing live sites, as well as issues using site features such as Bookings, Stores and Payments. All functionality was fully restored within approximately 30 minutes.

The root cause was an infrastructure change intended to remove unused resources in one of our European data center regions. Those resources were still serving live traffic.

We regularly decommission and clean up infrastructure that is no longer needed. In this case, the change was made in a region that had already been largely retired, and the resources targeted for removal were believed to be unused. That assumption was not fully verified before the change was applied, and configuration that was still routing European traffic was removed. Requests arriving in that region could not reach the services behind them, which led to errors across a number of Wix products.

Our monitoring detected the errors within minutes, our on-call team was engaged immediately, but, because the change had been made in a region that was being retired, it was not an obvious candidate for causing live traffic impact, and identifying it as the cause took longer than it should have. Once identified, we mitigated by shifting European traffic away from the affected region and serving it from other active regions, and restored the removed configuration in parallel. Once service was fully restored, traffic was returned to the affected region without further disruption.

Following this incident, we are introducing automated validation that verifies whether a network resource is still in use before it can be removed, blocking the change from being merged if it is. We are also strengthening review requirements for infrastructure changes, and improving high availability and automatic failover between regions, including a faster way to move traffic out of a region when needed, so that similar issues can be contained before it affects users.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are aware that users are experiencing multiple issues with Wix Services and we are currently investigating it. Further updates to follow shortly.

Postmortem: On September 15th at 8:27 AM UTC, some users in Europe experienced errors when accessing Wix.com, logging in, and viewing live sites, as well as issues using site features such as Bookings, Stores and Payments. All functionality was fully restored within approximately 30 minutes.

The root cause was an infrastructure change intended to remove unused resources in one of our European data center regions. Those resources were still serving live traffic.

We regularly decommission and clean up infrastructure that is no longer needed. In this case, the change was made in a region that had already been largely retired, and the resources targeted for removal were believed to be unused. That assumption was not fully verified before the change was applied, and configuration that was still routing European traffic was removed. Requests arriving in that region could not reach the services behind them, which led to errors across a number of Wix products.

Our monitoring detected the errors within minutes, our on-call team was engaged immediately, but, because the change had been made in a region that was being retired, it was not an obvious candidate for causing live traffic impact, and identifying it as the cause took longer than it should have. Once identified, we mitigated by shifting European traffic away from the affected region and serving it from other active regions, and restored the removed configuration in parallel. Once service was fully restored, traffic was returned to the affected region without further disruption.

Following this incident, we are introducing automated validation that verifies whether a network resource is still in use before it can be removed, blocking the change from being merged if it is. We are also strengthening review requirements for infrastructure changes, and improving high availability and automatic failover between regions, including a faster way to move traffic out of a region when needed, so that similar issues can be contained before it affects users.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are aware that users are experiencing multiple issues with Wix Services and we are currently investigating it. Further updates to follow shortly.