Teraswitch Status
Real-time updates of Teraswitch issues and outages
Teraswitch status is Operational
Intra-Market Connectivity
Internet Exchanges and Peering
Teraswitch MIA1 - Miami, FL
Teraswitch LON1 - London, United Kingdom
Teraswitch AMS1 - Amsterdam, Netherlands
Teraswitch AMS2 - Amsterdam, Netherlands
Teraswitch AMS3 - Amsterdam, Netherlands
You're checking on Teraswitch - but is your own site converting?
Outages aren't the only thing that costs you visitors.
Get a visual audit that shows exactly where your site loses conversions.
Free, takes 2 minutes.
Active Incidents
Teraswitch is looking into general Internet Instability.
Recently Resolved Incidents
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch Outage Survival Guide
Teraswitch Components
Teraswitch Public Services
API
Website (teraswitch.com)
Portal (console.tsw.io)
Teraswitch Global Network Services
Intra-Market Connectivity
Teraswitch is looking into general Internet Instability.
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Global External Internet
Internet Exchanges and Peering
Teraswitch is looking into general Internet Instability.
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Ultra-Low Latency Connectivity
Teraswitch PIT1 - Pittsburgh, PA
Teraswitch IAD1 - Ashburn, VA
Teraswitch EWR1 - Newark, NJ
Teraswitch EWR2 - Newark, NJ
Teraswitch MIA1 - Miami, FL
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch CHI1 - Chicago, IL
Teraswitch DAL1 - Dallas, TX
Teraswitch SLC1 - Salt Lake City, UT
Teraswitch LAX1 - Los Angeles, CA
Teraswitch SEA1 - Seattle, WA
Teraswitch VAN1 - Vancouver, Canada
Teraswitch LON1 - London, United Kingdom
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch AMS1 - Amsterdam, Netherlands
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch AMS2 - Amsterdam, Netherlands
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch AMS3 - Amsterdam, Netherlands
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch DUB1 - Dublin, Ireland
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch FRA1 - Frankfurt, Germany
Teraswitch DUB2 - Dublin, Ireland
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch FRA2 - Frankfurt, Germany
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch SGP1 - Singapore
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch SGP2 - Singapore
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch TYO1 - Tokyo, Japan
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch SEL1 - Seoul, South Korea
Teraswitch TYO2 - Tokyo, Japan
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.
Teraswitch TYO3 - Tokyo, Japan
(This status is split off from a previous issue that was originally believed to have been related.)
Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected.
MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated.
Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one.
A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed.
Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC.
Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated.
A full RFO will follow.