Adeptcore Status

Identified: We have received confirmation that data collection for our environment has now been completed. The engineering teams are currently reviewing the collected data and analyzing the packet captures in detail. At the same time, they are continuing to gather information from one additional affected client to help build a more complete picture of the issue.

At this stage, one of the working theories is that this may be related to an internal system automation that could be contributing to the problem. This remains one of two primary areas of focus as the investigation continues.

In parallel, a support ticket has been opened with the vendor responsible for routing and switching, and the matter is being actively escalated to ensure additional expertise is brought in.

Identified: We have received an update from the data center engineering team indicating that they are close to completing the collection of troubleshooting data for one affected client, after which our environment will be next in line for analysis. They will be gathering detailed data from our systems to support further investigation into the issue.

At this time, there is no estimated timeline for resolution. The teams are focused on collecting and analyzing the necessary information to better understand the exact nature of the incident and to evaluate the most effective path forward.

Identified: We have just spoken with the engineer currently assigned to our ticket at the data center. The issue is being actively worked on in close coordination with the data center cloud team.

At this stage, the data center team is performing packet captures and investigating the behavior across multiple locations. The next step in their process is to closely review and troubleshoot the packets being routed out of our bare metal environments.

Based on what is known so far, there is a working theory that the packets may contain malformed headers, which could be preventing them from routing correctly across the virtualized LAN layer. This appears to be a complex issue, and the teams involved are carefully analyzing the situation to better understand the root cause.

In parallel, the data center team is also opening a support ticket with the hardware vendor on their end to bring in additional expertise and help accelerate the resolution.

Investigating: Our engineering team is still currently investigating a significant issue affecting three of our bare metal environments located in Chicago, Ashburn, and Phoenix. At this time, all three locations are experiencing downtime. Initial findings indicate this may be part of a broader issue affecting multiple accounts, likely related to how intermediary virtual traffic tagging is handled across data centers.

We recently completed a call with our Tier 2 engineering team and they are now working closely with the data center cloud team to conduct a deeper analysis and move toward implementing a resolution as quickly as possible.

At this time, our colocation and other services remain operational and unaffected. However, we are observing an impact to network connectivity between our colocation environment and the affected bare metal datacenters, which is contributing to the overall disruption.

Investigating: The tier 2 datacenter networking team is still working on identifying and resolving the root cause affecting the three bare metal servers. At the time of writing, we do not yet have an exact root cause. Once the exact cause has been identified, this incident will be updated with the relevant details and the steps taken to resolve the issue.

Investigating: Our investigation into this issue is still ongoing. Our support ticket regarding this issue has been escalated to Tier 2 of the datacenter networking team and we are actively working with them in order to resolve the ongoing connectivity issue affecting three bare metal servers. The datacenter team is currently attempting to identify the exact cause of this outage. Once we have confirmation on the exact cause of the outage and the next steps, this incident will be updated with those details.

Investigating: We are continuing to investigate this issue alongside the datacenter engineers. We have confirmed that three separate bare-metal servers are currently impacted by the connectivity issue and that the issue does not appear to be caused by any hardware located on the servers themselves.

Additional updates will follow as soon as they become available.

Investigating: We are receiving reports of some clients on Baremetal servers experiencing connectivity issues. We are working with the datacenter engineers at this time to isolate and resolve this as quickly as possible.