Technolutions Status

Resolved: We have completed our review of the incident that affected the processing of certain exports and imports on 12/5.

Slate uses a cluster of worker nodes to process background jobs. These nodes are periodically refreshed with the latest version of Slate, and on 12/5 they were updated to a build that included a newer version of a third-party AWS SDK. Worker nodes employ a shadow-copying mechanism that creates local cached copies of binary dependencies to avoid locking files on shared storage. This mechanism refreshes cached assemblies only when it detects a change in the assembly’s identity, which typically occurs during breaking or versioned updates.

The updated AWS SDK introduced an internal structural change: a property was moved from a derived class to its base class. Although no public API signature changed, the compiled IL binding did. Because the assembly’s identity did not change, the shadow-copy cache was not refreshed. As a result, the worker nodes attempted to execute IL referencing the property on the derived class against the older cached version, causing a runtime “method not found” error.

On the afternoon of 12/5, we reverted the worker nodes to the earlier AWS SDK version, restoring normal operation.

We have since implemented enhancements to our worker-node architecture to provide stronger cache isolation and to ensure that updates to the AWS SDK and other dependencies are correctly recognized, even when internal changes occur without a corresponding change in assembly identity.

Web nodes use a different shadow-copying mechanism and were not affected.

Identified: We are currently investigating an intermittent issue that has affected the processing of certain exports and imports. We believe that this is due to an issue with an updated AWS SDK that interacts with a background storage layer. We have temporarily reverted to the previous version of this third-party tool. Imports that previously failed are being reprocessed now.