Buildkite Status

Postmortem: ## Service Impact

From 00:49 to 01:26 on Nov 15, 2024 (UTC), an estimated 1% of artifacts failed to upload due to signature verification errors.

Incident Summary

Up to 1% of build artifact uploads, principally those uploaded by Agent version v3.83.0 or later, encountered signature mismatch errors. This Agent version introduced multipart uploads. A backward-incompatible server library upgrade instigated the incident. Seven minutes after the incident was detected, we rolled back the change. The root cause was a recent upgrade of Ruby library dependencies responsible for URL presigning. One library added an additional header for all REST API calls, a change not accounted for in the older version of a related library which we did not upgrade at the same time. This omission led to the URL signature mismatch errors on upload.

Changes we're making

In future, we’ve ensured that this group of libraries is upgraded as a whole to prevent dependency mismatches that could introduce unintentional breaking changes. Additionally, we will enhance our test coverage around presigned URLs to ensure that their signatures match the expectations of our upload service, and improve monitoring of the upload completion rate to reduce detection time.

Resolved: This incident has been resolved.

Monitoring: Up to 10% of artifacts failed to upload due to a signature verification error. We have rolled back to a known good version while we investigate further.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating issues uploading artifacts from the Agent.