Hosted Mender Status

Postmortem: # Brief service disruption on Hosted Mender (EU)

On June 24, between 12:04 and 12:09 UTC, some customers on our EU region experienced a short interruption. During this five minute window, devices may have had trouble connecting, and parts of the web interface and API may have returned errors.

What happened

While cleaning up a configuration issue, we removed an internal duplicated definition (the same set of kubernetes manifests rendered twice from Kustomize). Our automation interpreted this as a request to remove and reinitialize two background components that handle messaging and caching (NATS and Redis). They were rebuilt automatically, but for about five minutes they were unavailable, which caused the disruption.

What we are doing about it

The automation completed the reconciliation loop and redeployed the background components automatically. We verified that normal operation was restored and confirmed all services are healthy. We are changing how we make this type of change so that components are never removed before their replacement is ready.

We apologize for the disruption.

Resolved: This incident has been resolved.

Monitoring: The affected services have been restarted and we're monitoring the results.

Identified: The issue has been identified: there was an issue with the Redis and NATS deployments. A fix has been implemented and we're restarting the impaired services.

Investigating: We are currently investigating this issue.