CloudAMQP Status

Monitoring: We are still experiencing issues with this region. If your services are impacted, we recommend you create a new cluster in a different region. When creating, use the copy config option, but do not copy plugin settings as this requires the source cluster to be running. Instead, enable once cluster is configured.

Monitoring: AWS has an ongoing issue with multiple services in region me-south-1. Shared server hummingbird-01 affected.

Monitoring: We are still experiencing issues with all AZ's in this region. If your services are impacted, we recommend you create a new cluster in a different region. When creating, use the copy config option, but do not copy plugin settings as this requires the source cluster to be running. Instead, enable once cluster is configured.

Monitoring: We are continuing to monitor. The outage is in regards to the AZ mec1-az2

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: AWS has an ongoing issue with multiple services in region me-central-1.

Monitoring: We are still experiencing issues with this region. If your services are impacted, we recommend you create a new cluster in a different region. When creating, use the copy config option, but do not copy plugin settings as this requires the source cluster to be running. Instead, enable once cluster is configured.

Monitoring: AWS has an ongoing issue with multiple services in region me-south-1. Shared server hummingbird-01 affected.

Postmortem: Broker metrics and metrics integrations degradation

Window: 2026-04-23 03:25 – 05:35 UTC (2h 10m)

Impact

Broker metrics (connections, channels, queues, consumers, message rates, node and netsplit state) were degraded, progressively dropping until recovery at 05:35 UTC. Downstream effects:

• Queue, Consumer, Connection, Channel, Connection flow, and Netsplit alarms: threshold breaches during the window may not have triggered, or triggered late. Alarms already tripped before 03:25 UTC kept
  their state.
• Metrics Integrations (Datadog, CloudWatch, New Relic, Splunk, Dynatrace, etc.): delivery of broker metrics was reduced for the duration of the window. Alarms configured on the receiving side may
  likewise have missed or lagged.

Not affected: server metrics (CPU, memory, disk) and their corresponding alarms, Notice alarms, broker availability, and the metric graphs shown in the CloudAMQP console (these use a separate data path
and continued to update normally).

Timeline (UTC)

• 03:25 — broker-metrics collection begins degrading
• 03:35 — ~30% of broker-metric samples missing
• 03:50 — ~50% plateau
• 05:25 — collector workers cycled and re-initialised
• 05:35 — full throughput restored

Root cause

Broker metrics are polled from each cluster's management HTTP API by a pool of collector workers and published to an internal message bus that both the Alarms service and Metrics Integrations consume
from. At 03:25 UTC this service became unresponsive without raising an error. Affected workers silently stopped polling while remaining alive to the platform, so automatic restart did not trigger. On-call
staff began investigating around 05:00 UTC; force-restarting the service restored metric flow.

The exact trigger has not been identified. Our focus is on ensuring the condition is detected and handled promptly if it recurs.

What we are changing

• Added metrics and alarms specific to this service, with high-urgency paging for on-call.
• Reliability improvements in the service itself to detect and restart stalled work.

What you can do

CloudAMQP offers a new generation of metrics integrations based on Prometheus, with a re-engineered pipeline that does not rely on these centralised services — each server forwards data directly to your
endpoint. These have been running in production for some time and have proved very reliable.

• https://www.cloudamqp.com/blog/prometheus-metrics-integrations.html
• https://www.cloudamqp.com/docs/monitoring_metrics_datadog_v3.html
• Background: https://www.cloudamqp.com/blog/decentralized-observability-with-open-telemetry-part-1.html

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: There appears to be an issue regarding sending metrics to all Legacy metrics integrations. Alarms based on broker metrics also got affected.

We are currently investigating the issue

Prometheus based metrics (Datadog v3 etc) not affected.

Resolved: This incident has been resolved.

Investigating: Servers in Azure West US 3 can experience connection issues due to underlying issues at provider. We monitor and will update when we know more.

Resolved: This incident has been resolved.

Investigating: We are getting indications of instances being down in Azure region westus. We are investigating the issue.