Honeycomb Status

Resolved: All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.

Monitoring: Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

Monitoring: We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

Investigating: We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

Resolved: All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.

Monitoring: Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

Monitoring: We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

Investigating: We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

Resolved: All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.

Monitoring: Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

Monitoring: We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

Investigating: We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

Resolved: This incident has been resolved.

Monitoring: We are monitoring the situation. Events may have been delayed between 17:53 and 18:04 UTC.

Investigating: We are investigating an issue with delayed ingest in the EU region. SLOs and Trigger evaluations may also be delayed.

Resolved: All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.

Monitoring: Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

Monitoring: We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

Investigating: We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

Resolved: This incident has been resolved.

Monitoring: We are monitoring the situation. Events may have been delayed between 17:53 and 18:04 UTC.

Investigating: We are investigating an issue with delayed ingest in the EU region. SLOs and Trigger evaluations may also be delayed.