Grafana Cloud Status

Resolved: This incident has been resolved.

Identified: TraceQL queries with "= nil" in Explore Traces and part of Drilldown Traces are failing with 400 Bad Request errors. The issue has been identified, and a fix is currently being rolled out.

Resolved: We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Monitoring: Synthetic Monitoring has now also recovered. Customers should no longer experience alert rules failing to evaluate.

We continue to monitor for recurrence and will provide updates accordingly.

Monitoring: Engineering has released a fix and as of 22:25 UTC, customers should no longer experience ingestion issues. We will continue to monitor for recurrence and provide updates accordingly.

Investigating: While investigating this issue, we also became aware that Synthetic Monitoring is affected. Some customers may have alert rules failing to evaluate.

Investigating: As of 21:30 UTC, we are experiencing a partial ingestion outage in Grafana Mimir. This is affecting the write path, where some ingestion requests are failing or timing out.

Our engineering team is actively investigating and working to identify the root cause.

Resolved: We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Monitoring: We are observing a trend in improvement after implementing a fix. We will continue to monitor and update accordingly.

During our investigation, we also became aware that some alerts associated with Synthetic Monitoring checks have been failing to evaluate correctly.

Identified: Our engineering team has identified a potential root cause, and a fix is being implemented.

Investigating: Our engineering team has engaged with our Cloud Service Provider and are working together to continue to investigate this issue.

Investigating: We are continuing to investigate this issue.

Investigating: We have also identified that trace ingestion may also be affected. Some customers may experience elevated latency and intermittent errors when sending traces. Investigation is ongoing.

Investigating: We have detected an issue causing some customers to experience failed metric pushes as well as increased latency when sending metrics. The issue was first observed at 18:30 UTC.

Our engineering team is actively investigating the root cause and working to restore normal operation as quickly as possible. We will provide further updates as more information becomes available.

Thank you for your patience while we work to resolve this issue.

Resolved: The incident has been resolved.

Monitoring: Read path has been restored at 08:23 UTC and queries are fully functioning again. The read path outage lasted from 08:04 to 08:23 UTC.

Investigating: At 08:04 UTC we detected read path outage (queries) on cortex-prod-13. We are currently investigating this issue.

The ingestion path (writes) is not affected.

Resolved: The issue has been resolved.

Monitoring: The query service is operational again, logs reads should be available on the cluster. Our engineers are monitoring the health status of the service to ensure full recovery.

Identified: Since today 9th, around 12:30 UTC time, we are experiencing problems on the Loki read path of cluster eu-west-2. This translates in difficulties to query logs for customers on this cluster, and can also impact alerts and other services based on these logs. Our engineers are actively working in restoring the service.