Grafana Cloud Status

Resolved: This incident has been resolved. Thank you for your patience.

Identified: We are continuing to work on rolling back a PR responsible for this behavior. Once we have more information, we will share it here.

Thank you for your patience.

Identified: Queries with drop error, including log volume histogram queries (the queries that generate the histogram visualization in Grafana), are failing due to a bug with series limit checks.

The Root cause has been identified, and we are working on a fix.

Resolved: This incident has been resolved by restarting the affected services.

Investigating: The AWS Logs integration in the same region is affected as well. We will provide further updates as our investigation progresses.

Investigating: We are currently investigating a major outage in Loki writes and Frontend Observability in the prod-us-central-0 region. Our Engineering team is investigating this and we will provide further updates as our investigation progresses.

Resolved: We’ve implemented a fix and can confirm the issue is fully resolved as of 20:25 UTC.

Thank you for your patience.

Investigating: We are investigating an issue where some customers may see higher Loki query byte usage reported than was actually consumed. This affects usage reporting only; there is no impact to query execution or service availability.

The issue began at approximately 13:20 UTC and is ongoing. We expect the issue to be resolved soon and will provide another update as more information becomes available.

Resolved: Since the mitigation has been applied, we have not seen the errors return. At this point, we are considering the incident resolved.

Identified: We've identified a possible cause, and a mitigation is in place to prevent further occurrences.

Investigating: The errors and latency have now recovered, we continue investigating the root cause.

Investigating: The errors are recovering, and we are still looking into the root cause of this.

Investigating: We are currently investigating an issue with Mimir in prod-eu-west-0 we are seeing read errors and high latency. This incident is currently ongoing. The errors are recovering but we are currently looking into the route cause of this.

Resolved: This incident has been resolved.

Investigating: We are investigating an issue affecting Confluent metrics ingestion across all regions. Due to an elevated error rate on the Confluent side, some metrics may not be ingested, resulting in potential data loss. We are actively investigating the issue and will provide updates as more information becomes available.

Resolved: This incident has been resolved.

Monitoring: A fix has been applied and we are currently monitoring results.

Investigating: We are currently investigating Rule Evaluation errors on the cluster prod-gb-south-0 which is leading to error codes showing within the stacks. We are looking into the issue and will update accordingly.

Resolved: This incident has been resolved.

Investigating: We've improved the metric ingestion delay time and are working on additional fixes to bring it down to expected range. Customers can currently expect a delay of 2 to 5 minutes before their test run metrics show up (after starting a test).

Investigating: Update: Changed incident title to "Test run metrics processing is delayed"

We have found the issue and are working on deploying the fix.

Investigating: We are experiencing intermittent delays with secondary metrics processing for k6 Cloud test runs due to heavy load. We don't expect any data loss or impact on user runs, but results may take longer time to appear in UI.

Investigating: A small update: The issue is isolated to new test runs, and users can go see the metrics of all the previous test runs

Investigating: We’re currently investigating an issue causing metrics not to appear during test runs. . Our team is actively working to identify the cause. Thank you for your patience.