
Grafana Cloud Status
Real-time updates of Grafana Cloud issues and outages
Grafana Cloud status is Operational
AWS Australia - prod-ap-southeast-2
AWS Brazil - prod-sa-east-1
AWS Canada - prod-ca-east-0
AWS Germany - prod-eu-west-2
AWS Germany - prod-eu-west-4
AWS India - prod-ap-south-1
AWS Japan - prod-ap-northeast-0
Active Incidents
No active incidents
Recently Resolved Incidents
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
During the secrets migration in https://status.grafana.com/incidents/47d1q4sphrmj, secrets proxy URLs for some customers updated in the following regions: prod-us-central-0, prod-us-east-0, and prod-eu-west-2. This was an unexpected breaking change affecting a subset of customers.
This will specifically affect customers who are using secrets on private probes behind a firewall.
We are investigating. If your private probes are impacted, we ask you to update firewall rules for the secrets proxy to allow outbound connections to the updated hosts:
gsm-proxy-prod-eu-west-2.grafana.net -> gsm-proxy-prod-eu-west-4.grafana.net gsm-proxy-prod-us-central-0.grafana.net -> gsm-proxy-prod-us-central-4.grafana.net gsm-proxy-prod-us-east-0.grafana.net -> gsm-proxy-prod-us-east-2.grafana.net
Note that this URL change affects only a small subset of customers, the majority of customers will not need to update firewall rules. For affected customers, private probes will show the following error in probe logs, for example: Error during test execution: failed to get secret: Get "https://gsm-proxy-prod-us-east-2.grafana.net/api/v1/secrets/.../decrypt": Forbidden undefined
We're currently investigating an issue with elevated write latency in Hosted Traces prod-us-central-0 region. It's experiencing sustained high write latency since 7:20 AM UTC. Only a small subset of the requests are impacted.
We are currently investigating an issue causing degraded write performance across multiple products in the AWS us-east-2 region. Our engineering team is actively working to determine the full scope and impact of the issue and restore normal service levels.
Grafana Cloud Outage Survival Guide
Grafana Cloud Components
Grafana Cloud: Hosted Grafana
AWS Australia - prod-ap-southeast-2
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Brazil - prod-sa-east-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Canada - prod-ca-east-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Germany - prod-eu-west-2
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Germany - prod-eu-west-4
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS India - prod-ap-south-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Japan - prod-ap-northeast-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS UAE - prod-me-central-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Singapore - prod-ap-southeast-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Sweden - prod-eu-north-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS US East - prod-us-east-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS US East - prod-us-east-2
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS US West - prod-us-west-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS Australia - prod-au-southeast-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
AWS UK - prod-gb-south-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
Azure Netherlands - prod-eu-west-3
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
Azure US Central - us-central2
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP Australia - prod-au-southeast-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP Belgium - prod-eu-west-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP Brazil - prod-sa-east-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP India - prod-ap-south-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP Singapore - prod-ap-southeast-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP UK - prod-gb-south-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP US Central - prod-us-central-0
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP US Central - prod-us-central-3
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP US Central - prod-us-central-4
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
GCP US East - prod-us-east-1
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
play.grafana.org
We are currently investigating an issues impacting Email delivery for some Services, including Alert Notifications.
Grafana Cloud: Graphite
AWS Australia - prod-ap-southeast-2: Querying
AWS Australia - prod-ap-southeast-2: Ingestion
AWS Brazil - prod-sa-east-1: Querying
AWS Brazil - prod-sa-east-1: Ingestion
AWS Canada - prod-ca-east-0: Querying
AWS Canada - prod-ca-east-0: Ingestion
AWS Germany - prod-eu-west-2: Querying
AWS Germany - prod-eu-west-2: Ingestion
AWS UAE - prod-me-central-1: Querying
AWS UAE - prod-me-central-1: Ingestion
AWS India - prod-ap-south-1: Querying
AWS India - prod-ap-south-1: Ingestion
AWS Japan - prod-ap-northeast-0: Querying
AWS Japan - prod-ap-northeast-0: Ingestion
AWS Singapore - prod-ap-southeast-1: Querying
AWS Singapore - prod-ap-southeast-1: Ingestion
AWS Sweden - prod-eu-north-0: Querying
AWS Sweden - prod-eu-north-0: Ingestion
AWS US East - prod-us-east-0: Querying
AWS US East - prod-us-east-0: Ingestion
AWS US West - prod-us-west-0: Querying
AWS US West - prod-us-west-0: Ingestion
Azure Netherlands - prod-eu-west-3: Querying
Azure Netherlands - prod-eu-west-3: Ingestion
Azure US Central - us-central2: Querying
Azure US Central - us-central2: Ingestion
GCP Australia - prod-au-southeast-0: Querying
GCP Australia - prod-au-southeast-0: Ingestion
GCP Belgium - prod-eu-west-0: Querying
GCP Belgium - prod-eu-west-0: Ingestion
GCP Brazil - prod-sa-east-0: Querying
GCP Brazil - prod-sa-east-0: Ingestion
GCP India - prod-ap-south-0: Querying
GCP India - prod-ap-south-0: Ingestion
GCP Singapore - prod-ap-southeast-0: Querying
GCP Singapore - prod-ap-southeast-0: Ingestion
GCP UK - prod-gb-south-0: Querying
GCP UK - prod-gb-south-0: Ingestion
GCP US Central - prod-us-central-0: Querying
GCP US Central - prod-us-central-0: Ingestion
GCP US Central - prod-us-central-5: Querying
GCP US Central - prod-us-central-5: Ingestion
Grafana Cloud: Prometheus
AWS Australia - prod-ap-southeast-2: Querying
AWS Australia - prod-ap-southeast-2: Ingestion
AWS Brazil - prod-sa-east-1: Querying
AWS Brazil - prod-sa-east-1: Ingestion
AWS Canada - prod-ca-east-0: Querying
AWS Canada - prod-ca-east-0: Ingestion
AWS Germany - prod-eu-west-2: Querying
AWS Germany - prod-eu-west-2: Ingestion
AWS UAE - prod-me-central-1: Querying
AWS UAE - prod-me-central-1: Ingestion
AWS India - prod-ap-south-1: Querying
AWS India - prod-ap-south-1: Ingestion
AWS Japan - prod-ap-northeast-0: Querying
AWS Japan - prod-ap-northeast-0: Ingestion
AWS Singapore - prod-ap-southeast-1: Querying
AWS Singapore - prod-ap-southeast-1: Ingestion
AWS Sweden - prod-eu-north-0: Querying
AWS Sweden - prod-eu-north-0: Ingestion
AWS US East - prod-us-east-0: Querying
AWS US East - prod-us-east-0: Ingestion
We are currently investigating an issue causing degraded write performance across multiple products in the AWS us-east-2 region. Our engineering team is actively working to determine the full scope and impact of the issue and restore normal service levels.
AWS US West - prod-us-west-0: Querying
AWS US West - prod-us-west-0: Ingestion
Azure Netherlands - prod-eu-west-3: Querying
Azure Netherlands - prod-eu-west-3: Ingestion
Azure US Central - us-central2: Querying
Azure US Central - us-central2: Ingestion
Azure US Central - us-central7: Ingestion
GCP Australia - prod-au-southeast-0: Querying
GCP Australia - prod-au-southeast-0: Ingestion
GCP Belgium - prod-eu-west-0: Querying
GCP Belgium - prod-eu-west-0: Ingestion
GCP Brazil - prod-sa-east-0: Querying
GCP Brazil - prod-sa-east-0: Ingestion
GCP India - prod-ap-south-0: Querying
GCP India - prod-ap-south-0: Ingestion
GCP Singapore - prod-ap-southeast-0: Querying
GCP Singapore - prod-ap-southeast-0: Ingestion
GCP UK - prod-gb-south-0: Querying
GCP UK - prod-gb-south-0: Ingestion
GCP US Central - prod-us-central-0: Querying
GCP US Central - prod-us-central-0: Ingestion
GCP US Central - prod-us-central-5: Querying
GCP US Central - prod-us-central-5: Ingestion
GCS US - cortex-prod-04: Ingestion
GCS US - cortex-prod-04: Querying
Grafana Cloud: Prometheus Rules and Alerts
AWS Australia - prod-ap-southeast-2: Rule Evaluation
AWS Australia - prod-ap-southeast-2: Alertmanager and Rules Configuration API
AWS Australia - prod-ap-southeast-2: Alertmanager
AWS Brazil - prod-sa-east-1: Rule Evaluation
AWS Brazil - prod-sa-east-1: Alertmanager and Rules Configuration API
AWS Brazil - prod-sa-east-1: Alertmanager
AWS Canada - prod-ca-east-0: Rule Evaluation
AWS Canada - prod-ca-east-0: Alertmanager and Rules Configuration API
AWS Canada - prod-ca-east-0: Alertmanager
AWS Germany - prod-eu-west-2: Rule Evaluation
AWS Germany - prod-eu-west-2: Alertmanager and Rules Configuration API
AWS Germany - prod-eu-west-2: Alertmanager
AWS UAE - prod-me-central-1: Rule Evaluation
AWS UAE - prod-me-central-1: Alertmanager and Rules Configuration API
AWS UAE - prod-me-central-1: Alertmanager
AWS India - prod-ap-south-1: Rule Evaluation
AWS India - prod-ap-south-1: Alertmanager and Rules Configuration API
AWS Japan - prod-ap-northeast-0: Rule Evaluation
AWS Japan - prod-ap-northeast-0: Alertmanager and Rules Configuration API
AWS Singapore - prod-ap-southeast-1: Rule Evaluation
AWS Singapore - prod-ap-southeast-1: Alertmanager and Rules Configuration API
AWS Singapore - prod-ap-southeast-1: Alertmanager
AWS Sweden - prod-eu-north-0: Alertmanager and Rules Configuration API
AWS Sweden - prod-eu-north-0: Alertmanager
AWS Sweden - prod-eu-north-0: Rule Evaluation
AWS US East - prod-us-east-0: Rule Evaluation
AWS US East - prod-us-east-0: Alertmanager and Rules Configuration API
AWS US East - prod-us-east-0: Alertmanager
AWS US West - prod-us-west-0: Rule Evaluation
AWS US West - prod-us-west-0: Alertmanager and Rules Configuration API
AWS US West - prod-us-west-0: Alertmanager
Azure Netherlands - prod-eu-west-3: Rule Evaluation
Azure Netherlands - prod-eu-west-3: Alertmanager and Rules Configuration API
Azure Netherlands - prod-eu-west-3: Alertmanager
Azure US Central - us-central2: Rule Evaluation
Azure US Central - us-central2: Alertmanager and Rules Configuration API
Azure US Central - us-central2: Alertmanager
GCP Australia - prod-au-southeast-0: Rule Evaluation
GCP Australia - prod-au-southeast-0: Alertmanager and Rules Configuration API
GCP Australia - prod-au-southeast-0: Alertmanager
GCP Belgium - prod-eu-west-0: Rule Evaluation
GCP Belgium - prod-eu-west-0: Alertmanager and Rules Configuration API
GCP Belgium - prod-eu-west-0: Alertmanager
GCP Brazil - prod-sa-east-0: Rule Evaluation
GCP Brazil - prod-sa-east-0: Alertmanager and Rules Configuration API
GCP Brazil - prod-sa-east-0: Alertmanager
GCP India - prod-ap-south-0: Rule Evaluation
GCP India - prod-ap-south-0: Alertmanager and Rules Configuration API
GCP India - prod-ap-south-0: Alertmanager
GCP Singapore - prod-ap-southeast-0: Rule Evaluation
GCP Singapore - prod-ap-southeast-0: Alertmanager and Rules Configuration API
GCP Singapore - prod-ap-southeast-0: Alertmanager
GCP UK - prod-gb-south-0: Rule Evaluation
GCP UK - prod-gb-south-0: Alertmanager and Rules Configuration API
GCP UK - prod-gb-south-0: Alertmanager
GCP US Central - prod-us-central-0: Rule Evaluation
GCP US Central - prod-us-central-0: Alertmanager and Rules Configuration API
GCP US Central - prod-us-central-0: Alertmanager
Grafana Cloud: Loki
AWS Australia - prod-ap-southeast-2
AWS Brazil - prod-sa-east-1
AWS Canada - prod-ca-east-0
AWS Germany - prod-eu-west-2
AWS UAE - prod-me-central-1
AWS India - prod-ap-south-1
AWS Japan - prod-ap-northeast-0
AWS Singapore - prod-ap-southeast-1
AWS Sweden - prod-eu-north-0
AWS US East - prod-us-east-0
AWS US West - prod-us-west-0
Azure Netherlands - prod-eu-west-3
Azure US Central - us-central2
Azure US Central - us-central7
GCP Australia - prod-au-southeast-0
GCP Belgium - prod-eu-west-0
GCP Brazil - prod-sa-east-0
GCP India - prod-ap-south-0
GCP Singapore - prod-ap-southeast-0
GCP UK - prod-gb-south-0
GCP US Central - prod-us-central-0
GCP US Central - prod-us-central-5
Grafana Cloud: Tempo
AWS Australia - prod-ap-southeast-2: Querying
AWS Australia - prod-ap-southeast-2: Ingestion
AWS Australia - prod-ap-southeast-2: Metrics Generator
AWS Brazil - prod-sa-east-1: Querying
AWS Brazil - prod-sa-east-1: Ingestion
AWS Brazil - prod-sa-east-1: Metrics Generator
AWS Canada - prod-ca-east-0: Querying
AWS Canada - prod-ca-east-0: Ingestion
AWS Canada - prod-ca-east-0: Metrics Generator
AWS Germany - prod-eu-west-2: Querying
AWS Germany - prod-eu-west-2: Ingestion
AWS Germany - prod-eu-west-2: Metrics Generator
AWS India - prod-ap-south-1: Querying
AWS India - prod-ap-south-1: Ingestion
AWS India - prod-ap-south-1: Metrics Generator
AWS Japan - prod-ap-northeast-0: Querying
AWS Japan - prod-ap-northeast-0: Ingestion
AWS Japan - prod-ap-northeast-0: Metrics Generator
AWS Singapore - prod-ap-southeast-1: Querying
AWS Singapore - prod-ap-southeast-1: Ingestion
AWS Singapore - prod-ap-southeast-1: Metrics Generator
AWS Sweden - prod-eu-north-0: Querying
AWS Sweden - prod-eu-north-0: Ingestion
AWS Sweden - prod-eu-north-0: Metrics Generator
AWS US East - prod-us-east-0: Querying
AWS US East - prod-us-east-0: Ingestion
AWS US East - prod-us-east-0: Metrics Generator
AWS US West - prod-us-west-0: Querying
AWS US West - prod-us-west-0: Ingestion
AWS US West - prod-us-west-0: Metrics Generator
Azure Germany - prod-eu-west-1: Querying
Azure Germany - prod-eu-west-1: Ingestion
Azure Germany - prod-eu-west-1: Metrics Generator
Azure Netherlands - prod-eu-west-3: Querying
Azure Netherlands - prod-eu-west-3: Ingestion
Azure Netherlands - prod-eu-west-3: Metrics Generator
Azure US Central - us-central2: Querying
Azure US Central - us-central2: Ingestion
Azure US Central - us-central2: Metrics Generator
Azure US Central - us-central7: Ingestion
GCP Australia - prod-au-southeast-0: Querying
GCP Australia - prod-au-southeast-0: Ingestion
GCP Australia - prod-au-southeast-0: Metrics Generator
GCP Belgium - prod-eu-west-0: Querying
GCP Belgium - prod-eu-west-0: Ingestion
GCP Belgium - prod-eu-west-0: Metrics Generator
GCP Brazil - prod-sa-east-0: Querying
GCP Brazil - prod-sa-east-0: Ingestion
GCP Brazil - prod-sa-east-0: Metrics Generator
GCP India - prod-ap-south-0: Querying
GCP India - prod-ap-south-0: Ingestion
GCP India - prod-ap-south-0: Metrics Generator
GCP Singapore - prod-ap-southeast-0: Querying
GCP Singapore - prod-ap-southeast-0: Ingestion
GCP Singapore - prod-ap-southeast-0: Metrics Generator
GCP UK - prod-gb-south-0: Querying
GCP UK - prod-gb-south-0: Ingestion
GCP UK - prod-gb-south-0: Metrics Generator
GCP US Central - prod-us-central-0: Querying
GCP US Central - prod-us-central-0: Ingestion
We're currently investigating an issue with elevated write latency in Hosted Traces prod-us-central-0 region. It's experiencing sustained high write latency since 7:20 AM UTC. Only a small subset of the requests are impacted.
GCP US Central - prod-us-central-0: Metrics Generator
AWS US East - prod-us-east-2: Querying
AWS US East - prod-us-east-2: Ingestion
AWS US East - prod-us-east-2: Metrics Generator
GCP US Central - prod-us-central-5: Querying
GCP US Central - prod-us-central-5: Ingestion
Grafana Cloud: Machine Learning
AWS Australia - prod-au-southeast-1
AWS Brazil - prod-sa-east-1
AWS Canada - prod-ca-east-0
AWS Germany - prod-eu-west-2
AWS India - prod-ap-south-1
AWS Japan - prod-ap-northeast-0
AWS Singapore - prod-ap-southeast-1
AWS Sweden - prod-eu-north-0
AWS US East - prod-us-east-0
AWS US West - prod-us-west-0
Azure Germany - prod-eu-west-1
Azure Netherlands - prod-eu-west-3
Azure US Central - us-central2
GCP Australia - prod-au-southeast-0
GCP Belgium - prod-eu-west-0
GCP Brazil - prod-sa-east-0
GCP India - prod-ap-south-0
GCP Singapore - prod-ap-southeast-0
GCP UK - prod-gb-south-0
Azure Germany - prod-eu-west-1: API
GCP US Central - prod-us-central-0
Azure Germany - prod-eu-west-1: Public Probes
Grafana Cloud: Synthetic Monitoring
AWS Australia - prod-au-southeast-1: API
AWS Australia - prod-au-southeast-1: Public Probes
AWS Brazil - prod-sa-east-1: API
AWS Brazil - prod-sa-east-1: Public Probes
AWS Canada - prod-ca-east-0: API
AWS Canada - prod-ca-east-0: Public Probes
AWS Germany - prod-eu-west-2: API
During the secrets migration in https://status.grafana.com/incidents/47d1q4sphrmj, secrets proxy URLs for some customers updated in the following regions: prod-us-central-0, prod-us-east-0, and prod-eu-west-2. This was an unexpected breaking change affecting a subset of customers.
This will specifically affect customers who are using secrets on private probes behind a firewall.
We are investigating. If your private probes are impacted, we ask you to update firewall rules for the secrets proxy to allow outbound connections to the updated hosts:
gsm-proxy-prod-eu-west-2.grafana.net -> gsm-proxy-prod-eu-west-4.grafana.net gsm-proxy-prod-us-central-0.grafana.net -> gsm-proxy-prod-us-central-4.grafana.net gsm-proxy-prod-us-east-0.grafana.net -> gsm-proxy-prod-us-east-2.grafana.net
Note that this URL change affects only a small subset of customers, the majority of customers will not need to update firewall rules. For affected customers, private probes will show the following error in probe logs, for example: Error during test execution: failed to get secret: Get "https://gsm-proxy-prod-us-east-2.grafana.net/api/v1/secrets/.../decrypt": Forbidden undefined
AWS Germany - prod-eu-west-2: Public Probes
AWS India - prod-ap-south-1: API
AWS India - prod-ap-south-1: Public Probes
AWS Japan - prod-ap-northeast-0: API
AWS Japan - prod-ap-northeast-0: Public Probes
AWS Singapore - prod-ap-southeast-1: API
AWS Singapore - prod-ap-southeast-1: Public Probes
AWS Sweden - prod-eu-north-0: API
AWS Sweden - prod-eu-north-0: Public Probes
AWS US East - prod-us-east-0: API
We are currently investigating an issue causing degraded write performance across multiple products in the AWS us-east-2 region. Our engineering team is actively working to determine the full scope and impact of the issue and restore normal service levels.
During the secrets migration in https://status.grafana.com/incidents/47d1q4sphrmj, secrets proxy URLs for some customers updated in the following regions: prod-us-central-0, prod-us-east-0, and prod-eu-west-2. This was an unexpected breaking change affecting a subset of customers.
This will specifically affect customers who are using secrets on private probes behind a firewall.
We are investigating. If your private probes are impacted, we ask you to update firewall rules for the secrets proxy to allow outbound connections to the updated hosts:
gsm-proxy-prod-eu-west-2.grafana.net -> gsm-proxy-prod-eu-west-4.grafana.net gsm-proxy-prod-us-central-0.grafana.net -> gsm-proxy-prod-us-central-4.grafana.net gsm-proxy-prod-us-east-0.grafana.net -> gsm-proxy-prod-us-east-2.grafana.net
Note that this URL change affects only a small subset of customers, the majority of customers will not need to update firewall rules. For affected customers, private probes will show the following error in probe logs, for example: Error during test execution: failed to get secret: Get "https://gsm-proxy-prod-us-east-2.grafana.net/api/v1/secrets/.../decrypt": Forbidden undefined
AWS US East - prod-us-east-0: Public Probes
We are currently investigating an issue causing degraded write performance across multiple products in the AWS us-east-2 region. Our engineering team is actively working to determine the full scope and impact of the issue and restore normal service levels.
AWS US East - prod-us-east-3: API
AWS US East - prod-us-east-3: Public Probes
AWS US West - prod-us-west-0: API
AWS US West - prod-us-west-0: Public Probes
Azure Netherlands - prod-eu-west-3: API
Azure Netherlands - prod-eu-west-3: Public Probes
Azure US Central - prod-us-central2: API
Azure US Central - prod-us-central2: Public Probes
Azure US Central - prod-us-central7: API
Azure US Central - prod-us-central7: Public Probes
GCP Australia - prod-au-southeast-0: API
GCP Australia - prod-au-southeast-0: Public Probes
GCP Belgium - prod-eu-west-0: API
GCP Belgium - prod-eu-west-0: Public Probes
GCP Brazil - prod-sa-east-0: API
GCP Brazil - prod-sa-east-0: Public Probes
GCP India - prod-ap-south-0: API
GCP India - prod-ap-south-0: Public Probes
GCP Singapore - prod-ap-southeast-0: API
GCP Singapore - prod-ap-southeast-0: Public Probes
GCP UK - prod-gb-south-0: API
GCP UK - prod-gb-south-0: Public Probes
GCP US Central - prod-us-central-0: API
During the secrets migration in https://status.grafana.com/incidents/47d1q4sphrmj, secrets proxy URLs for some customers updated in the following regions: prod-us-central-0, prod-us-east-0, and prod-eu-west-2. This was an unexpected breaking change affecting a subset of customers.
This will specifically affect customers who are using secrets on private probes behind a firewall.
We are investigating. If your private probes are impacted, we ask you to update firewall rules for the secrets proxy to allow outbound connections to the updated hosts:
gsm-proxy-prod-eu-west-2.grafana.net -> gsm-proxy-prod-eu-west-4.grafana.net gsm-proxy-prod-us-central-0.grafana.net -> gsm-proxy-prod-us-central-4.grafana.net gsm-proxy-prod-us-east-0.grafana.net -> gsm-proxy-prod-us-east-2.grafana.net
Note that this URL change affects only a small subset of customers, the majority of customers will not need to update firewall rules. For affected customers, private probes will show the following error in probe logs, for example: Error during test execution: failed to get secret: Get "https://gsm-proxy-prod-us-east-2.grafana.net/api/v1/secrets/.../decrypt": Forbidden undefined
GCP US Central - prod-us-central-0: Public Probes
AWS UAE - prod-me-central-1: API
AWS UAE - prod-me-central-1: Public Probes
AWS UK - prod-gb-south-1: API
AWS UK - prod-gb-south-1: Public Probes
AWS Switzerland - prod-eu-central-0: API
AWS Switzerland - prod-eu-central-0: Public Probes
AWS Indonesia - prod-ap-southeast-2: API
AWS Indonesia - prod-ap-southeast-2: Public Probes
GCP Saudi Arabia - prod-me-central-0: API
GCP Saudi Arabia - prod-me-central-0: Public Probes
GCP US East - prod-us-east-1: API
GCP US East - prod-us-east-1: Public Probes
Grafana Cloud: OTLP Endpoint
AWS Australia - prod-au-southeast-1
AWS Brazil - prod-sa-east-1
AWS Canada - prod-ca-east-0
AWS Germany - prod-eu-west-2
AWS India - prod-ap-south-1
AWS Japan - prod-ap-northeast-0
AWS Singapore - prod-ap-southeast-1
AWS Sweden - prod-eu-north-0
AWS US East - prod-us-east-0
We are currently investigating an issue causing degraded write performance across multiple products in the AWS us-east-2 region. Our engineering team is actively working to determine the full scope and impact of the issue and restore normal service levels.