Grafana Labs outage history and live incidents

Status: identified Starting at approximately 12:37 UTC, a some Hosted Grafana instances across multiple regions have been experiencing availability issues. Our team has identified the root cause and a fix is currently in progress. Affected instances will recover automatically once the fix has been applied. No action is required on your end. We will provide further updates as they become available and apologize for any disruption this may cause.

Status: investigating We are investigating elevated latency and intermittent errors when ingesting traces for Grafana Cloud Traces in AWS Germany. Customers may see delayed trace ingestion and failed trace writes. Next update by 10:30 UTC.

Status: identified We are currently investigating this issue impacting some GCK6 app project pages. We are in the process of providing a fix for this.

Status: investigating Since approximately 20:30 UTC on October 20, we have observed an increase in alert rule evaluation errors affecting rules that query Grafana Cloud Prometheus (Mimir) data sources in Cluster AWS Germany. Our engineering team is actively investigating and working on remediation.

Status: resolved As of 21 Oct, 06:20 am UTC, we became aware of an issue with possible incorrect evaluation of Synthetic Monitoring alert rules in cluster prod-us-east-3. The issue has been ended 07:00 am UTC

Status: investigating We are currently investigating an issue in the prod-sa-east-1 region causing slow requests, or some stacks struggling to start. We will provide an update as soon as we have more information to share.

Status: investigating We are currently investigating this issue impacting some GCK6 app project pages. We will provide an update once we have more information.

Status: investigating As of  06:30 UTC, we became aware of to an issue with Grafana Cloud unavailibity in the prod-gb-south-0 region. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.

Status: investigating We are currently investigating this issue.

Status: investigating We are currently investigating an issue causing elevated read and ruler errors. We will provide another update as soon as we have more to share.

Status: investigating Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.

Status: investigating We are currently investigating an issue impacting all SMS attempts to Austria for IRM. All SMS attempts to Austria for IRM are currently blocked. We are working on a resolution, and an update will follow as soon as we have more info. Users should configure mobile push or phone calls as notifications.