Grafana Cloud Outage History

From 15:30 to 15:45 UTC and from 16:53 to 17:03 UTC, the prod-us-central-0 and prod-us-central-5 regions saw elevated latency and error rates on the write path. We're monitoring now.

Some users in prod-us-central-0 may be seeing elevated rate of errors when fetching configurations. Our engineers are currently investigating this issue.

We continue to observe a continued period of stability since 19:40 UTC. At this time, we are considering this issue resolved

Since 16:35 UTC we have experienced stability and services are recovering. We are actively monitoring and working to fully stabilize

Our engineering team is investigating issues on the read path of Loki services on AWS US West since today aroun ~13:25UTC. These issues can cause timeouts and 5xx errors when query logs for customers on the cluster. The team is currently working to restore the service.

This incident has been resolved.

We continue to monitor mitigation efforts and work with our CSP.

The impacted has been reduced to slight intermittency. We continue to work with our CSP to reach a complete resolution.

Since today at 11:55 UTC time we are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. We are also reporting impact to Faro performance in the same region. Our engineering team is already working on restoring the service.

Since today at 11:55 UTC time we are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing an issue with Fleet Management in prod-us-central-0. Users in prod-us-central-0 may observe elevated rate of errors when fetching configurations.

This incident has been resolved.

Our team has identified the issue, and are in the process of testing a fix.

We're currently working on an issue where portions of data may be temporarily unretrievable, affecting a small percentage of tenants in all Tempo clusters.

A recent rollout caused the AuthZ (RBAC) service to perform many redundant folder-tree fetches for each authorization check. For a small number of tenants in the prod-us-east-0 and prod-eu-west-2 regions with very large folder trees. This added a few milliseconds to every check, which increased request latency. The approximate timeframe of the impact is: 2026-02-26 17:24:43 UTC to 2026-02-27 14:33:53 UTC. This has now been resolved.

This incident has been resolved.

The issue has been identified and we are working on a fix.

We are investigating issues with incorrect pipeline assignment after custom attributes are assigned.

This incident has been resolved.

Uploads should work without an issue now. However, listing might still result in occasional timeouts - we're actively addressing this problem.

We're experiencing an issue for all Grafana Cloud regions, which manifest in slowness when uploading and listing sourcemaps. The issue most significantly affects users who have a large sourcemap files. We've identified the issue and our team is currently working on a fix.

We are rolling out a mitigation across the environments in these regions, and preemptively where possible to ensure it doesn’t spread elsewhere.

We have seen an increase in latency in our cloud providers services, and are rolling out a change to mitigate the issue. We are monitoring.

We are continuing to investigate this issue alongside the CSP, and have taken steps to escalate through the appropriate channels. The mitigation in place continues to work as expected, and any notable updates will continue to be shared here for tracking.

We are continuing to investigate this issue alongside the CSP. Any notable updates will continue to be shared here for tracking.

We've implemented mitigation in place and are continuing to monitoring and investigating this issue.

We have begun rolling out mitigation steps to reduce write latency in the prod-us-central-0 and prod-us-central-5 regions. While these measures are expected to improve performance, we are continuing to investigate the underlying root cause of the issue. We will provide additional updates as more information becomes available.

Since February 19, we have been investigating an intermittent issue causing increased write latency in the prod-us-central-0 and prod-us-central-5 regions. The issue does not affect all traffic but may result in delayed write operations for some customers. Our engineering team is actively working to identify the root cause and stabilize performance. We will share additional updates as progress is made.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We see intermittent failures and slow start-up of test-runs. We are currently investigating this issue.

We experienced an issue impacting a cell within the Azure prod-us-central-7 region, which occurred between 14:26 and 14:36. Affected users may have noticed increased errors with rule evaluations, as well as a some read/write errors. We have resolved this issue, and will continue to monitor.

This incident has been resolved.

We are continuing to investigate this issue.

We've been alerted to issues querying and are investigating

This incident has been resolved.

Alert instances for Synthetic Monitoring ProbeFailedExecutionsTooHigh provisioned alert rule that are firing during this maintenance might resolve and fire again in the next evaluation. Only the API is affected. Estimated time window is 15:00–16:00 UTC Impacted clusters are: prod-eu-west-5 prod-us-east-4 prod-eu-west-6 prod-sa-east-0 prod-ap-south-0 prod-ap-southeast-0 prod-me-central-0 prod-au-southeast-0 prod-ap-southeast-2

We no longer observed any problems with our services - this incident has been resolved.

The fix has been implemented and services are back to normal. We're currently monitoring health of the services before resolving this incident.

The issue has been identified and our team is currently working on a fix.

Since 12:17 UTC, we're observing an increased latency for data ingestion and rule evaluation in Grafana Cloud Metrics, prod-eu-west-2 region. We're currently investigating the issue.

This incident has been resolved.

We have deployed temporary measures to mitigate the issue, but there was a writes outage from  06:26 to 06:37 UTC.

cloud logging is facing write issues in this region, our team is looking into this.

This incident has been resolved.

We are seeing recovery in impacted environments. We will continue to monitor the progress.

Our engineering team is still investigating this issue.

We are currently investigating an issue resulting in broken queries for the Athena data source.

This incident has been resolved.

We are continuing to monitor for any further issues.

Between 09:47 and 10:14 UTC, Grafana Cloud Logs within a single cell residing in the prod-ap-southeast-1 region experienced an issue affecting write ingestion only. During this time, some log writes may have failed or been delayed. Log reads were not impacted and remained fully available throughout the incident. Our engineering team quickly identified the cause of the issue and are monitoring the service. The service has been operating normally since 10:14 UTC.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

The issue causing the incident has been identified, and the fix has been deployed. All new test runs work consistently

We are continuing to work on a fix for this issue.

We encountered a subtle bug which caused our test-run finalization process to read from stale threshold status because of a synchronization issue. We have since resolved the bug, and new test runs will work properly. Impacted test runs will need to be fixed via further correction on our end. We will continue to provide updates on the progress of the fix for impacted test runs.

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

As of 22:55 UTC, we have observed marked improvement with the incident impacting IRM and OnCall. We are still investigating and will continue to monitor and provide updates.

We are currently investigating an issue impacting some customers when accessing Grafana Oncall and IRM. Impacted customers may experience long load times, or even time-outs when attempting to access these components. We'll provide more information as it becomes available.

We consider this incident as resolved since the latency hasn't been elevated since the fix was applied. The issue was caused by a latency spike in a downstream dependency, causing an increased backpressure on the Hosted Traces ingestion path, which degraded gateway performance and resulted in an elevated write latency. After clearing the affected gateway services the degraded state went away and normal operation was restored.

The issue was identified and a fix was applied. After applying the fix, latency went down to a regular and expected value. We're currently monitoring the component's health before resolving the incident.

We're currently investigating an issue with elevated write latency in Hosted Traces prod-us-central-0 region. It's experiencing sustained high write latency since 7:20 AM UTC. Only a small subset of the requests are impacted.