Grafana Cloud Outage History

We have been observing stability for a period of time and will mark the incident as resolved at this time.

We are continuing to investigate this issue with our CSP, and will provide updates as they become available.

We are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.

This incident has been resolved.

We are also reporting impact to Faro performance in the same region. We are continuing to investigate this issue.

We are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.

This incident has been resolved.

From 15:30 to 15:45 UTC and from 16:53 to 17:03 UTC, the prod-us-central-0 and prod-us-central-5 regions saw elevated latency and error rates on the write path. We're monitoring now.

This incident has been resolved.

Our engineering team continues to work towards a resolution for this issue.

Some users in prod-us-central-0 may be seeing elevated rate of errors when fetching configurations. Our engineers are currently investigating this issue.

This incident has been resolved.

There has been a reoccurrence o the issues on the Read path of Loki services on AWS US West since yesterday 9th around ~17:15UTC. The issue has been identified, and resolutions steps has been taken to restore full service. We are currently monitoring the service status. The impact of this includes timeouts and 5xx errors when query logs for customers on this cluster.

We continue to observe a continued period of stability since 19:40 UTC. At this time, we are considering this issue resolved

Since 16:35 UTC we have experienced stability and services are recovering. We are actively monitoring and working to fully stabilize

Our engineering team is investigating issues on the read path of Loki services on AWS US West since today aroun ~13:25UTC. These issues can cause timeouts and 5xx errors when query logs for customers on the cluster. The team is currently working to restore the service.

This incident has been resolved.

We continue to monitor mitigation efforts and work with our CSP.

The impacted has been reduced to slight intermittency. We continue to work with our CSP to reach a complete resolution.

Since today at 11:55 UTC time we are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. We are also reporting impact to Faro performance in the same region. Our engineering team is already working on restoring the service.

Since today at 11:55 UTC time we are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing an issue with Fleet Management in prod-us-central-0. Users in prod-us-central-0 may observe elevated rate of errors when fetching configurations.

This incident is now resolved. During the incident the Cloud Metrics platform experienced intermittent latency spikes communicating with a backend cloud service in the prod-us-central-0 and prod-us-central-5 regions. During the incident the internal CSP-facing issue was escalated to a P1. After determining the scope of the latency spikes was limited to only one availability zone, the team mitigated the situation by migrating all write traffic from to the single nearly unaffected availability zone. As the CSP service team attempted to remedy the situation, the situation became worse and began affecting the previously unaffected zone. Given this, another mitigation path was needed. Changing the connection strategy employed by Cloud Metrics to a different method was deployed to all environments, stabilizing the write path once again as we found the different connection method was more reliable and not affected by these increases in latency. We have migrated all tenants back to multi-zone write paths and are happy with and confident in the current method of connectivity to the backend cloud service, which is the one we migrated to during the course of the incident. We have no immediate plans to use the previous problematic connectivity method for the foreseeable future.

We are rolling out a mitigation across the environments in these regions, and preemptively where possible to ensure it doesn’t spread elsewhere.

We have seen an increase in latency in our cloud providers services, and are rolling out a change to mitigate the issue. We are monitoring.

We are continuing to investigate this issue alongside the CSP, and have taken steps to escalate through the appropriate channels. The mitigation in place continues to work as expected, and any notable updates will continue to be shared here for tracking.

We are continuing to investigate this issue alongside the CSP. Any notable updates will continue to be shared here for tracking.

We've implemented mitigation in place and are continuing to monitoring and investigating this issue.

We have begun rolling out mitigation steps to reduce write latency in the prod-us-central-0 and prod-us-central-5 regions. While these measures are expected to improve performance, we are continuing to investigate the underlying root cause of the issue. We will provide additional updates as more information becomes available.

Since February 19, we have been investigating an intermittent issue causing increased write latency in the prod-us-central-0 and prod-us-central-5 regions. The issue does not affect all traffic but may result in delayed write operations for some customers. Our engineering team is actively working to identify the root cause and stabilize performance. We will share additional updates as progress is made.

This incident has been resolved.

Our team has identified the issue, and are in the process of testing a fix.

We're currently working on an issue where portions of data may be temporarily unretrievable, affecting a small percentage of tenants in all Tempo clusters.

A recent rollout caused the AuthZ (RBAC) service to perform many redundant folder-tree fetches for each authorization check. For a small number of tenants in the prod-us-east-0 and prod-eu-west-2 regions with very large folder trees. This added a few milliseconds to every check, which increased request latency. The approximate timeframe of the impact is: 2026-02-26 17:24:43 UTC to 2026-02-27 14:33:53 UTC. This has now been resolved.

This incident has been resolved.

The issue has been identified and we are working on a fix.

We are investigating issues with incorrect pipeline assignment after custom attributes are assigned.

This incident has been resolved.

Uploads should work without an issue now. However, listing might still result in occasional timeouts - we're actively addressing this problem.

We're experiencing an issue for all Grafana Cloud regions, which manifest in slowness when uploading and listing sourcemaps. The issue most significantly affects users who have a large sourcemap files. We've identified the issue and our team is currently working on a fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We see intermittent failures and slow start-up of test-runs. We are currently investigating this issue.

We experienced an issue impacting a cell within the Azure prod-us-central-7 region, which occurred between 14:26 and 14:36. Affected users may have noticed increased errors with rule evaluations, as well as a some read/write errors. We have resolved this issue, and will continue to monitor.

This incident has been resolved.

We are continuing to investigate this issue.

We've been alerted to issues querying and are investigating

This incident has been resolved.

Alert instances for Synthetic Monitoring ProbeFailedExecutionsTooHigh provisioned alert rule that are firing during this maintenance might resolve and fire again in the next evaluation. Only the API is affected. Estimated time window is 15:00–16:00 UTC Impacted clusters are: prod-eu-west-5 prod-us-east-4 prod-eu-west-6 prod-sa-east-0 prod-ap-south-0 prod-ap-southeast-0 prod-me-central-0 prod-au-southeast-0 prod-ap-southeast-2

We no longer observed any problems with our services - this incident has been resolved.

The fix has been implemented and services are back to normal. We're currently monitoring health of the services before resolving this incident.

The issue has been identified and our team is currently working on a fix.

Since 12:17 UTC, we're observing an increased latency for data ingestion and rule evaluation in Grafana Cloud Metrics, prod-eu-west-2 region. We're currently investigating the issue.

This incident has been resolved.

We have deployed temporary measures to mitigate the issue, but there was a writes outage from  06:26 to 06:37 UTC.

cloud logging is facing write issues in this region, our team is looking into this.

This incident has been resolved.

We are seeing recovery in impacted environments. We will continue to monitor the progress.

Our engineering team is still investigating this issue.

We are currently investigating an issue resulting in broken queries for the Athena data source.

This incident has been resolved.

We are continuing to monitor for any further issues.

Between 09:47 and 10:14 UTC, Grafana Cloud Logs within a single cell residing in the prod-ap-southeast-1 region experienced an issue affecting write ingestion only. During this time, some log writes may have failed or been delayed. Log reads were not impacted and remained fully available throughout the incident. Our engineering team quickly identified the cause of the issue and are monitoring the service. The service has been operating normally since 10:14 UTC.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

The issue causing the incident has been identified, and the fix has been deployed. All new test runs work consistently

We are continuing to work on a fix for this issue.

We encountered a subtle bug which caused our test-run finalization process to read from stale threshold status because of a synchronization issue. We have since resolved the bug, and new test runs will work properly. Impacted test runs will need to be fixed via further correction on our end. We will continue to provide updates on the progress of the fix for impacted test runs.