Sentry Outage History

From approximately 10:55 UTC to 11:10 UTC we experienced a loss of outcomes which power the "Stats" in the organization settings. The data loss is purely informational without impact on billing or ingested data.

This issue has been resolved.

The issue involving some alerts not firing has been mitigated. We will continue to monitor the situation to ensure it does not reoccur.

We have identified an issue with one of our cloud providers that might have caused some alert notifications to not fire starting at around 4:00 AM GMT+2. Since around 1:30 PM GMT+2, alert notifications are now fully operational in the US region, while delivery is still partially degraded in the EU region.

Notifications delivery is now close to fully functional in US and DE. We have identified the root cause as caused by one of our cloud providers, and are closely monitoring the situation.

We are continuing to investigate the degraded notifications.

We are continuing to investigate the degraded notifications.

We're currently investigating degraded notification delivery.

Our auth systems are back operational

We rolled back a configuration change we identified as the root cause of the issue. We'll keep monitoring our systems

We deployed a fix and are monitoring the situation.

We are currently investigating the issue.

Between 06:00-07:00 UTC, we saw increased numbers of API errors in the EU region. The largest impact was between 06:46-07:00. We've implemented a fix and we're no longer seeing elevated errors.

The issue has been resolved

The fix has been deployed and we are monitoring the situation

We have identified the cause of the issue and are deploying a fix

We are currently investigating the issue

Access to sentry.io is now fully operational

We have implemented a fix and are monitoring.

We are currently investigating an issue causing 500 errors and preventing access to Sentry.io organizations in the DE region. The US region remains fully operational.

Latency for sentry.io has returned to normal.

We have implemented a fix and are monitoring.

We are currently investigating slow page loads on sentry.io.

Ingestion for symbolicated issues is back to normal.

We are implementing a fix and monitoring the situation. Non-symbolicated issues are not experiencing delays.

We are experiencing ingestion delays for errors in the US region.

Ingestion and alerting back to normal

We've put in fixes and are monitoring the situation

We have implemented a fix. Ingestion and alerting have returned to normal operation

We are continuing to investigate the issue.

We are still investigating the issue.

We are still investigating the issue.

We are still investigating the issue.

We are experiencing ingestion and alerting delays in us for spans, trace metrics, and logs

Ingestion and alerting are back to steady state in the US region.

Ingestion and alerting are back to normal. We continue to monitor the situation

We are experiencing ingestion and alerting delays in us for spans, transactions, and replays

This incident has been resolved.

We've applied a configuration change to mitigate the API/UI errors being experienced.

We are continuing to investigate increased UI and API timeouts and errors.

We are continuing to investigate increased UI and API timeouts and errors

We are seeing increased UI and API timeouts and errors, due to an overloaded database cluster. Ingestion and alerting should not be affected.

Database performance has returned to normal

We have implemented a fix and are monitoring the situation

We are working on restoring normal service. API and application performance should be improved, ingestion still has delays.

Our event database continues to have issues and we are actively investigating the issue.

Ingestion for spans, replays, and crons is delayed by about 15 minutes.

We are continuing to investigate degraded database performance.

We are seeing increased UI and API timeouts and errors, due to an overloaded database cluster. Ingestion and alerting should not be affected

This incident has been resolved.

We are currently investigating the issue.

This incident has been resolved.

The system is operating as normal, we are monitoring the fix.

Ingestion is now back to normal, we are slowly increasing write traffic to reduce intermittent API failures

We are experiencing delays on our ingestion pipeline. Explorer views are in a degraded state

All application features are now operating as normal.

Ingestion for spans and replays is about 10 minutes behind and catching up. API and Dashboard performance has returned to normal.

We are currently investigating this issue.

Between 07:25 UTC and 07:28 UTC, customers sending data to the legacy ingestion endpoint in our US region experienced a brief ingestion disruption lasting approximately 3 minutes. The issue has been fully resolved and ingestion is operating normally. Customers using modern DSN endpoints (`*.ingest.us.sentry.io`) were not affected. The disruption occurred during a scheduled configuration change. We're investing in additional tooling to detect and mitigate this class of issue faster. We apologize for the inconvenience and appreciate your patience.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

The fix has been rolled out and ingestion is recovered.

The delay has recovered and we are monitoring the fix.

We have identified the issue and are working on a fix.

We are currently investigating this issue.

Login with SSO in Chromium browser was broken. Users experienced form submission failures due to a bug in CSP. The problem is now fixed.

Login with SSO in Chromium browser is broken right now. Signing in using cmd or ctrl + click on the "Login with Provider" button is a workaround you can use while our team fixes the issue.

Ingestion and alerts should be back to normal

US customers may experience delay in data ingestion and alerts. The issue is resolved and we are burning the backlog.

Following the ingestion delay experienced between 08:40 – 08:55 UTC, all systems are now fully operational. Data ingestion is running normally across all data types.

We experienced a delay in data ingestion affecting all data types between 08:40 – 08:55 UTC.

Hot fix merged, all customers can access their subscription pages now.

We are rolling out a fix

We have identified an issue for customers trying to access their subscription page. we are working on a fix

This incident has been resolved

We performed a manual change that caused degraded performance and delayed ingestion in spans, crons, replays, and uptime that caused issues between 13:45 and 13:52 PT. The change has been cancelled and operations have returned to normal

This incident has been resolved and backlogs have been processed.

We have identified the source of postgres load and mitigated it. We are continuing to monitor conditions as backlogs process.

We are continuing to mitigate load in our postgres clusters.

We are continuing to mitigate load in our postgres clusters.

We're continuing to mitigate load in our caching and postgres infrastructure.

We have identified the problem as a load issue in our caching layer and are working to mitigate the issue.

We are investigating processing delays for errors and spans in the US region

Ingestions and alerts are caught up and the incident is resolved.

Customers may experience some delay in ingestion and alerts. We are currently burning the backlog

The incident has been resolved

Ingestion is back to normal, alerts may still be delayed

We are actively mitigating the issue

We are currently investigating the issue. Some alerts may be dropped if they are for older time windows.

Ingestion and alert latency is back to normal.

We are burning the backlog and actively monitoring the progress.

US customers may experience delay in alerts. We've identified the issue and will be putting in a fix.

This incident has been resolved.

EU ingestion has been restored and latency is back to normal levels. US continues to recover and will likely be caught up within the next hour.

We have implemented a fix and are monitoring

The issue has been identified and the fix is being implemented.

We are currently investigating this issue.

Ingestion backlog has finished processing and our system is now operating normally.

Our cloud provider has resolved an underlying problem and our dashboard availability issues have been resolved. We're continuing to process our ingestion backlog and monitor the situation.

We're investigating intermittent failures loading our dashboard (all regions) and increased latency for ingestion of all events types in US

This incident has been resolved. Ingestion in our South America point-of-presence is fully functional.

We are currently investigating the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Due to a transient error, some calls to Seer are failing in EU. We identified the issue and are in the process of deploying a fix.

We have resolved the issue, and our system is now working as expected

Ingestion is back to normal, alerts may still be delayed. We will continue to monitor the recovery.

We are still continuing to process the backlog and are still monitoring the recovery.

We are still continuing to process the backlog and are still monitoring the recovery.

We are still continuing to process the backlog and are still monitoring the recovery.

We are continuing to process the backlog and are still monitoring the recovery.

We've identified and fixed the issue. We are burning the backlog and actively monitoring the progress.

We are investigating the resurfaced Zookeeper issue

We identified a problem with our Zookeeper cluster and fixed it. We now need to ingest the backlog.

We are currently investigating the issue

The incident has been resolved

We are currently investigating

Ingestion has recovered.

The issue has been identified and a fix is being implemented.

We are investigating a delay in ingesting spans in the US region.

Backlog has been processed and crons should be running as normal

The issue has been identified and we working through backlogs.

We are currently investigating this issue.

This incident has been resolved.

Our UI has recovered from intermittent availability issues due to an increase in load on our DB, the load has recovered and the UI shouldn't experience any further issues. We're continuing to monitor.

Our UI is unavailable again, we're investigating the issue.

The UI availability issue has been resolved, we're continuing to monitor the situation.

We are currently investigating this issue.

All backlogs have been processed and no data was lost.

We are now ingesting recent transaction data again and are processing old data in the background.

We have implemented a fix and are monitoring the situation closely. No data has been lost.

We are investigating an issue affecting transaction ingestion in the EU region

The issue has been resolved, and services are operating normally.

A fix has been implemented, and we are monitoring the results.

We are investigating increased request timeouts and 50x errors across multiple API endpoints that may be impacting service availability.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the cause of our current login issues and are working on a fix.

We are continuing to investigate the issue related to logging in to the Sentry frontend in our US and DE regions.

We are investigating an issue with logging in to the Sentry frontend.

Our UI was unavailable between 14:20 UTC and 14:23 UTC due to a bad configuration change, we have reverted the change and service has been restored.

All backlogged errors and transactions have been processed, and ingestion is fully functional.

There was an outage of the issues page and discover feature between approximately 12:30 to 12:40pm PST. The issue has been resolved.

Between 12:01 pm - 12:06 pm UTC our UI was unavailable, we have resolved the issue and apologize for any inconvenience this caused.