LaunchDarkly Outage History

The issue with Observability Traces has been resolved. Missing trace data has been recovered.

The issue with Observability Traces has been identified and a fix has been implemented. We have recovered most trace data that was missing from July 30th 5:16pm to July 31st 10:59am PT. We will resolve the incident once we fully complete data recovery (expected to complete by 5pm PT).

Customers are experiencing missing Observability Traces ingested from July 30th 5:16pm until July 31st 10:59am PT. Observability Logs / Metrics are unaffected. We have identified the issue and are continuing our work to recover the missing Traces data.

All customers are experiencing missing Observability Traces starting July 30th at 5:16pm PT. We are investigating and will provide updates as they become available. Trace data is not lost and we are working to recover it.

This incident has been resolved.

All LaunchDarkly services are operational, though some customers need to take action to restore their connections to our Flag Delivery Network. Customers continue to successfully reestablish connections by completing the action guidance below and we continue to support customers working through the residual impact. SDKs and Relay Proxy instances which attempted to establish a connection to LaunchDarkly on July 10 between 09:11 and 11:42 AM PT were intermittently affected. This includes situations where the host application was restarted or redeployed during this time. SDKs and Relay Proxy instances which did not attempt to establish a connection during this time were unaffected. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it, but we recommend reviewing your application logs to confirm if restarting your applications is necessary. - If you use server-side SDKs without the Relay Proxy, please restart the application to re-establish its connection to LaunchDarkly. - Client-side and Mobile SDKs will reconnect automatically as their applications attempt to reconnect to LaunchDarkly under normal circumstances, for example when the application loads or the device regains network connectivity. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

We are continuing to monitor for any further issues, see our previous update for recovery steps.

Customers continue to successfully reestablish connections to our streaming services. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use server-side SDKs, please restart the application to re-establish its connection to LaunchDarkly. - If re-establishing connection to the server-side SDK does not work, please create and use a second SDK key. - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it. - Client-side SDKs will reconnect automatically as they restart. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

Our Flag Delivery services have recovered and we're observing an increase in customer traffic successfully reestablish connections to our streaming services. Customers who continue to encounter 401 connection errors in their SDKs or Relay Proxy instances are advised to restart their applications and/or Relay Proxy instances.

We are investigating continued impact to customers' mobile and server SDK caused by the previous incident (https://status.launchdarkly.com/incidents/chywwz01ptb0). If customers are receiving SDK or Relay Proxy 401s please re-establish connections.

The issue with session replay and error ingest has been resolved. We've caught up on all backlogged data as of 1:01pm PT.

The issue with session replay and error ingest has been identified and a fix has been implemented. We expect to catch up on all backlogged data ingest in ~15 minutes. No data has been lost.

Customers are experiencing delayed session replay and error ingest as of 11:03am PT. We have identified the issue and are continuing our work to resolve it. Data loss is not expected.

Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue and are working towards releasing a fix.

We are currently investigating an issue where experiments using frequentist analysis cannot be saved or started.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some customers are not receiving account verification emails. Our team is actively working to mitigate this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating elevated error rates on our event data ingestion endpoints affecting production customers, between approximately 7:00 AM PDT and 7:15 AM PDT. During this window, a portion of feature flag, experimentation, and AI/agent event data may have been rejected or delayed. Flag delivery and evaluation are not affected. SDKs continue to serve flags normally. Observability ingest is not affected.

We're now seeing restored access in the affected regions.

We've identified the source of the problem for areas that were still seeing issues with the LaunchDarkly.com and docs and have a solution in place. We are now monitoring to ensure everything is resolved.

Incident seems to be resolved for the majority of users, however, we are continuing to see some issues for in the San Jose and Los Angeles region. We're working to remediate for these regions.

A fix has been implemented and monitoring the results.

LaunchDarkly web application and flag delivery unaffected, but documentation and LaunchDarkly.com are unavailable for some users. A fix is on the way.

The issue with Observability data ingest for OpenTelemetry data has been resolved. Impacted services have returned to normal operation.

The issue with Observability data ingest for OpenTelemetry data has been identified and a fix has been implemented. We are continuing to monitor the performance of impacted services. We will continue to update this page until it is resolved. Observability signals (errors / logs / traces / OTel metrics) are currently delayed by <1 minutes on average.

Some customers are experiencing delayed Observability data ingest for OpenTelemetry data. We have identified the issue and are continuing our work to resolve it. Observability signals (errors / logs / traces / OTel metrics) are currently delayed by ~6 minutes on average. Data ingest delays began at 10:02am and are improving as of 10:16am PT.

This incident has been resolved.

We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

We have made partial progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

We are continuing to monitor for any further issues.

We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.

We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.

We are continuing to monitor for any further issues.

We are monitoring the status of our services.

A fix has been implemented and we are monitoring the results.

We are currently experiencing delays in event ingestion impacting multiple product areas starting at 12:40pm PT. We have identified the issue and are working on a fix. No data has been lost.

All queued events have been processed and data ingestion is operating normally. No data was lost during this incident.

We have identified the issue and a fix is in progress. The backlog is actively decreasing. No data has been lost — all queued events will be processed.

We are currently experiencing a delay in processing Session Replay and Error event data. Incoming data is being queued and will be processed — no data has been lost. Customers may observe a temporary lag in session and error data appearing in the LaunchDarkly UI.

We've implemented a fix and have mitigated impact as of 9:25pm PT. Streaming SDKs should be able to connect successfully at this time.

We're identified an infrastructure issue causing elevated error rates and are working on a fix.

We are investigating elevated error rates for streaming initialization in the APAC region. Some customers using streaming server-side SDKs may have experienced longer than normal initialization times or timeouts when connecting to LaunchDarkly.

All experiment and multi-armed bandit result data is caught up. No data was lost.

We have implemented a fix and are catching up on backlogged data. We estimate that we will be fully caught up in about 1 hour.

We have identified the issue and are continuing our work to resolve it.

We are currently investigating delays in experiment and multi-armed bandit results. Data is currently delayed by 1-2 hours. No data loss is expected.

This incident has been resolved.

We've implemented a fix and have mitigated impact as of 8:35am PT. Streaming SDKs should be able to connect successfully at this time.

We're continuing to address the root cause of the elevated error rate. Server SDKs may require additional time to initialize while requests are retried.

We're identified an infrastructure issue causing elevated error rates and are working on a fix.

We are investigating elevated error rates for streaming initialization in the Europe region. Some customers using streaming server-side SDKs may have experienced longer than normal initialization times or timeouts when connecting to LaunchDarkly.

Experimentation results processing and data warehouse export was caught up as of 12:28 pm PT.

We have implemented a fix and are catching up on backlogged experimentation results and data export. We estimate that we will be fully caught up in about 8 hours.

We've identified the root cause of the experimentation results processing and data export delays and are working on implementing a fix.

We are investigating an issue causing delayed experimentation results processing and delayed experimentation data warehouse export.

We have validated a fix to resolve all remaining experiments with inaccurate sample ratio mismatches (SRMs) or empty experiment results. The vast majority of experiments and experimentation customers were not impacted.

We've released a fix to prevent this issue in new experiments and iterations. We're working to resolve this issue for experiment data recently processed.

We've identified the issue and are implementing a fix for this behavior.

We are aware of an issue affecting some customers that may result in sample ratio mismatches (SRMs) or empty experiment results if flag variations are edited between experiment iterations.

We've caught up on the streaming data export backlog.

We've identified latency in Pub/Sub export that were causing streaming data export delays that has since resolved. We're catching up on the export backlog now.

We are investigating a delay with streaming data export delivery. No data is lost, but customers using streaming data export may notice a multi-minute lag in receiving data.

Observability alerts are functioning and delayed alerts have been redelivered.

A fix has been deployed for Observability alerting and we are working to redeliver delayed notifications.

We've identified the root cause delaying Observability alert notifications and are deploying a fix. All other notification types are operational.

We are currently investigating an issue prevent Observability alert notifications from being delivered.

We've fully caught up on the metrics data backlog.

We're catching up on backlogged metric data in EU and are monitoring progress. No metric data has been lost.

We've identified that metrics ingest was delayed between 11:11 - 11:27am ET. No metric data has been lost. We're working on catching up on the delayed data.

We are investigating an issue causing under-reporting of customer metrics in the EU region.

We've addressed an issue that prevented warehouse data export from 12pm to 6pm PT on April 23. All missing data has been retroactively pushed to customer warehouses.

The issue has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue where observability alerts are not delivering for Slack notifications to all channels. DM alerts are still functional.

The delayed processing of Session Replay and Error event data has been resolved. All queued events have been processed and data ingestion is operating normally. No data was lost during this incident. Thank you for your patience.

The delayed processing of Session Replay and Error event data has been resolved. All queued events have been processed and data ingestion is operating normally. No data was lost during this incident. Thank you for your patience.

We have identified the cause of the delayed Session Replay and Error event processing and a fix is in progress. The backlog is actively decreasing. No data has been lost — all queued events will be processed.

We are currently experiencing a delay in processing Session Replay and Error event data. Incoming data is being queued and will be processed — no data has been lost. Customers may observe a temporary lag in session and error data appearing in the LaunchDarkly UI. We have identified the root cause as elevated database utilization and are actively working to resolve the backlog. We will provide updates as processing returns to normal.

Our team completed the data restoration for previously affected experiment iterations. Complete data is now available for all experiment iterations in the UI.

Our team continues to work on restoring data for previously affected experiment iterations. Complete data restoration expected by tomorrow.

Our team continues to work on restoring data for previously affected experiment iterations. We will provide a further update once full historical data recovery is complete.

We have implemented a mitigation that restores data availability for all current experiment iterations. Affected accounts should now see up-to-date reporting and exposure data in the UI. Our team continues to work on restoring data for previously affected experiment iterations. We will provide a further update once full historical data recovery is complete.

The issue has been identified and a fix is being implemented.

We are aware of an issue affecting experiment reporting data for a subset of accounts. Affected users may see incomplete data for some experiment iterations in the UI.

The issue with Event processing has been resolved. Impacted services have returned to normal operation. Flag Delivery was not impacted. There was no data loss experienced.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

Event processing is currently delayed for and will show stale data for several product areas, including: - Autogenerated metric creation - Data Export - Experimentation No data has been lost.

We've caught up on the session/error data backlog.

We've identified the root cause and have deployed a fix. We're catching up on the session/error data backlog and should be caught up in ~15 minutes. No data has been lost.

Sessions and errors may be delayed by up to 1 hours. We are investigating the root cause. No data is lost.

This incident has been resolved. Telemetry data was dropped between 5:03 PM and 7:30 PM PT. All systems are now operating normally.

We are currently investigating an issue affecting OTel telemetry ingestion in LaunchDarkly Observability. Some telemetry data may not be processed as expected. Our team is actively working to identify the root cause and mitigate impact.

This issue has been resolved. Observability ingestion from AWS CloudWatch Metric Streams and CloudWatch Firehose log export has returned to normal. Data sent during the incident window may not have been ingested, and customers may see gaps in metrics or logs for that period.

Some customers using LaunchDarkly Observability with AWS CloudWatch Metric Stream and/or CloudWatch Firehose log export may experience delayed ingestion of metrics/logs into Observability. The impact is limited to Observability data ingestion and related dashboards/alerts.

Customers are seeing expected behavior when accessing the LaunchDarkly UI and API.

We are investigating an issue where a small number of customers may experience errors when accessing the LaunchDarkly UI and API. Feature flag delivery is not impacted. We will provide updates as the investigation continues.

This incident has been resolved and all data processing pipelines are fully caught up. No data was lost.

A fix has been implemented and our event processing pipeline for Observability and Opentelemetry are fully caught up. We're continuing to monitor as our event processing pipeline catches up for Flag Status, Evaluations, and Contexts.

We have identified the issue and are continuing our work to resolve it.

All customers are experiencing data ingestion delays with the following: - Observability sessions and errors - Opentelemetry logs, traces, and metrics - Flag status - Evaluations - Contexts We are investigating and will provide updates as they become available. No data loss is expected.

This incident has been resolved.

We believe the issue is resolved for all customers. We're continuing to monitor the situation.

Some customers using Okta SCIM are encountering errors when provisioning and managing LaunchDarkly members. We continue to work on a remediation and have engaged with Okta's support team.

Some customers are experiencing errors when configuring LaunchDarkly with Okta SCIM. We have identified the issue and are continuing our work to resolve it.

This incident has been resolved.

A fix has been implemented for the issue preventing editing some JSON flag variations.

We've identified a front-end issue that is causing issues editing certain JSON flag variations and are working on a fix.

Customers may experience issues editing JSON flag variation. We are investigating the root cause and will provide updates shortly.

This incident has been resolved.

Events are caught up.

A fix has been implemented and we are monitoring the results. We expect to catch up on all events within the next 15 minutes, no data loss is expected.

We are currently experiencing delays with guarded releases event ingestion. We are investigating and will provide updates as they become available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the cause of the ingest delay and are catching up on the backlogged messages. We expect to be caught up on all delayed sessions and errors in the next hour. Data loss is not expected.

Sessions and errors may be delayed by up to 3 hours. We are investigating the root cause.