PlanetScale Outage History

We've observed service restored and successful credential changes across PlanetScale's environments.

We've identified the resource exhaustion in credential distribution and are monitoring the restoration of service.

We are investigating internal and external reports of Vitess database connectivity problems for certain branches and long delays in credential changes (including creating new users/passwords) propagating into Vitess databases.

AWS us-east-1 cross-zone networking has fully recovered.

A small number of database branches in AWS us-east-1 were impacted by cross-zone networking issues, for which the impact window has ended. If you continue to see impact, please reach out to PlanetScale support.

We are investigating reports of connectivity failures scoped to a single AZ in AWS us-east-1 causing replica delay and connectivity issues.

We've identified an issue with database connectivity for Vitess databases in GCP. We've taken steps to resolve the issue and are making immediate improvements to address root-causes and detection. Databases connectivity is now restored.

We've identified an issue in our networking and deployed mitigation. Connectivity is back to normal. We're continuing to monitor.

We're currently experiencing issues with connectivity to Vitess databases in GCP. We're investigating the issue.

The incident has been resolved. Between 12:17 and 12:24 UTC, the dashboard and API were returning 500 errors for postgres database management pages and API endpoints.

For postgres databases, API requests and the dashboard were returning 500 errors. We've deployed a fix.

The incident has been resolved.

Mitigations are in place and we're seeing signs of recovery. We continue to monitor the health of the system.

Insights metrics are underreporting for Postgres and Vitess databases. Database traffic and connectivity are unaffected. We are actively investigating the issue and are deploying mitigations.

Connection issues have been resolved. If you experience any further issues, please contact our support team.

We remediated the issue, and clients can now connect to their database branches in us-east-1 normally. We will continue monitoring the situation.

We are observing connectivity issues on Postgres in the us-east-1 region. We are remediating.

API and dashboard service is restored.

We have increased the memory available to the API and are monitoring the performance and stability of the API and dashboard to confirm service is restored.

We have identified the cause of the errors - a pattern of requests causing the application to run out of memory and crash - and are working to better handle this traffic pattern now.

We are aware that both our API and dashboard are returning errors. We are investigating the cause and will restore service as soon as possible.

The changes we applied are effectively preventing further unexpected restarts to the high availability component shared by Postgres clusters. We have not seen further restarts to this component nor unplanned failovers in Postgres clusters since applying these changes.

We applied changes to prevent unexpected restarts of the high availability component shared by Postgres clusters. We have not seen further restarts of this component nor unplanned Postgres failovers since applying these changes. We are actively monitoring the situation.

We have identified the likely cause of unexpected restarts to the high availability component used by Postgres clusters, and are applying changes to prevent recurrences.

A high availability component that Postgres clusters depend on is restarting unexpectedly, triggering unplanned failovers in Postgres clusters.

This incident has been resolved.

We're continuing to monitor. We've updated the status history to reflect the time periods where customers experienced connection issues.

At **12:51 PM UTC** and **1:38 PM UTC** some Vitess databases in us-east-1 were impacted by connection errors for approximately 5 minutes due to an issue in our edge network. We resolved the issue and are no longer seeing errors. We will be taking action to protect customers from this issue happening again in the future and apologize for the error.

We're investigating an issue with connection errors to Vitess databases in us-east-1.

Vitess branch creation is available once again.

We're rolling out a mitigation now that will allow Vitess branch creation to succeed while we continue to investigate. No database operations or anything about Postgres database creation are impacted. Next update by 12:00 UTC.

New Vitess databases are being accepted by the PlanetScale API but we are unable to provision the infrastructure to serve them. We've confirmed this impact and are investigating the cause. We will provide further updates by 11:00 UTC.

Problem has been resolved. We saw three nodes fail across three availability zones simultaneously. Impacting databases hosted on those nodes. All databases have been moved to healthy nodes and we're investigating how we can further reduce impact from similar events in the future.

Impacted databases are recovering, we're continuing to monitor.

We're working on a fix for the impacted databases.

We are currently investigating impact.

The issue has been resolved.

A fix has been applied, and we are monitoring recovery.

We are investigating internal errors on some API endpoints. Database connectivity is unaffected.

This incident is resolved.

We've deployed a fix and are seeing error rates resolve.

We're continuing to work on a fix and will share an update once it has been deployed.

We're aware of an issue impacting new branch creation, deploy requests and cluster size changes for Vitess databases deployed to GCP. We are working on a fix.

We've made a fix and the errors have resolved.

The dashboard and API are currently returning 500 errors. We are rolling out a fix.