Groq API Failover: Keep Serving When Groq Goes Down

Groq is on the critical path precisely because it is fast. Teams put it in front of a user because sub-second inference is the product feature, which means a Groq stall is not a background job retrying quietly — it is a spinner your user is watching.

The Three Ways Groq Fails

These need different responses, and a failover path that treats them identically will either flap constantly or never fire at all.

The third row is where most incidents actually live. Groq rarely returns a clean 503 for an hour; it gets slow, requests pile up behind it, and the errors your users see are your own timeouts several layers up the stack.

Where the Traffic Should Go

A fallback is only useful if it is a genuinely independent failure domain. A second Groq key on the same account is not one.

Pick one and configure it. A ranked list of three candidates in a design doc is not a failover path; a base URL, a model id, and a tested credential is.

A Circuit Breaker You Can Paste In

Groq exposes llama-3.3-70b-versatile, mixtral, and the Whisper endpoints at https://api.groq.com/openai/v1. The wrapper below is deliberately small — the parts that matter are the timeout on the primary call and the half-open probe, both of which get dropped from most hand-rolled versions.

// Failover wrapper: Groq primary, second provider on trip.
// Key idea — a slow response is a failure too, so the timeout is part of the check.
const BREAKER = { failures: 0, openedAt: 0 };
const TRIP_AFTER = 3;          // consecutive failures before we stop trying
const COOLDOWN_MS = 60_000;    // how long to stay on the fallback
const PRIMARY_TIMEOUT_MS = 8_000;
const FALLBACK_TIMEOUT_MS = 30_000; // fallbacks are legitimately slower

function breakerOpen() {
  if (BREAKER.failures < TRIP_AFTER) return false;
  if (Date.now() - BREAKER.openedAt > COOLDOWN_MS) {
    BREAKER.failures = 0;       // half-open: let one probe through
    return false;
  }
  return true;
}

export async function complete(messages) {
  if (!breakerOpen()) {
    try {
      const res = await withTimeout(
        primary.chat.completions.create({ model: PRIMARY_MODEL, messages }),
        PRIMARY_TIMEOUT_MS,
      );
      BREAKER.failures = 0;
      return res;
    } catch (err) {
      if (!isRetryable(err)) throw err;   // 400s are your bug, not an outage
      BREAKER.failures += 1;
      if (BREAKER.failures >= TRIP_AFTER) BREAKER.openedAt = Date.now();
      logFailover('groq', err);
    }
  }

  return withTimeout(
    fallback.chat.completions.create({ model: FALLBACK_MODEL, messages }),
    FALLBACK_TIMEOUT_MS,
  );
}

Two things to keep when you adapt it. Do not fail over on 4xx — a 400 or 401 is your bug, and sending it to a second provider just produces the same error twice at double the cost. And keep the cooldown: without it, every request retries a dead provider and pays the full timeout before falling back, which turns a Groq outage into a latency outage of your own.

What Breaks the First Time It Fires

Failover paths fail on their first real invocation more often than they succeed, and the reasons are specific to Groq:

• Model ids are not portable — Groq exposes llama-3.3-70b-versatile; Together AI wants meta-llama/Llama-3.3-70B-Instruct-Turbo. A hard-coded model string is the single most common reason a failover path 404s the first time it fires in production.
• Speed assumptions leak into timeouts — If your client timeout is tuned for Groq — 3 or 5 seconds is common — every fallback provider will look like it is also down, because they are legitimately slower. The fallback call needs its own, longer timeout.
• Rate limits are per-key and per-model — Groq returns 429 with headers for both request and token budgets. Failing over on a 429 is right; retrying the same key on the same model is not.

Test It Before You Need It

Untested failover is worse than none, because it converts a known outage into an unknown one. Three tests, in order of how much they are worth:

1. Break the key. Point the primary client at an invalid Groq credential in staging and confirm traffic lands on the fallback with the output your parser expects. This catches the model-id and response-shape bugs, which are the majority.
2. Break the clock. Set PRIMARY_TIMEOUT_MS to 1ms and confirm the latency path trips too. Most implementations handle errors and quietly ignore slowness.
3. Run it on a schedule. Send a small share of production traffic through the fallback weekly. A path exercised once a quarter is a path that has drifted — the model was deprecated, the key expired, the parser changed.

The Hard Part Is Detection, Not Routing

Every piece of code above is downstream of one question: how fast do you know? groqstatus.com tells you after Groq has confirmed an incident, which is routinely twenty minutes into it and sometimes never for regional degradations. Your own error rate tells you once enough users have already hit it.

Independent probing from outside your stack closes that gap. It is also the only evidence you will have when you need to explain the incident to a customer — or claim credits against a contract that requires you to prove the failure was theirs.

Frequently Asked Questions

Related Guides

• Is Groq Down? Outage Checking Guide
• Groq API Error Codes Explained
• Groq API Rate Limits Explained
• Groq API Latency: Why Responses Get Slow
• Groq API SLA and Uptime Guarantee
• Groq API Alternatives
• Groq API Monitoring Guide
• Live Groq Status Check