Cohere API Failover: Keep Serving When Cohere Goes Down

Cohere is unusual in that most production dependencies on it are not chat. Embed and Rerank sit inside retrieval pipelines, and those have a failover constraint generation does not: you cannot swap an embedding model mid-flight without invalidating the vectors already in your index.

The Three Ways Cohere Fails

These need different responses, and a failover path that treats them identically will either flap constantly or never fire at all.

The third row is where most incidents actually live. Cohere rarely returns a clean 503 for an hour; it gets slow, requests pile up behind it, and the errors your users see are your own timeouts several layers up the stack.

Where the Traffic Should Go

A fallback is only useful if it is a genuinely independent failure domain. A second Cohere key on the same account is not one.

Pick one and configure it. A ranked list of three candidates in a design doc is not a failover path; a base URL, a model id, and a tested credential is.

A Circuit Breaker You Can Paste In

Cohere exposes Command for generation, plus Embed and Rerank at https://api.cohere.com/v2. The wrapper below is deliberately small — the parts that matter are the timeout on the primary call and the half-open probe, both of which get dropped from most hand-rolled versions.

// Failover wrapper: Cohere primary, second provider on trip.
// Key idea — a slow response is a failure too, so the timeout is part of the check.
const BREAKER = { failures: 0, openedAt: 0 };
const TRIP_AFTER = 3;          // consecutive failures before we stop trying
const COOLDOWN_MS = 60_000;    // how long to stay on the fallback
const PRIMARY_TIMEOUT_MS = 8_000;
const FALLBACK_TIMEOUT_MS = 30_000; // fallbacks are legitimately slower

function breakerOpen() {
  if (BREAKER.failures < TRIP_AFTER) return false;
  if (Date.now() - BREAKER.openedAt > COOLDOWN_MS) {
    BREAKER.failures = 0;       // half-open: let one probe through
    return false;
  }
  return true;
}

export async function complete(messages) {
  if (!breakerOpen()) {
    try {
      const res = await withTimeout(
        primary.chat.completions.create({ model: PRIMARY_MODEL, messages }),
        PRIMARY_TIMEOUT_MS,
      );
      BREAKER.failures = 0;
      return res;
    } catch (err) {
      if (!isRetryable(err)) throw err;   // 400s are your bug, not an outage
      BREAKER.failures += 1;
      if (BREAKER.failures >= TRIP_AFTER) BREAKER.openedAt = Date.now();
      logFailover('cohere', err);
    }
  }

  return withTimeout(
    fallback.chat.completions.create({ model: FALLBACK_MODEL, messages }),
    FALLBACK_TIMEOUT_MS,
  );
}

Two things to keep when you adapt it. Do not fail over on 4xx — a 400 or 401 is your bug, and sending it to a second provider just produces the same error twice at double the cost. And keep the cooldown: without it, every request retries a dead provider and pays the full timeout before falling back, which turns a Cohere outage into a latency outage of your own.

What Breaks the First Time It Fires

Failover paths fail on their first real invocation more often than they succeed, and the reasons are specific to Cohere:

• Embeddings are not interchangeable — Cohere embed-v4 vectors and OpenAI vectors are not in the same space. You cannot query a Cohere-built index with an OpenAI query vector — the results are noise, not degraded results, and nothing in the response says so.
• Rerank failure degrades quality, not availability — If Rerank fails and you return unranked retrieval results, the request succeeds and the answers get quietly worse. This never shows up in an error-rate dashboard; it shows up in user complaints two weeks later.
• v1 and v2 response shapes differ — Cohere's own API versions changed the chat response format. Pin the version explicitly in the client and in the fallback path so a library upgrade does not become an incident.

Test It Before You Need It

Untested failover is worse than none, because it converts a known outage into an unknown one. Three tests, in order of how much they are worth:

1. Break the key. Point the primary client at an invalid Cohere credential in staging and confirm traffic lands on the fallback with the output your parser expects. This catches the model-id and response-shape bugs, which are the majority.
2. Break the clock. Set PRIMARY_TIMEOUT_MS to 1ms and confirm the latency path trips too. Most implementations handle errors and quietly ignore slowness.
3. Run it on a schedule. Send a small share of production traffic through the fallback weekly. A path exercised once a quarter is a path that has drifted — the model was deprecated, the key expired, the parser changed.

The Hard Part Is Detection, Not Routing

Every piece of code above is downstream of one question: how fast do you know? status.cohere.com tells you after Cohere has confirmed an incident, which is routinely twenty minutes into it and sometimes never for regional degradations. Your own error rate tells you once enough users have already hit it.

Independent probing from outside your stack closes that gap. It is also the only evidence you will have when you need to explain the incident to a customer — or claim credits against a contract that requires you to prove the failure was theirs.

Frequently Asked Questions

Related Guides

• Is Cohere Down? Outage Checking Guide
• Cohere API Error Codes Explained
• Cohere API Rate Limits Explained
• Cohere API Latency: Why Responses Get Slow
• Cohere API SLA and Uptime Guarantee
• Cohere API Alternatives
• Cohere API Monitoring Guide
• Live Cohere Status Check