Cohere API 502 Bad Gateway

A 502 is the only 5xx that is definitionally written by someone other than the component that failed. A gateway asked an upstream for a response, got something it could not use — an empty body, a reset connection, a malformed reply — and substituted an error of its own. Everything useful about a 502 follows from that one fact: your request was fine, the model may never have seen it, and the error you are reading was authored by a middleman.

Which middleman is the entire question. It might be Cohere's own edge. It might equally be your ingress controller, a service-mesh sidecar, a corporate egress proxy, a CDN, or a routing library sitting between your code and the provider. These produce an identical status code and wildly different remediation, and teams routinely spend an incident refreshing a status page for a failure their own infrastructure emitted.

502 against the rest of the 5xx family

These five get filed together as “the API is broken” and then handed one retry policy, which guarantees that the policy is wrong for at least three of them. The column that matters is the last one, because it is the only one that changes what you do next.

502 and 504 are siblings and both are written by gateways — the difference is that a 504 means the intermediary waited and gave up, while a 502 means it got an answer it could not use. Neither tells you the upstream failed. The capacity case is covered separately in the Cohere 503 guide, and the throttling case in the Cohere 429 guide.

Why Cohere returns 502 specifically

Cohere is three workloads behind one key — chat, embed and rerank — and they generate wildly different request shapes. Chat sends a small body and streams a small one back. Embed, during an indexing run, sends batches of documents that can be megabytes per call. Rerank sits in between. Any intermediary in the path with an opinion about request body size will express that opinion on the embed leg only, and 502 is one of the ways it does so.

That is why a Cohere 502 so often appears to be endpoint-specific in a way that makes no sense as a provider incident: chat is fine, rerank is fine, embed fails on the large batches and succeeds on the small ones. A provider outage does not care how big your body is. A reverse proxy does.

Sample rather than probe once. One request cannot distinguish a total failure from the far more common partial one, and the two call for different responses:

for i in $(seq 1 10); do
  curl -s -o /tmp/body -w "%{http_code} %{time_total}s %{content_type}\n" \
    https://api.cohere.com/v2/chat \
    -H "Authorization: Bearer $CO_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"command-r-plus","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'
done

Three columns, three answers. The status code tells you whether it failed, the elapsed time tells you whether something waited before failing, and the content type tells you who wrote the error. Ten out of ten failing with a JSON body is an incident. Two out of ten failing with an HTML body is your own infrastructure having a moment, and no amount of provider failover will fix it.

The Cohere trap: the bad gateway is often yours, and it fails by batch size

Before you file anything with Cohere, test the same batch at a tenth of the size. If the small batch succeeds and the large one 502s, deterministically, the ceiling is a body-size or buffer limit somewhere in your egress path — an nginx client_max_body_size, an ingress controller default, a corporate proxy — and it will follow you to any provider you fail over to. This is the fastest and cheapest test in this entire guide and it resolves a large share of Cohere 502 reports on the spot.

The second-order damage is worse than the error. A bulk embed run that fails partway leaves the index inconsistent rather than merely incomplete, and a retrieval system querying a half-built index returns confident nonsense with no error anywhere. Checkpoint by document id and resume, never restart — restarting re-embeds under a possibly different model version and leaves you with vectors that are not comparable to the ones already stored.

The streaming case, where the 502 never reaches your status-code metrics

If you stream, a large share of your gateway failures will never be counted as 502s at all. The response headers arrived long ago with a 200; the failure happens partway through the token stream, and what your client receives is a truncated event or an HTML error fragment spliced into a channel that was expecting server-sent events. Most SDKs raise that as a decoding error, so it lands in your logs as a parse failure in the JSON layer and your Cohere error-rate panel stays flat through the entire incident.

• Assert on the terminal event. An iterator finishing is not the same as a stream completing. If the terminal marker never arrived, the response was truncated, whatever the status code said.
• Count partial responses as their own class. Not a success, not a 5xx. They behave differently and they are the metric that moves first.
• Record bytes and tokens received on failure. A stream that died at token three is a different problem from one that died at token three hundred, and only one of them is worth showing the user.
• Decide the resume policy in advance. Re-issuing the whole prompt is the honest default and it bills twice; showing the partial output with a visible label is often the better product answer.

A retry policy that fits a 502 specifically

502s are the most retry-friendly 5xx there is — they are usually brief, usually one hop, and usually gone by the time you look. That makes blanket “never retry 5xx” advice actively wrong here. Four rules keep the retry from becoming its own incident:

• One quick jittered retry first. Randomise across the full window rather than adding a small wobble, so a fleet of clients does not resynchronise and re-burst against the first recovered backend.
• Assume the work may have happened. The gateway could not read the response; that does not mean the upstream never produced one. Send an idempotency key for anything with a side effect, and expect that a retried completion can bill twice.
• Budget by ratio, not by count. Allow retries as a fraction of recent successes so the allowance collapses on its own when almost everything is failing — no deploy required. See the Cohere retry budget guide.
• Retry against a deadline. If the user-facing request gave up eight seconds ago, attempt three is load on behalf of nobody. Propagate the deadline and skip attempts that cannot finish inside it.

If 502s persist past a handful of retries, they have stopped being transient and the failover machinery should take over — which only works if it was kept warm. The Cohere failover guide covers keeping a second path tested rather than merely configured, and the circuit breaker guide covers stopping the retries automatically.

Frequently Asked Questions

Related Cohere Guides

• Cohere API 503 Service Unavailable
• Cohere API 429 Too Many Requests
• Cohere API 500 Internal Server Error
• Cohere API Timeout Errors
• Cohere API Error Codes Reference
• Cohere Streaming Errors Guide
• Cohere API Monitoring Guide