Cohere API 504 Gateway Timeout

A 504 is a statement about a clock, not about a failure. Some intermediary decided in advance how long it was willing to wait, the answer did not arrive inside that window, and the intermediary wrote an error of its own. Nothing in the status code tells you that Cohere failed. It tells you that a deadline expired, and deadlines are configuration — which means the fault is findable by reading a number rather than by guessing.

Whose clock is the whole question. It might be Cohere's edge. It might just as easily be your ingress controller, a service-mesh sidecar, a corporate egress proxy, a platform request limit, or the timeout argument you passed to your own SDK three releases ago and have not looked at since. These produce an identical status code and entirely different remediation.

504 against the rest of the 5xx family

These five get filed together as “the API is broken” and then handed one retry policy, which guarantees the policy is wrong for at least three of them. The last column is the one that matters, because it is the only one that changes what you do next.

502 and 504 are the two gateway-authored codes and they are routinely conflated. The difference is simple and it changes your diagnosis: a 502 means the gateway got an answer it could not use, so the evidence is in the response body; a 504 means it got nothing and stopped waiting, so the evidence is in the elapsed time. The unusable-reply case is covered in theCohere 502 guide, the capacity case in theCohere 503 guide, and throttling in theCohere 429 guide.

Why Cohere returns 504 specifically: batch size is the independent variable

Cohere's production footprint is weighted differently from a pure chat provider. The volume lives in embed and rerank — the retrieval-augmented-generation plumbing — and those endpoints take batches. A batch is not one request that happens to be large; it is N units of work billed and processed as one call, and its latency scales with N in a way that a chat completion's simply does not.

That gives Cohere 504s a property almost no other provider's have: they are usually deterministic. The same batch that timed out will time out again. A batch half the size will very often succeed. This is enormously good news diagnostically, because a failure you can reproduce on demand is one you can bisect, and bisecting resolves in two requests what log archaeology would not resolve in an hour.

It also means the usual incident reflex is wrong here. Retrying an oversized batch unchanged simply reproduces the timeout while billing you for the attempt, and failing over to another provider re-sends the same oversized payload down a different pipe with the same result. The correct response to a size-dependent 504 is to change the size, which is a code change rather than an incident response.

Sample rather than probe once, and record all three columns. One request cannot tell a total failure apart from the far more common partial one:

for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s %{content_type}\n" \
    --max-time 90 \
    https://api.cohere.com/v2/chat \
    -H "Authorization: Bearer $COHERE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"command-r-plus","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'
done

The status code says whether it failed, the elapsed time says which timer expired, and the content type says who wrote the error. Set --max-time deliberately higher than the deadline you are investigating, otherwise curl becomes the shortest timer in the path and you measure your own probe instead of the system.

The Cohere trap: a deterministic size threshold is often your own egress limit

Run the bisect before anything else. Resend the failing batch at one tenth the size. If it succeeds, the failure is size-dependent and you have converted an outage investigation into a chunking task. If it fails identically, size is not the variable and you should treat it as a genuine availability question.

When the bisect does find a threshold, look carefully at where the threshold sits before blaming Cohere. Embed batches are frequently megabytes of JSON, and a clean cutoff at a round payload size — one megabyte, ten megabytes — is the fingerprint of a request-body limit in your own path: an ingress controller, an API gateway, a corporate egress proxy. Those write their own 504s and 413s and never mention the provider.

Then chunk deliberately rather than empirically. Pick a batch size comfortably below the threshold you measured, make the chunking a property of your client rather than a value someone tuned during an incident, and record the batch size on every error so the next occurrence is diagnosable from the log line alone.

The streaming case, where the 504 stops happening at all

Streaming is the most underrated fix for 504s, and it works for a reason that is easy to miss: most gateway timeouts are idle timeouts rather than total-duration limits. They fire when no bytes have moved for N seconds, not when the request has been open for N seconds. A streamed response puts bytes on the wire as soon as the first tokens exist, which resets the idle timer continuously and keeps a long generation alive under a deadline that would have killed the buffered equivalent.

• Check which kind of timeout you have. An idle timeout is defeated by streaming; a hard request-duration cap is not. They are configured separately and confusing them wastes a day.
• Assert on the terminal event. An iterator finishing is not the same as a stream completing. If the terminal marker never arrived, the response was truncated no matter what the status code said.
• Count partial responses as their own class. Not a success, not a 5xx. On a streamed path they are the metric that moves first, and a status-code-only dashboard stays flat through the entire incident.
• Decide the resume policy in advance. Re-issuing the whole prompt is the honest default and it bills twice; showing the partial output with a visible label is often the better product answer.

A retry policy that fits a 504 specifically

504 is the 5xx where the standard retry advice is most likely to hurt you, because the one thing a 504 does not tell you is whether the work happened. Four rules keep the retry from becoming its own incident:

• Assume the request may have succeeded. Your gateway stopped listening; the upstream did not necessarily stop working. Send an idempotency key for anything with a side effect, and expect a retried completion to be billable twice.
• Do not retry into the same deadline. If the timeout was too short for this request, a second identical attempt will expire identically. Change something — stream it, shrink it, or route it to a path with a longer deadline — or do not retry at all.
• Budget by ratio, not by count. Allow retries as a fraction of recent successes so the allowance collapses on its own when almost everything is failing, with no deploy required. See the Cohere retry budget guide.
• Retry against a deadline, not a counter. If the user-facing request gave up eight seconds ago, attempt three is load on behalf of nobody. Propagate the deadline and skip attempts that cannot finish inside it.

If 504s persist past a couple of attempts they have stopped being transient, and the failover machinery should take over — which only works if it was kept warm. TheCohere failover guidecovers keeping a second path tested rather than merely configured, and thecircuit breaker guidecovers stopping the retries automatically.

Frequently Asked Questions

Related Cohere Guides

• Cohere API 502 Bad Gateway
• Cohere API 503 Service Unavailable
• Cohere API Timeout Errors
• Cohere API 429 Too Many Requests
• Cohere API Latency Guide
• Cohere Streaming Errors Guide
• Cohere API Error Codes Reference
• Cohere API Monitoring Guide