Cohere API Timeout Error: Why Requests Hang

Cohere timeouts usually get blamed on chat, and usually are not chat. The endpoints that hang in production are embed and rerank, because those are the ones people hand enormous batches to.

A timeout is not a response. Nothing came back, so nothing tells you whether the request reached Cohere, whether it was processed, or whether something in between gave up first. Every other error hands you a status code to search for; this one hands you silence. That is why teams lose hours to it — the debugging instinct is to check the provider's status page, which is the layer least likely to be responsible.

Four layers can kill the request, and only one is Cohere

Before changing any value, find out who enforced the limit. The shortest timeout in the chain wins, and it is almost never the one you configured — it is the one you inherited from a framework default or a hosting platform.

The round numbers in that second row are the most useful clue on this page. A request that dies at almost exactly 10, 29, 30 or 60 seconds was killed by infrastructure, not by a model. If your chat calls are fine and only embed or rerank hangs, stop looking at Cohere's status page — the variable that changed is batch size, not availability.

Prove it with curl before you change code

Take your application entirely out of the picture. This runs the same request with a long ceiling and prints a full timing breakdown, so you can see where the seconds actually went:

curl -sS -o /dev/null --max-time 120 \
  -w "dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s firstbyte=%{time_starttransfer}s total=%{time_total}s\n" \
  https://api.cohere.com/v2/chat \
  -H "Authorization: Bearer $CO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"command-r-plus","messages":[{"role":"user","content":"ping"}]}'

Read it from the left. A large connect or tls figure is a network or handshake problem and has nothing to do with the model. A small connect with a large firstbyte means the connection was established instantly and Cohere spent the time thinking — that is real processing latency, and the answer is a longer budget or a smaller request, not a retry. If the whole thing completes well inside your application's configured timeout, the request never had a chance and your own limit is the bug.

Why embed and rerank time out before chat does

Cohere's value in most stacks is retrieval infrastructure: embedding a corpus and reranking candidates. Both endpoints accept batches, and batch size is the variable teams tune upward when a job is slow — which is exactly backwards from the perspective of a timeout. A single embed call carrying a large batch of long documents is one HTTP request that must complete entirely before a single byte comes back, and unlike chat there is no streaming to keep the connection alive.

Rerank has the same shape with a sharper edge. Handing it hundreds of long candidate documents means every one of them is scored against the query before the response exists. That is legitimate work, it scales with the batch, and it happens inside your timeout window with total silence on the wire. A rerank call that works fine against 25 candidates in testing can hang against 500 in production against the same timeout value.

The fix is almost never a longer timeout. It is smaller batches, run concurrently. Splitting a large embed job into chunks and issuing them in parallel with a bounded concurrency limit finishes faster in wall-clock terms than one giant request, fails partially instead of totally, retries cheaply, and stays comfortably inside any platform duration cap. The one thing to respect is the rate limit — unbounded parallelism converts a timeout problem into a 429 problem.

Set two timeouts, not one

A single overall deadline is the wrong shape for this problem. Connecting should be fast and failing to connect should fail immediately; generating can legitimately be slow. Collapsing both into one number means either network failures hang for the full budget, or slow-but-healthy requests get killed. Split them:

// Short connect budget, generous read budget.
async function callCohere(body: unknown) {
  const ac = new AbortController();
  const connectGuard = setTimeout(() => ac.abort(new Error('connect-timeout')), 5_000);
  const readGuard = setTimeout(() => ac.abort(new Error('read-timeout')), 60_000);

  try {
    const res = await fetch('https://api.cohere.com/v2/chat', {
      method: 'POST',
      signal: ac.signal,
      headers: {
        Authorization: `Bearer ${process.env.CO_API_KEY}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(body),
    });
    clearTimeout(connectGuard); // headers arrived — connection is alive
    if (!res.ok) throw new Error(`upstream ${res.status}`);
    return await res.json();
  } finally {
    clearTimeout(connectGuard);
    clearTimeout(readGuard);
  }
}

Then make sure the platform agrees with you. A 60-second client timeout inside a function capped at 10 seconds is decoration — the platform kills the invocation first and your carefully chosen value never applies. Raise the function's duration limit to sit above the client timeout, or move the call out of the request path entirely.

Streaming turns a total failure into a partial one

The single highest-leverage change is to stop waiting for the whole response. With stream: true the first bytes arrive quickly, which keeps every idle timer in the chain — proxies, load balancers, serverless runtimes — from concluding the connection is dead. Those idle timers are what kill most non-streaming calls long before the total duration cap is reached.

It also changes what failure costs. A buffered request that dies at the deadline returns nothing at all, and you pay for tokens the user never saw. A stream that breaks after several hundred tokens has already delivered something useful and gives you a point to resume from.

With streaming, replace the total deadline with an idle deadline that resets on every chunk. A request that has produced tokens two seconds ago is healthy no matter how long it has been running; one that has produced nothing for thirty seconds is stuck regardless of how recently it started:

// Idle timeout — resets whenever a chunk arrives.
const reader = res.body!.getReader();
let idle = setTimeout(() => reader.cancel('stalled'), 30_000);

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  clearTimeout(idle);
  idle = setTimeout(() => reader.cancel('stalled'), 30_000);
  yield value;
}
clearTimeout(idle);

Retries that help, and retries that make it worse

Timeouts are the most dangerous class of error to retry blindly, because you do not know what happened. The request may have failed instantly, or it may have been fully processed with the response lost on the way back — and retrying the second case pays for the work twice while adding load to a system that was already struggling.

• Retry on zero bytes received. Nothing was generated, so nothing is duplicated. This is the safe case.
• Do not blindly retry a mid-stream failure. Tokens were produced and billed; resume or degrade instead.
• Always use jitter. Fixed backoff synchronises every client into waves that arrive together and re-fail together.
• Cap attempts at two or three, then fail loudly. Unbounded retries turn a provider blip into a self-inflicted outage.
• Put a circuit breaker in front. When failures cross a threshold, stop calling for a cooling period — see the Cohere failover guide.

And measure the right number. Averages hide timeouts completely: a p50 of 900 milliseconds tells you nothing about the 1% of requests sitting at your ceiling. Alert on p99 and on the timeout rate itself, because that tail is the entire population of users experiencing the bug.

Frequently Asked Questions

Related Cohere Guides

• Is Cohere Down? Outage Checking Guide
• Cohere API Latency Guide
• Cohere API Error Codes Reference
• Why Your Cohere API Stream Breaks
• Cohere API Rate Limits Explained
• Cohere API Monitoring Guide
• Cohere API CORS Error Explained