Cohere API Connection Reset: The Write-Side Failure Nobody Looks For

Cohere is not one API surface, it is three with very different traffic shapes. Chat sends a small body and receives a large one. Embed sends a large body — a batch of documents, often near the maximum — and receives a compact array of vectors. Rerank sends a query plus an entire candidate set. A connection reset on chat and a connection reset on embed have almost nothing in common.

That matters because a reset that occurs during the request upload behaves differently from one that occurs while waiting for a response. It fails almost immediately, it scales with body size rather than with idle time, and depending on the client and the timing you may see it as EPIPE or “broken pipe” rather than ECONNRESET at all. Teams grep their logs for ECONNRESET, miss the EPIPE entries entirely, and conclude the problem is intermittent when it is in fact perfectly deterministic in document count.

Read the failure by surface, not by status

The single most common misdiagnosis here is treating a RAG pipeline as one thing. A retrieval pipeline calls embed to index, embed again to encode the query, rerank to order candidates and chat to generate — four separate surfaces, each with its own serving path, collapsed by your error handling into one “Cohere failed” log line.

Split them before anything else. If only embed resets, you are looking at a large-body problem and the variable is document count. If only chat resets, you are looking at the ordinary stale-socket or long-generation causes. If rerank resets, it is the same large-body problem as embed, since the candidate set travels in the request.

The clean signal is that embed and rerank failures track batch size while chat failures track idle time. Any explanation that does not account for that split is the wrong explanation.

Why large bodies get reset mid-upload

When you POST a large body, the bytes go out over several round trips. If any intermediary in the path decides it will not accept the request — a body-size limit, a rejected header, an authentication failure evaluated early, a memory limit on a proxy — it can close the connection while you are still writing. Your client is mid-send when the RST arrives, which is why it surfaces as a broken pipe on some stacks and a connection reset on others.

The distinguishing characteristic is determinism. A stale-socket reset is random: the same request succeeds and fails depending on how long the connection happened to sit idle. A write-side reset is reproducible at a threshold — 96 documents fails and 48 succeeds, every time, from every machine. If you can reproduce it on demand by growing the batch, stop looking at your connection pool.

There is a second, subtler version worth knowing: unusually long individual documents can push a batch past a limit even when the document count looks modest. Track total payload bytes, not just array length, because a batch of 40 long documents can be larger than a batch of 90 short ones.

Chunk by bytes, with bounded concurrency

The fix for a write-side reset is to make the body smaller, and the right unit is bytes rather than array length. Chunk on a payload budget, keep concurrency bounded so you do not simply move the failure into the pool, and retry per chunk so one bad batch does not fail an entire indexing run.

// Chunk by payload bytes, not by document count — long docs blow a small array.
const MAX_BYTES = 400_000;   // stay comfortably under any intermediary body limit

function chunkByBytes(texts: string[]) {
  const out: string[][] = [];
  let cur: string[] = [], size = 0;
  for (const t of texts) {
    const b = Buffer.byteLength(t, 'utf8');
    if (cur.length && size + b > MAX_BYTES) { out.push(cur); cur = []; size = 0; }
    cur.push(t); size += b;
  }
  if (cur.length) out.push(cur);
  return out;
}

export async function embedAll(texts: string[]) {
  const batches = chunkByBytes(texts);
  const vectors: number[][] = [];
  for (const batch of batches) {          // bounded concurrency: sequential is fine for indexing
    const res = await fetch('https://api.cohere.com/v2/embed', {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.COHERE_API_KEY}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ model: 'embed-v4.0', input_type: 'search_document', texts: batch }),
    });
    if (!res.ok) throw new Error(`embed ${res.status}`);
    vectors.push(...(await res.json()).embeddings.float);
  }
  return vectors;
}

Then handle the failure itself. The rule that keeps retries honest is the same everywhere: retry only when nothing was read back, cap the attempts, and always add jitter — a reset that hits many sockets at once will otherwise synchronise every client into a single reconnect wave that re-fails together.

// Log the surface and the payload size, or the two causes stay indistinguishable.
type Surface = 'chat' | 'embed' | 'rerank';

export async function callCohere(surface: Surface, url: string, body: object) {
  const payload = JSON.stringify(body);
  const t0 = Date.now();
  try {
    const res = await fetch(url, {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.COHERE_API_KEY}`, 'Content-Type': 'application/json' },
      body: payload,
    });
    if (!res.ok) throw new Error(`upstream ${res.status}`);
    return res;
  } catch (err) {
    const cause = String((err as Error).cause ?? err);
    console.error('cohere_conn_error', {
      surface,
      bytes: Buffer.byteLength(payload, 'utf8'),
      ms: Date.now() - t0,
      writeSide: cause.includes('EPIPE'),   // grep for BOTH — EPIPE is the one teams miss
      cause,
    });
    throw err;
  }
}

The chat-side reset, which is the ordinary one

On the chat surface, Cohere behaves like every other provider and the causes are the familiar ones. A pooled keep-alive socket that the edge closed on its own schedule produces an immediate RST when you write to it; the failure is random rather than deterministic and it clusters after quiet periods and serverless warm starts.

Fix it the usual way. Set the client’s idle timeout strictly below the server’s, cap the maximum lifetime of any connection so long-lived sockets churn, and size the pool so a burst does not force reuse of marginal connections. Four seconds idle with a ten second ceiling is a reasonable default.

Do this even if embed is your real problem, because it removes the random component and leaves the deterministic, batch-size-scaled failures standing out clearly in the logs.

Frequently Asked Questions

Related Cohere Guides

• Is Cohere Down? Outage Checking Guide
• Cohere API Timeout Error Explained
• Cohere API Error Codes Reference
• Cohere API Batch Processing Guide
• Why Your Cohere API Stream Breaks
• Cohere API Retry Budget Guide
• Cohere API Monitoring Guide