Cohere API Response Caching: What to Cache, What Never to Cache, and How It Behaves During an Outage (2026)

Every caching layer in front of an LLM API begins as four lines of code around a dictionary and ends as a correctness surface nobody owns. The four lines are not the problem. The problem is that a cache hit and a correct answer are different things, and the gap between them is invisible from the outside: a stale response arrives with a 200, in the right shape, in less time than a fresh one, and it is only wrong in ways your monitoring was never built to see.

Embeddings are the outlier in this whole discussion. For a fixed model, input and input type, the vector is reproducible, which removes the usual objection that a cached generative response is only approximately the response you would get today. Re-embedding a corpus you have already embedded is pure waste, and it is waste that scales with your data rather than your traffic.

The short version

Key on the entire outbound request, not the prompt. Set time-to-live from how fast the correct answer changes, never from how much you want to save. Prefer exact matching until you have measured its ceiling, because a semantic hit is a substitution you did not authorise. Emit hit, miss and stale-served as three separate counters, and alert on the third one — during a Cohere incident it is often the earliest signal you have.

Three Different Caches, Routinely Confused for One

The word cache covers at least three mechanisms in an LLM stack, and they save different things, fail differently, and are owned by different people. Conflating them is why so many caching discussions end with two participants agreeing while describing incompatible systems.

Provider-side prompt caching is a server feature. Where Cohere offers something in this family, it reduces the cost or latency of re-processing a repeated prefix — typically a long system prompt or a fixed document — while still generating fresh output. It does not return a previous answer, and it therefore carries no staleness risk at all. It is also not under your control, which means you cannot reason about it during an incident.

Your exact-match response cache stores completions keyed by a hash of the request. It is the only one of the three that eliminates the API call entirely, so it is the only one that helps with rate limits and the only one that does anything during an outage. It is also the only one that can serve a wrong answer, and every remaining section of this guide is about that.

Semantic caching replaces hash equality with vector similarity, so a question close enough to a previous one returns that previous answer. It raises hit rate on natural-language traffic and it introduces a class of failure the other two do not have: a confident, well-formed response to a question nobody asked.

Decide which problem you have first

If the pain is cost, all three help. If the pain is rate limits or Cohere availability, only your own response cache helps, because it is the only one that removes the request. Teams that skip this step enable a provider feature, watch their limiter keep firing, and conclude that caching does not work.

The Cache Key Is the Whole Request

Nearly every stale-hit bug traces back to a key that captured less than the request did. The prompt is the obvious input and it is rarely the only one that changes the output. A key that omits any of the following will, sooner or later, serve a response your current code could not have produced:

The model identifier — and where possible the resolved one, not an alias you sent.
The full system prompt, hashed by content, plus a revision number you control.
Sampling parameters that affect output: temperature, top-p, seed, stop sequences.
The maximum token budget, which changes where an answer is truncated.
The response format or schema, and the tool definitions if any are attached.
Any retrieval or filter parameters that change what the model was given to work with.
A key-schema version you bump by hand when the semantics change but the bytes do not.

The simplest implementation that satisfies all of this is to hash the serialised request body you are about to send, with keys sorted so that ordering noise does not fragment the cache, and to prefix that hash with your schema version. This has the useful property of being correct by construction: anything that changes the request changes the key, including the parameter someone adds next quarter without reading this page.

The manual version prefix earns its place in one specific situation. When you change what a stored response means without changing any bytes on the wire — a downstream parser that now expects different fields, a policy change about what an answer may contain — the automatic hash cannot know. Bumping the prefix invalidates everything at once, which is exactly what you want, and costs one cold period.

Time-to-Live Is a Correctness Setting

Time-to-live gets tuned as a cost dial: raise it, save more. That framing is backwards. The right ceiling is set by how quickly the correct answer changes, and cost is whatever falls out of that constraint. Three rough classes cover most traffic.

Evergreen work — classification of static text, extraction from a fixed document, translation of unchanging strings — tolerates days or longer, and the practical limit becomes your own prompt-revision cadence rather than the content.

Application-coupled work reads from data your own system also writes to. Here the ceiling is minutes, because the failure mode is not a slightly dated answer, it is the cache contradicting your database while both behave as designed. Users report that as a bug, and they are right.

Volatile work depends on something changing continuously outside your control. Keep the window to seconds or low minutes and accept that the cache is now burst protection rather than a saving. A low hit rate here is an accurate reflection of the workload, not a tuning failure.

Two mechanisms are worth adding regardless of class. Event-driven invalidation — deleting keys when the underlying record changes — is strictly better than waiting for expiry and is usually a few lines wherever you already publish change events. And a small random jitter on each entry prevents a population of keys written during one traffic spike from expiring together and stampeding Cohere at the same moment, which is a self-inflicted outage that looks exactly like a provider one.

Semantic Caching and the False Hit

Exact matching has a hard ceiling: on free-text traffic, users phrase the same intent in unboundedly many ways and almost nothing repeats byte-for-byte. Semantic caching lifts that ceiling by embedding the incoming request and returning the stored answer of any sufficiently similar previous one.

The cost is a failure mode with no error attached to it. Above the threshold, two questions are treated as one; below it, they are not. There is no setting at which the classifier is right, only settings that trade false hits against hit rate. And the false hit is unusually convincing here, because the answer you serve is a real, fluent, on-topic answer — to the neighbouring question.

If you deploy it, deploy it narrowly. Scope entries to one model and one call site rather than sharing a global space. Exclude anything containing an identifier, a number or a date, since those are exactly the tokens that similarity treats as noise and users treat as the entire question. Never use it on a call whose output triggers an action rather than being displayed. And log the matched source question alongside every semantic hit — without that field, false hits are effectively undebuggable, because the only evidence is a user saying the answer felt off.

Caching During a Cohere Outage

A response cache is the cheapest partial mitigation available for an upstream incident, and it is important to be precise about how partial. It covers requests you have already served and returns nothing for anything new. If your steady-state hit rate is forty percent, a serve-stale-on-error policy converts roughly forty percent of a hard Cohere outage into a degraded-but-working experience and leaves the remainder failing exactly as before.

That is worth having, and it is not failover. Treat the cache as the first tier of a two-tier policy: serve stale on error where an entry exists, route to a second provider where one does not, and fail loudly only when both are unavailable. The ordering matters, because a stale answer is usually better than a substituted-model answer for a user, and both beat an error page.

The dangerous part is what serve-stale does to your telemetry. Requests that would have been errors now return 200. Your error rate stays flat. Your latency improves, because cache hits are fast. Every dashboard you own reports an unusually healthy afternoon while a growing share of users receive answers computed hours ago. Emit stale-served as its own counter, alert on its rate rather than its total, and show an as-of timestamp in the interface wherever a user could act on the content.

Six ways a cache fails, and what your monitoring sees

What happened	What your app sees	What the user gets
Prompt text changed by one character	Miss, correctly	None — this is the cache working
System prompt edited, key unchanged	HIT (200)	Users receive answers from the previous instruction set
Model identifier repointed upstream	HIT (200)	Output attributed to a model that no longer serves it
Temperature or max tokens changed	HIT (200)	Sampling settings in code have no effect in production
Upstream returning 5xx, serve-stale enabled	HIT (200)	Incident is invisible in your error rate
Underlying data source updated	HIT (200)	Cache contradicts your own database, and wins

Five of the six rows return a clean 200. A cache does not fail by erroring; it fails by succeeding with the wrong content, which is why cache correctness has to be instrumented deliberately rather than inferred from an error rate.

What to Log

A cache with one hit-rate gauge is a cache you cannot debug. The minimum useful set is small and every field earns its place during an incident rather than in a weekly report:

Outcome as three counters — hit, miss, stale-served — never one ratio.
Key schema version, so a deploy that silently invalidates everything is visible as a step change rather than a mystery cost spike.
Model requested and model served, which is what turns an upstream repoint from a support ticket into an alert.
Entry age at serve time, so you can see the staleness distribution instead of assuming your time-to-live describes it.
Tokens and cost avoided, which is the only number that justifies the layer to anyone outside the team.
Matched source question for semantic hits, without which false hits cannot be investigated at all.

Hit rate belongs per call site rather than globally. The aggregate is dominated by whichever endpoint is busiest and it hides the one path that is quietly pressing against a Cohere rate limit — which is usually the path you built the cache for in the first place.

The Cohere Trap: An Embedding Cache That Outlives Its Model Poisons the Index

The property that makes embeddings worth caching is the same property that makes a stale embedding cache dangerous. Vectors from two different models are not comparable — they are coordinates in different spaces — so a cache that survives a model change starts feeding old-space vectors into a new-space index. Nothing raises. Similarity scores stay in their usual numeric range. Retrieval simply gets worse, in a way that reads as a prompt problem or a chunking problem for as long as you are willing to believe it.

Put the embedding model identifier and the input type in the key, not merely the text, and treat a model change as a full re-embed of the corpus rather than a cache flush you can do lazily. A half-migrated index is worse than either end state, because the mixed distances are confidently wrong rather than uniformly degraded.

Rerank sits in a similar position and is often overlooked. Rerank calls repeat heavily — the same query against the same candidate set recurs across pagination, retries and refreshes — and the output is a stable ordering rather than free text, so caching it carries little correctness risk provided the candidate set is part of the key. Hash the ordered document identifiers, not just the query, or you will serve an ordering computed for a set of documents that has since changed.

Is It the Cache, or Is It Cohere?

Once a cache is in the request path, every ambiguous production symptom acquires a second suspect. Answers that feel dated could be a time-to-live set too high or a model repointed upstream. A sudden cost increase could be a key-schema change that invalidated everything, or a Cohere pricing change, or a traffic-mix shift toward the long tail. A latency improvement during an incident could be a healthy cache doing its job, or serve-stale quietly absorbing an outage nobody has noticed yet.

None of those are distinguishable from application logs alone, because in five of the six failure rows above the wire looks perfectly normal. Resolving them requires an independent signal: something outside your application making real requests to api.cohere.com on a fixed interval and recording error rate and latency regardless of what your own traffic is doing. With that series in hand, the question takes seconds. Without it, the first half hour of every incident is spent reading a cache implementation that never changed.

Frequently Asked Questions

What should I cache first on a Cohere stack?

Embeddings, by a wide margin. For a fixed model, input and input type the vector is reproducible, so the usual objection to caching generative output does not apply, and the waste you eliminate scales with corpus size rather than traffic. Rerank is the second target: the calls repeat heavily across pagination and retries, and the output is an ordering rather than prose. Chat completions come third.

Why did retrieval quality drop after I changed embedding models?

Very likely a cache or an index holding vectors from both models at once. Embeddings from different models are coordinates in different spaces and are not comparable, so mixed distances are meaningless while remaining perfectly well-formed numbers. Nothing errors and no score looks anomalous. Put the model identifier in the embedding cache key and treat a model change as a full re-embed, because a half-migrated index is worse than either end state.

Does input type belong in the embedding cache key?

Yes. Where an API distinguishes between embedding a document and embedding a search query, the same text produces different vectors by design, and that distinction is the whole point of the parameter. A key built on text alone will hand you a document vector for a query, retrieval will degrade slightly and consistently, and the cause will not appear anywhere in your logs.

How do I cache rerank results safely?

Key on the query plus a hash of the ordered candidate document identifiers plus the rerank model. The candidate set is the part teams omit, and omitting it means serving an ordering computed for documents that may since have changed. With the full key in place the risk is low and the saving is high, because rerank is called repeatedly against identical inputs during pagination and retries.

Does a cache protect me if the Cohere API goes down?

For embeddings of previously seen text and for repeated rerank calls, substantially — those are exactly the requests a cache covers well. For chat completions and for any first-time input, not at all. A retrieval pipeline that can serve cached document vectors but cannot embed a new user query is still broken from the user side, so the query path needs a fallback provider or a degraded lexical search behind it, not just a cache.

Related Guides

A Serve-Stale Policy Hides the Outage It Survives

When your cache absorbs a Cohere incident, your error rate stays flat and your latency improves. API Status Check probes api.cohere.com independently of your traffic and alerts on errors and latency, so the incident shows up on a dashboard instead of in a support queue.

Start Your Free Trial →

Current API Status