Mistral API Response Caching

Every caching layer in front of an LLM API begins as four lines of code around a dictionary and ends as a correctness surface nobody owns. The four lines are not the problem. The problem is that a cache hit and a correct answer are different things, and the gap between them is invisible from the outside: a stale response arrives with a 200, in the right shape, in less time than a fresh one, and it is only wrong in ways your monitoring was never built to see.

The cost case for caching Mistral is the ordinary one and it holds. The reason to design it deliberately rather than reach for a default is that two Mistral-specific properties both attack the cache key: aliases that resolve to different weights over time, and a deployment story where the same model name can be served from arrangements with different obligations attached.

Three Different Caches, Routinely Confused for One

The word cache covers at least three mechanisms in an LLM stack, and they save different things, fail differently, and are owned by different people. Conflating them is why so many caching discussions end with two participants agreeing while describing incompatible systems.

Provider-side prompt caching is a server feature. Where Mistral offers something in this family, it reduces the cost or latency of re-processing a repeated prefix — typically a long system prompt or a fixed document — while still generating fresh output. It does not return a previous answer, and it therefore carries no staleness risk at all. It is also not under your control, which means you cannot reason about it during an incident.

Your exact-match response cache stores completions keyed by a hash of the request. It is the only one of the three that eliminates the API call entirely, so it is the only one that helps with rate limits and the only one that does anything during an outage. It is also the only one that can serve a wrong answer, and every remaining section of this guide is about that.

Semantic caching replaces hash equality with vector similarity, so a question close enough to a previous one returns that previous answer. It raises hit rate on natural-language traffic and it introduces a class of failure the other two do not have: a confident, well-formed response to a question nobody asked.

The Cache Key Is the Whole Request

Nearly every stale-hit bug traces back to a key that captured less than the request did. The prompt is the obvious input and it is rarely the only one that changes the output. A key that omits any of the following will, sooner or later, serve a response your current code could not have produced:

• The model identifier — and where possible the resolved one, not an alias you sent.
• The full system prompt, hashed by content, plus a revision number you control.
• Sampling parameters that affect output: temperature, top-p, seed, stop sequences.
• The maximum token budget, which changes where an answer is truncated.
• The response format or schema, and the tool definitions if any are attached.
• Any retrieval or filter parameters that change what the model was given to work with.
• A key-schema version you bump by hand when the semantics change but the bytes do not.

The simplest implementation that satisfies all of this is to hash the serialised request body you are about to send, with keys sorted so that ordering noise does not fragment the cache, and to prefix that hash with your schema version. This has the useful property of being correct by construction: anything that changes the request changes the key, including the parameter someone adds next quarter without reading this page.

The manual version prefix earns its place in one specific situation. When you change what a stored response means without changing any bytes on the wire — a downstream parser that now expects different fields, a policy change about what an answer may contain — the automatic hash cannot know. Bumping the prefix invalidates everything at once, which is exactly what you want, and costs one cold period.

Time-to-Live Is a Correctness Setting

Time-to-live gets tuned as a cost dial: raise it, save more. That framing is backwards. The right ceiling is set by how quickly the correct answer changes, and cost is whatever falls out of that constraint. Three rough classes cover most traffic.

Evergreen work — classification of static text, extraction from a fixed document, translation of unchanging strings — tolerates days or longer, and the practical limit becomes your own prompt-revision cadence rather than the content.

Application-coupled work reads from data your own system also writes to. Here the ceiling is minutes, because the failure mode is not a slightly dated answer, it is the cache contradicting your database while both behave as designed. Users report that as a bug, and they are right.

Volatile work depends on something changing continuously outside your control. Keep the window to seconds or low minutes and accept that the cache is now burst protection rather than a saving. A low hit rate here is an accurate reflection of the workload, not a tuning failure.

Two mechanisms are worth adding regardless of class. Event-driven invalidation — deleting keys when the underlying record changes — is strictly better than waiting for expiry and is usually a few lines wherever you already publish change events. And a small random jitter on each entry prevents a population of keys written during one traffic spike from expiring together and stampeding Mistral at the same moment, which is a self-inflicted outage that looks exactly like a provider one.

Semantic Caching and the False Hit

Exact matching has a hard ceiling: on free-text traffic, users phrase the same intent in unboundedly many ways and almost nothing repeats byte-for-byte. Semantic caching lifts that ceiling by embedding the incoming request and returning the stored answer of any sufficiently similar previous one.

The cost is a failure mode with no error attached to it. Above the threshold, two questions are treated as one; below it, they are not. There is no setting at which the classifier is right, only settings that trade false hits against hit rate. And the false hit is unusually convincing here, because the answer you serve is a real, fluent, on-topic answer — to the neighbouring question.

If you deploy it, deploy it narrowly. Scope entries to one model and one call site rather than sharing a global space. Exclude anything containing an identifier, a number or a date, since those are exactly the tokens that similarity treats as noise and users treat as the entire question. Never use it on a call whose output triggers an action rather than being displayed. And log the matched source question alongside every semantic hit — without that field, false hits are effectively undebuggable, because the only evidence is a user saying the answer felt off.

Caching During a Mistral Outage

A response cache is the cheapest partial mitigation available for an upstream incident, and it is important to be precise about how partial. It covers requests you have already served and returns nothing for anything new. If your steady-state hit rate is forty percent, a serve-stale-on-error policy converts roughly forty percent of a hard Mistral outage into a degraded-but-working experience and leaves the remainder failing exactly as before.

That is worth having, and it is not failover. Treat the cache as the first tier of a two-tier policy: serve stale on error where an entry exists, route to a second provider where one does not, and fail loudly only when both are unavailable. The ordering matters, because a stale answer is usually better than a substituted-model answer for a user, and both beat an error page.

The dangerous part is what serve-stale does to your telemetry. Requests that would have been errors now return 200. Your error rate stays flat. Your latency improves, because cache hits are fast. Every dashboard you own reports an unusually healthy afternoon while a growing share of users receive answers computed hours ago. Emit stale-served as its own counter, alert on its rate rather than its total, and show an as-of timestamp in the interface wherever a user could act on the content.

What to Log

A cache with one hit-rate gauge is a cache you cannot debug. The minimum useful set is small and every field earns its place during an incident rather than in a weekly report:

• Outcome as three counters — hit, miss, stale-served — never one ratio.
• Key schema version, so a deploy that silently invalidates everything is visible as a step change rather than a mystery cost spike.
• Model requested and model served, which is what turns an upstream repoint from a support ticket into an alert.
• Entry age at serve time, so you can see the staleness distribution instead of assuming your time-to-live describes it.
• Tokens and cost avoided, which is the only number that justifies the layer to anyone outside the team.
• Matched source question for semantic hits, without which false hits cannot be investigated at all.

Hit rate belongs per call site rather than globally. The aggregate is dominated by whichever endpoint is busiest and it hides the one path that is quietly pressing against a Mistral rate limit — which is usually the path you built the cache for in the first place.

The Mistral Trap: The Alias Moves and the Key Does Not

Pointing at a latest-style alias is a subscription to change. That is a reasonable choice on its own, and it is incompatible with a cache keyed on the string you sent. The alias repoints, the identifier in your request body is byte-identical, the key matches, and you serve responses generated by the previous weights for as long as your time-to-live allows. Nothing errors. Nothing in your logs distinguishes it from a normal hit.

The fix is to key on what was served rather than what was requested. If the response carries a resolved model identifier, put that in the key and log requested-versus-served on every miss; a divergence between the two is your cache-invalidation trigger and it costs nothing to watch. If no resolved identifier is available, pin the version explicitly in the request and treat the alias as something you read in release notes rather than something you send.

The second Mistral property is jurisdictional. A response cache stores prompts and completions, which means it holds exactly the content the residency decision was made about, in whatever region your cache happens to run. A managed cache in a convenient region can undo a deliberate provider choice with no code review flagging it, because to the reviewer the change reads as infrastructure rather than data handling. Region-pin the cache alongside the API, apply the same retention schedule, and make cache eviction part of any deletion cascade you have promised.

Is It the Cache, or Is It Mistral?

Once a cache is in the request path, every ambiguous production symptom acquires a second suspect. Answers that feel dated could be a time-to-live set too high or a model repointed upstream. A sudden cost increase could be a key-schema change that invalidated everything, or a Mistral pricing change, or a traffic-mix shift toward the long tail. A latency improvement during an incident could be a healthy cache doing its job, or serve-stale quietly absorbing an outage nobody has noticed yet.

None of those are distinguishable from application logs alone, because in five of the six failure rows above the wire looks perfectly normal. Resolving them requires an independent signal: something outside your application making real requests to api.mistral.ai on a fixed interval and recording error rate and latency regardless of what your own traffic is doing. With that series in hand, the question takes seconds. Without it, the first half hour of every incident is spent reading a cache implementation that never changed.

Frequently Asked Questions

Related Guides

• Mistral API Pricing Explained
• Mistral API Rate Limits Explained
• Mistral API Latency Guide
• Mistral API Cost Routing Guide
• Mistral API Monitoring Guide
• Mistral API Failover Guide
• Is Mistral Down? Outage Checking Guide
• Live Mistral Status Check