Perplexity API Response Caching

Every caching layer in front of an LLM API begins as four lines of code around a dictionary and ends as a correctness surface nobody owns. The four lines are not the problem. The problem is that a cache hit and a correct answer are different things, and the gap between them is invisible from the outside: a stale response arrives with a 200, in the right shape, in less time than a fresh one, and it is only wrong in ways your monitoring was never built to see.

Perplexity is the one provider in this tier where caching is primarily a correctness decision and only incidentally a cost one. The output is expensive enough that the savings are real, and the content is time-bound enough that an aggressive time-to-live turns your cache into a machine for confidently repeating last week.

Three Different Caches, Routinely Confused for One

The word cache covers at least three mechanisms in an LLM stack, and they save different things, fail differently, and are owned by different people. Conflating them is why so many caching discussions end with two participants agreeing while describing incompatible systems.

Provider-side prompt caching is a server feature. Where Perplexity offers something in this family, it reduces the cost or latency of re-processing a repeated prefix — typically a long system prompt or a fixed document — while still generating fresh output. It does not return a previous answer, and it therefore carries no staleness risk at all. It is also not under your control, which means you cannot reason about it during an incident.

Your exact-match response cache stores completions keyed by a hash of the request. It is the only one of the three that eliminates the API call entirely, so it is the only one that helps with rate limits and the only one that does anything during an outage. It is also the only one that can serve a wrong answer, and every remaining section of this guide is about that.

Semantic caching replaces hash equality with vector similarity, so a question close enough to a previous one returns that previous answer. It raises hit rate on natural-language traffic and it introduces a class of failure the other two do not have: a confident, well-formed response to a question nobody asked.

The Cache Key Is the Whole Request

Nearly every stale-hit bug traces back to a key that captured less than the request did. The prompt is the obvious input and it is rarely the only one that changes the output. A key that omits any of the following will, sooner or later, serve a response your current code could not have produced:

• The model identifier — and where possible the resolved one, not an alias you sent.
• The full system prompt, hashed by content, plus a revision number you control.
• Sampling parameters that affect output: temperature, top-p, seed, stop sequences.
• The maximum token budget, which changes where an answer is truncated.
• The response format or schema, and the tool definitions if any are attached.
• Any retrieval or filter parameters that change what the model was given to work with.
• A key-schema version you bump by hand when the semantics change but the bytes do not.

The simplest implementation that satisfies all of this is to hash the serialised request body you are about to send, with keys sorted so that ordering noise does not fragment the cache, and to prefix that hash with your schema version. This has the useful property of being correct by construction: anything that changes the request changes the key, including the parameter someone adds next quarter without reading this page.

The manual version prefix earns its place in one specific situation. When you change what a stored response means without changing any bytes on the wire — a downstream parser that now expects different fields, a policy change about what an answer may contain — the automatic hash cannot know. Bumping the prefix invalidates everything at once, which is exactly what you want, and costs one cold period.

Time-to-Live Is a Correctness Setting

Time-to-live gets tuned as a cost dial: raise it, save more. That framing is backwards. The right ceiling is set by how quickly the correct answer changes, and cost is whatever falls out of that constraint. Three rough classes cover most traffic.

Evergreen work — classification of static text, extraction from a fixed document, translation of unchanging strings — tolerates days or longer, and the practical limit becomes your own prompt-revision cadence rather than the content.

Application-coupled work reads from data your own system also writes to. Here the ceiling is minutes, because the failure mode is not a slightly dated answer, it is the cache contradicting your database while both behave as designed. Users report that as a bug, and they are right.

Volatile work depends on something changing continuously outside your control. Keep the window to seconds or low minutes and accept that the cache is now burst protection rather than a saving. A low hit rate here is an accurate reflection of the workload, not a tuning failure.

Two mechanisms are worth adding regardless of class. Event-driven invalidation — deleting keys when the underlying record changes — is strictly better than waiting for expiry and is usually a few lines wherever you already publish change events. And a small random jitter on each entry prevents a population of keys written during one traffic spike from expiring together and stampeding Perplexity at the same moment, which is a self-inflicted outage that looks exactly like a provider one.

Semantic Caching and the False Hit

Exact matching has a hard ceiling: on free-text traffic, users phrase the same intent in unboundedly many ways and almost nothing repeats byte-for-byte. Semantic caching lifts that ceiling by embedding the incoming request and returning the stored answer of any sufficiently similar previous one.

The cost is a failure mode with no error attached to it. Above the threshold, two questions are treated as one; below it, they are not. There is no setting at which the classifier is right, only settings that trade false hits against hit rate. And the false hit is unusually convincing here, because the answer you serve is a real, fluent, on-topic answer — to the neighbouring question.

If you deploy it, deploy it narrowly. Scope entries to one model and one call site rather than sharing a global space. Exclude anything containing an identifier, a number or a date, since those are exactly the tokens that similarity treats as noise and users treat as the entire question. Never use it on a call whose output triggers an action rather than being displayed. And log the matched source question alongside every semantic hit — without that field, false hits are effectively undebuggable, because the only evidence is a user saying the answer felt off.

Caching During a Perplexity Outage

A response cache is the cheapest partial mitigation available for an upstream incident, and it is important to be precise about how partial. It covers requests you have already served and returns nothing for anything new. If your steady-state hit rate is forty percent, a serve-stale-on-error policy converts roughly forty percent of a hard Perplexity outage into a degraded-but-working experience and leaves the remainder failing exactly as before.

That is worth having, and it is not failover. Treat the cache as the first tier of a two-tier policy: serve stale on error where an entry exists, route to a second provider where one does not, and fail loudly only when both are unavailable. The ordering matters, because a stale answer is usually better than a substituted-model answer for a user, and both beat an error page.

The dangerous part is what serve-stale does to your telemetry. Requests that would have been errors now return 200. Your error rate stays flat. Your latency improves, because cache hits are fast. Every dashboard you own reports an unusually healthy afternoon while a growing share of users receive answers computed hours ago. Emit stale-served as its own counter, alert on its rate rather than its total, and show an as-of timestamp in the interface wherever a user could act on the content.

What to Log

A cache with one hit-rate gauge is a cache you cannot debug. The minimum useful set is small and every field earns its place during an incident rather than in a weekly report:

• Outcome as three counters — hit, miss, stale-served — never one ratio.
• Key schema version, so a deploy that silently invalidates everything is visible as a step change rather than a mystery cost spike.
• Model requested and model served, which is what turns an upstream repoint from a support ticket into an alert.
• Entry age at serve time, so you can see the staleness distribution instead of assuming your time-to-live describes it.
• Tokens and cost avoided, which is the only number that justifies the layer to anyone outside the team.
• Matched source question for semantic hits, without which false hits cannot be investigated at all.

Hit rate belongs per call site rather than globally. The aggregate is dominated by whichever endpoint is busiest and it hides the one path that is quietly pressing against a Perplexity rate limit — which is usually the path you built the cache for in the first place.

The Perplexity Trap: Staleness Has No Error Code

A grounded answer decays on a schedule set by its subject, not by your configuration. A question about a stable definition is good for weeks. A question about pricing, availability, a version number or anything with a date in it can be wrong within hours, and the cached text will not announce this — it reads exactly as fluently on day nine as it did on day one. There is no status code for an answer that was true when it was generated.

The practical response is to make time-to-live a property of the query class rather than a single global constant. Route questions through a coarse classifier at the call site — evergreen, slow-moving, volatile — and give each its own ceiling, with volatile short enough that a hit is essentially a burst-protection measure rather than a saving. Where you cannot classify, default to short. A low hit rate on live-web content is not a tuning failure; it is the honest cost of the thing you bought.

Citations deserve separate handling from the prose. A cached answer holds URLs that were resolvable at generation time, and links rot faster than claims do — a user clicking a citation and landing on a 404 damages trust more than a slightly dated sentence, because it is visible and checkable. Store the citation set as its own field, revalidate it on a shorter interval than the answer body, and treat a citation that no longer resolves as an invalidation signal for the entry that contains it, not merely a broken link to hide.

Is It the Cache, or Is It Perplexity?

Once a cache is in the request path, every ambiguous production symptom acquires a second suspect. Answers that feel dated could be a time-to-live set too high or a model repointed upstream. A sudden cost increase could be a key-schema change that invalidated everything, or a Perplexity pricing change, or a traffic-mix shift toward the long tail. A latency improvement during an incident could be a healthy cache doing its job, or serve-stale quietly absorbing an outage nobody has noticed yet.

None of those are distinguishable from application logs alone, because in five of the six failure rows above the wire looks perfectly normal. Resolving them requires an independent signal: something outside your application making real requests to api.perplexity.ai on a fixed interval and recording error rate and latency regardless of what your own traffic is doing. With that series in hand, the question takes seconds. Without it, the first half hour of every incident is spent reading a cache implementation that never changed.

Frequently Asked Questions

Related Guides

• Perplexity API Pricing Explained
• Perplexity API Rate Limits Explained
• Perplexity API Latency Guide
• Perplexity API Cost Routing Guide
• Perplexity API Monitoring Guide
• Perplexity API Failover Guide
• Is Perplexity Down? Outage Checking Guide
• Live Perplexity Status Check