Cohere API Multi-Tenant Guide: One Key, Many Customers, No Blast Radius

There is a specific moment where a Cohere integration stops being an integration and becomes a platform: the point at which the person whose prompt you are forwarding is not you. Before that moment, every hard problem is availability-shaped — is it up, did the call fail, how do I retry, where do I fail over. After it, a whole second category appears that none of those controls touch, because the adversary and the cost centre and the data owner are all your own users, and the provider sees exactly one customer: you.

That inversion is what makes multi-tenancy its own problem rather than a harder version of an existing one. A circuit breaker protects you from Cohere. A retry budget protects Cohere from you. Nothing in either protects your tenants from each other, and nothing in either notices when one of them is quietly the entire invoice.

What the Provider Cannot See About Your Tenants

Start from the provider's point of view, because every gap in it is a control you have to build. Cohere sees one account, a set of keys, and a stream of requests. Here is what that view structurally cannot resolve, and what each blind spot costs you.

Every row resolves to the same sentence: the provider is not a participant in your multi-tenancy. That is not a criticism of Cohere — it is the correct design for a wholesale API. It just means the entire mechanism lives on your side of the wire, and any plan that begins “we will use the provider's per-key limits” is skipping the part where you check whether those limits are per key.

The Isolation Ladder

Multi-tenant Cohere deployments sit on one of five rungs. Most teams believe they are one rung higher than they are, because attribution and isolation feel like the same thing until the first contention incident proves they are not.

The jump that matters is 2 to 3. Rung 2 is where most production systems actually live, and it feels safe because the dashboard is informative — you can see the noisy tenant clearly. But a dashboard is a record of a decision you did not make. Rung 3 is the first rung where the system says no on its own, at 3am, without anyone reading anything.

Admission Control, In the Request Path

The control that does the work is small and unglamorous: before the outbound call, ask whether this tenant is allowed to make it. Two independent checks — a rate bucket and a concurrency cap — because they fail differently. A tenant can stay under any per-minute rate and still occupy your entire connection pool with long streaming requests, and a tenant can hold one connection and still burn a per-minute allowance in a loop.

// Per-tenant admission for Cohere. State lives in Redis so the budget
// holds across every process — a per-instance bucket silently multiplies
// the real limit by your instance count, which is the most common way
// this control is built and then does not work.

const TIERS = {
  free:       { rpm: 5,   concurrent: 1,  usdPerDay: 0.50 },
  pro:        { rpm: 60,  concurrent: 4,  usdPerDay: 25 },
  enterprise: { rpm: 600, concurrent: 25, usdPerDay: 500 },
};

async function admit(tenant) {
  const tier = TIERS[tenant.tier];

  // 1. Rolling-window rate, not a counter reset on the minute boundary.
  //    A minute-boundary counter lets a tenant spend two full allowances
  //    back to back across the boundary.
  const rate = await redis.eval(ROLLING_WINDOW_LUA,
    ['rate:' + tenant.id], [tier.rpm, 60, nowMs()]);
  if (!rate.allowed) return deny('rate', rate.retryAfterMs);

  // 2. Concurrency, held for the life of the request. Independent of rate:
  //    long streaming calls exhaust connections without moving the rate.
  const slot = await acquireSlot('conc:' + tenant.id, tier.concurrent);
  if (!slot) return deny('concurrency', 250);

  // 3. Spend, priced at WRITE time from a dated rate table. Soft threshold
  //    degrades, hard threshold refuses — and the hard one is only ever
  //    applied to free and trial tiers.
  const spentToday = await redis.get('spend:' + tenant.id + ':' + today());
  if (spentToday >= tier.usdPerDay) {
    if (tenant.tier === 'free') return deny('budget', null);
    degradeToCheaperModel(tenant);   // paid tiers degrade, never hard-fail
    alertAccountOwner(tenant);
  }

  return { ok: true, slot };
}

// The sum of concurrent tenant allowances must sit BELOW the account
// ceiling. Tuning it to exactly 100% means normal variance produces 429s
// that look like a provider incident.

Three details in there are the ones that get dropped and each has a characteristic failure. State in process memory rather than shared storage multiplies your effective limit by the instance count, so the control appears to work in staging with one instance and does nothing in production with twelve. A minute-boundary counter instead of a rolling window lets a tenant spend two full allowances back to back across the boundary. And a hard spend refusal applied to a paying customer converts a cost problem into a churn problem — degrade paid tiers, refuse free ones.

The Cohere Trap: The Isolation Boundary Is in the Index, Not in the API Call

Most Cohere deployments are retrieval pipelines: embed, store, search, rerank. In that shape the multi-tenant boundary is almost never the API call — it is the vector store the embeddings land in, which is a system you own and the provider has no visibility into. You can hold every provider-side control perfectly and still leak, because the leak happens after the call succeeds.

The failure is a missing filter, and it returns HTTP 200. A search that omits the tenant predicate returns neighbours from the whole index, ranked by similarity with no notion of ownership, and then the rerank step does exactly what it is supposed to and puts another tenant's most relevant document at position one. Every component behaves correctly. The output is a confident, well-cited answer built from data the requesting tenant has no right to see, and no error is raised anywhere in the chain.

Because of that, tenant scoping must be structural rather than a parameter a caller can forget. Either give each tenant its own namespace or collection, or route every query through a single function that takes the tenant id as a required argument and constructs the filter itself, with no code path that reaches the index directly. Add a test that asserts a cross-tenant query returns zero results, and run it on every deploy — it is the only signal that will ever fire, since the production failure mode is a successful response.

Six Multi-Tenant Failures That Return HTTP 200

This is why multi-tenancy resists the monitoring you already have. Availability problems announce themselves with a status code; tenancy problems do not. Every row below is a successful response.

The pattern across all six is that the request-level metrics you already collect — status code, latency, error rate — are blind to every one of them. Each requires a metric with a tenant dimension, which is the real argument for putting the tenant id on the ledger row long before you think you need it. You cannot add that dimension retroactively to traffic that has already happened.

Data Isolation Is Not a Cohere Setting

Nothing in the Cohere API mixes one customer's data into another's response. Every real cross-tenant disclosure in an LLM product happens in code that runs before or after the call, in three places that are all optimisations someone added for good reasons.

• The response cache. Keyed on the prompt, it is correct for a single-tenant app and a disclosure channel for a multi-tenant one. The tenant id belongs in the key, including in the negative and error caches that get added later without a review.
• The retrieval index. A filter that a caller can forget will eventually be forgotten. Prefer a namespace per tenant, or a single query function that takes the tenant id as a required argument and builds the filter itself, with no code path that reaches the store directly.
• Prompt assembly. Few-shot examples mined from recent production traffic, summaries carried between turns, and any “similar past request” feature all move text between requests. If that text is user content, the selector needs a tenant argument for the same reason the index does.

There is also a contractual layer that outlives all three. Whatever data-retention and training posture you have agreed with Cohere applies to every tenant's content, because it is all your account. A tenant that needs stricter terms than the ones you hold cannot get them inside your account at any rung below bring-your-own-key — see the data privacy guidefor what those terms actually cover.

When Bring-Your-Own-Key Is Worth It

BYOK is the top rung and it is genuinely different in kind: the tenant's requests are billed to the tenant, bounded by the tenant's own quota, and governed by the tenant's agreement with Cohere. Cost blast radius, quota contention and the content-policy exposure all leave your account in one move. It also creates three new obligations that are easy to underestimate.

The honest recommendation is that BYOK is an enterprise-tier feature, driven by procurement rather than engineering, and that shipping it does not remove the need for rungs 2 through 4. You will run both models simultaneously — pooled for self-serve, BYOK for enterprise — so the ledger needs a field recording which key paid for each row, or your per-tenant unit economics will silently include traffic you never paid for.

Abuse: When a Tenant Is Not a Customer

The last category is the one that looks like healthy growth. An endpoint that accepts arbitrary text and forwards it to Cohere is a free inference proxy for anyone who can reach it, and the traffic it attracts is indistinguishable from enthusiastic usage in every aggregate metric. Cost per active user rises, engagement looks excellent, and nobody investigates a chart that is going up.

The reliable discriminator is pacing rather than volume. Human sessions have irregular gaps — seconds to minutes, with pauses to read the output. Scripted sessions have machine-uniform intervals and no reading pause at all, and they keep going through the hours where your real users are asleep. Two cheap signals catch nearly all of it: per-tenant request intervals with a low variance, and a ratio of request volume to any product action that is not the API call itself. A tenant generating thousands of completions and saving nothing, exporting nothing and viewing nothing is not using your product.

What to do about it is a product decision, but the technical prerequisites are the same three things in this guide: authenticate before the call so anonymous traffic is impossible, meter per tenant so the cost is attributable, and hold a hard budget on free tiers so the worst case is bounded at a number you chose. For the systematic version of the cost side, see the cost attribution guide; for tenant-supplied text that tries to redirect the model rather than just consume it, see the prompt injection guide.

Where Overflow Traffic Should Go

A per-tenant budget that only ever refuses is a blunt instrument. Once admission control exists, the interesting option is routing overflow to a second provider on separate infrastructure and a separate account ceiling, so a tenant over their share degrades rather than stops.

Deciding whether a destination's answer is worth the same is its own problem — see the fallback ranking guide, and note that a per-tenant overflow route needs the tenant id carried through to the second provider's ledger rows too, or the cheaper path becomes the unattributed one.

Frequently Asked Questions

Related Cohere Guides

• Cohere API Cost Attribution Guide
• Cohere API Rate Limits
• Cohere API Key Rotation Guide
• Cohere API Data Privacy Guide
• Cohere API Prompt Injection Guide
• Cohere API Quota Increase Guide
• Cohere API Response Caching Guide