Together AI API Multi-Tenant Guide: One Key, Many Customers, No Blast Radius

There is a specific moment where a Together AI integration stops being an integration and becomes a platform: the point at which the person whose prompt you are forwarding is not you. Before that moment, every hard problem is availability-shaped — is it up, did the call fail, how do I retry, where do I fail over. After it, a whole second category appears that none of those controls touch, because the adversary and the cost centre and the data owner are all your own users, and the provider sees exactly one customer: you.

That inversion is what makes multi-tenancy its own problem rather than a harder version of an existing one. A circuit breaker protects you from Together AI. A retry budget protects Together AI from you. Nothing in either protects your tenants from each other, and nothing in either notices when one of them is quietly the entire invoice.

What the Provider Cannot See About Your Tenants

Start from the provider's point of view, because every gap in it is a control you have to build. Together AI sees one account, a set of keys, and a stream of requests. Here is what that view structurally cannot resolve, and what each blind spot costs you.

Every row resolves to the same sentence: the provider is not a participant in your multi-tenancy. That is not a criticism of Together AI — it is the correct design for a wholesale API. It just means the entire mechanism lives on your side of the wire, and any plan that begins “we will use the provider's per-key limits” is skipping the part where you check whether those limits are per key.

The Isolation Ladder

Multi-tenant Together AI deployments sit on one of five rungs. Most teams believe they are one rung higher than they are, because attribution and isolation feel like the same thing until the first contention incident proves they are not.

The jump that matters is 2 to 3. Rung 2 is where most production systems actually live, and it feels safe because the dashboard is informative — you can see the noisy tenant clearly. But a dashboard is a record of a decision you did not make. Rung 3 is the first rung where the system says no on its own, at 3am, without anyone reading anything.

Admission Control, In the Request Path

The control that does the work is small and unglamorous: before the outbound call, ask whether this tenant is allowed to make it. Two independent checks — a rate bucket and a concurrency cap — because they fail differently. A tenant can stay under any per-minute rate and still occupy your entire connection pool with long streaming requests, and a tenant can hold one connection and still burn a per-minute allowance in a loop.

// Per-tenant admission for Together AI. State lives in Redis so the budget
// holds across every process — a per-instance bucket silently multiplies
// the real limit by your instance count, which is the most common way
// this control is built and then does not work.

const TIERS = {
  free:       { rpm: 5,   concurrent: 1,  usdPerDay: 0.50 },
  pro:        { rpm: 60,  concurrent: 4,  usdPerDay: 25 },
  enterprise: { rpm: 600, concurrent: 25, usdPerDay: 500 },
};

async function admit(tenant) {
  const tier = TIERS[tenant.tier];

  // 1. Rolling-window rate, not a counter reset on the minute boundary.
  //    A minute-boundary counter lets a tenant spend two full allowances
  //    back to back across the boundary.
  const rate = await redis.eval(ROLLING_WINDOW_LUA,
    ['rate:' + tenant.id], [tier.rpm, 60, nowMs()]);
  if (!rate.allowed) return deny('rate', rate.retryAfterMs);

  // 2. Concurrency, held for the life of the request. Independent of rate:
  //    long streaming calls exhaust connections without moving the rate.
  const slot = await acquireSlot('conc:' + tenant.id, tier.concurrent);
  if (!slot) return deny('concurrency', 250);

  // 3. Spend, priced at WRITE time from a dated rate table. Soft threshold
  //    degrades, hard threshold refuses — and the hard one is only ever
  //    applied to free and trial tiers.
  const spentToday = await redis.get('spend:' + tenant.id + ':' + today());
  if (spentToday >= tier.usdPerDay) {
    if (tenant.tier === 'free') return deny('budget', null);
    degradeToCheaperModel(tenant);   // paid tiers degrade, never hard-fail
    alertAccountOwner(tenant);
  }

  return { ok: true, slot };
}

// The sum of concurrent tenant allowances must sit BELOW the account
// ceiling. Tuning it to exactly 100% means normal variance produces 429s
// that look like a provider incident.

Three details in there are the ones that get dropped and each has a characteristic failure. State in process memory rather than shared storage multiplies your effective limit by the instance count, so the control appears to work in staging with one instance and does nothing in production with twelve. A minute-boundary counter instead of a rolling window lets a tenant spend two full allowances back to back across the boundary. And a hard spend refusal applied to a paying customer converts a cost problem into a churn problem — degrade paid tiers, refuse free ones.

The Together AI Trap: Isolation That Works by Accident Until Two Tenants Agree

Together AI is a marketplace, and its rate limits are scoped per model rather than pooled across the account. That produces the most dangerous shape a multi-tenant system can have: isolation that appears to work and was never designed. While tenant A is on one open-weights model and tenant B is on another, they genuinely do not contend, every load test passes, and the absence of admission control looks like a decision that was correct. The day a model becomes popular and both tenants land on it, the isolation vanishes with no code change on your side and no incident on the provider's.

Anything model-scoped is a boundary you do not control, because tenants — or your own defaults, or a deprecation that migrates everyone onto the successor model — choose which side of it they sit on. Treat per-model limits as a happy accident that reduces the frequency of contention, never as the mechanism that prevents it. The admission control has to exist regardless, and the bucket key should be tenant plus model so you can see contention forming before it becomes a report.

The related marketplace hazard is that per-model limits and per-model pricing move independently and per model. A tenant that migrates from one model to another can multiply both their cost and their share of a ceiling in a single deploy, and neither the invoice nor the rate-limit headers will attribute the change to them unless the exact model id is on every ledger row.

Six Multi-Tenant Failures That Return HTTP 200

This is why multi-tenancy resists the monitoring you already have. Availability problems announce themselves with a status code; tenancy problems do not. Every row below is a successful response.

The pattern across all six is that the request-level metrics you already collect — status code, latency, error rate — are blind to every one of them. Each requires a metric with a tenant dimension, which is the real argument for putting the tenant id on the ledger row long before you think you need it. You cannot add that dimension retroactively to traffic that has already happened.

Data Isolation Is Not a Together AI Setting

Nothing in the Together AI API mixes one customer's data into another's response. Every real cross-tenant disclosure in an LLM product happens in code that runs before or after the call, in three places that are all optimisations someone added for good reasons.

• The response cache. Keyed on the prompt, it is correct for a single-tenant app and a disclosure channel for a multi-tenant one. The tenant id belongs in the key, including in the negative and error caches that get added later without a review.
• The retrieval index. A filter that a caller can forget will eventually be forgotten. Prefer a namespace per tenant, or a single query function that takes the tenant id as a required argument and builds the filter itself, with no code path that reaches the store directly.
• Prompt assembly. Few-shot examples mined from recent production traffic, summaries carried between turns, and any “similar past request” feature all move text between requests. If that text is user content, the selector needs a tenant argument for the same reason the index does.

There is also a contractual layer that outlives all three. Whatever data-retention and training posture you have agreed with Together AI applies to every tenant's content, because it is all your account. A tenant that needs stricter terms than the ones you hold cannot get them inside your account at any rung below bring-your-own-key — see the data privacy guidefor what those terms actually cover.

When Bring-Your-Own-Key Is Worth It

BYOK is the top rung and it is genuinely different in kind: the tenant's requests are billed to the tenant, bounded by the tenant's own quota, and governed by the tenant's agreement with Together AI. Cost blast radius, quota contention and the content-policy exposure all leave your account in one move. It also creates three new obligations that are easy to underestimate.

The honest recommendation is that BYOK is an enterprise-tier feature, driven by procurement rather than engineering, and that shipping it does not remove the need for rungs 2 through 4. You will run both models simultaneously — pooled for self-serve, BYOK for enterprise — so the ledger needs a field recording which key paid for each row, or your per-tenant unit economics will silently include traffic you never paid for.

Abuse: When a Tenant Is Not a Customer

The last category is the one that looks like healthy growth. An endpoint that accepts arbitrary text and forwards it to Together AI is a free inference proxy for anyone who can reach it, and the traffic it attracts is indistinguishable from enthusiastic usage in every aggregate metric. Cost per active user rises, engagement looks excellent, and nobody investigates a chart that is going up.

The reliable discriminator is pacing rather than volume. Human sessions have irregular gaps — seconds to minutes, with pauses to read the output. Scripted sessions have machine-uniform intervals and no reading pause at all, and they keep going through the hours where your real users are asleep. Two cheap signals catch nearly all of it: per-tenant request intervals with a low variance, and a ratio of request volume to any product action that is not the API call itself. A tenant generating thousands of completions and saving nothing, exporting nothing and viewing nothing is not using your product.

What to do about it is a product decision, but the technical prerequisites are the same three things in this guide: authenticate before the call so anonymous traffic is impossible, meter per tenant so the cost is attributable, and hold a hard budget on free tiers so the worst case is bounded at a number you chose. For the systematic version of the cost side, see the cost attribution guide; for tenant-supplied text that tries to redirect the model rather than just consume it, see the prompt injection guide.

Where Overflow Traffic Should Go

A per-tenant budget that only ever refuses is a blunt instrument. Once admission control exists, the interesting option is routing overflow to a second provider on separate infrastructure and a separate account ceiling, so a tenant over their share degrades rather than stops.

Deciding whether a destination's answer is worth the same is its own problem — see the fallback ranking guide, and note that a per-tenant overflow route needs the tenant id carried through to the second provider's ledger rows too, or the cheaper path becomes the unattributed one.

Frequently Asked Questions

Related Together AI Guides

• Together AI API Cost Attribution Guide
• Together AI API Rate Limits
• Together AI API Key Rotation Guide
• Together AI API Data Privacy Guide
• Together AI API Prompt Injection Guide
• Together AI API Quota Increase Guide
• Together AI API Response Caching Guide