Together AI API Fallback Ranking: Choosing Which Substitute Is Actually Acceptable (2026)

Resilience work on an AI API tends to answer three questions and stop. How long should one call be allowed to take. How many times should it be retried. When should we stop calling altogether and cut over. Those are the timeout, the retry budget and the failover, and a team that has built all three will describe Together AI as covered.

None of them answers the fourth question, which is the one that decides what your users actually receive: cut over to what? Failover is a mechanism, and every mechanism takes its destination as an input. Supply that input badly and the mechanism works perfectly — it moves traffic quickly and correctly to a place that cannot do the job. The dashboards will not tell you, because a fallback that returns a response looks exactly like a fallback that returns the right response.

That is the inversion this page is built on. Every other resilience control is graded on whether a response ARRIVED. A fallback ranking is the only one graded on whether the response that arrived was WORTH THE SAME, and those two questions have different answers far more often than the architecture diagram suggests. Ranking is the work of deciding, in advance and with measurements, which substitutes preserve enough of the original capability to be allowed to serve — and which ones must be refused even though they would happily have replied.

Uptime and Price Are the Wrong Ranking Keys

Almost every fallback list is ordered by some combination of availability, cost and latency, because those are the three numbers a provider publishes. They are the wrong primary keys. Uptime predicts whether a rung will answer. Price predicts what the answer costs. Neither predicts whether the answer is acceptable for the request that triggered the fallback, and acceptability is the only property that determines whether the incident was actually handled.

Rank on preserved capability first. For each candidate rung, establish which of the capabilities your request class depends on survive the substitution — and treat a missing hard requirement as disqualifying rather than as a penalty. A rung that drops schema-constrained output is not a worse rung for a pipeline that parses structured bodies; it is not a rung at all, because it will return 200 and break the parser two layers downstream, where the error will be attributed to your code rather than to the failover. Only once the capability filter has run do latency, cost and uptime decide the order of what remains.

This reordering has a second effect that surprises people the first time they do it: some request classes end up with exactly one rung, or none. That is not a failure of the exercise. It is the exercise working. Discovering at configuration time that a particular call has no acceptable substitute lets you design its degraded behaviour deliberately. Discovering it during an incident means you already shipped the unacceptable substitute to users.

The Six Capability Axes a Substitution Silently Drops

These are the dimensions along which two providers that look interchangeable on the wire are not interchangeable in production. Each one fails without an error code.

Capability axis	What you observe	What is actually happening	How to rank it
Usable context length	The rung accepts your prompt and returns a fluent answer	Input over the rung’s real working limit is truncated or attended to poorly; long-document tasks quietly answer from the first half	Store a usable-context figure per rung and drop rungs below the request class’s measured input size
Tool calling fidelity	Tool calls come back and parse	Argument construction, multi-step planning and tool_choice forcing vary widely between rungs even when the wire format matches	Score tool-call correctness on the golden set, not tool-call presence
Schema-constrained output	Valid JSON on the happy path	Some rungs guarantee the schema, some only encourage it; the difference appears as a small percentage of unparseable bodies under load	Treat guaranteed and best-effort as different capability tiers, never as one column
Streaming semantics	Tokens stream, the UI fills in	Chunk shape, first-token latency and how a truncated stream terminates differ; clients that infer completion from stream end mis-handle it	Assert stream termination behaviour per rung in a contract test
Grounding and citations	A confident, well-written answer	A retrieval-backed response and a generated one are indistinguishable in shape; only the sources differ, and the sources are what you sold	Rank any rung that drops grounding BELOW the non-model options
Embedding space	Vectors are returned and written	Vectors from a different model are in a different space; similarity search breaks for those rows and stays broken after the incident	No model rungs on an embedding ladder — queue and retry, or reject the write

The request class declares which of these are hard requirements and which are soft. A hard requirement removes rungs from the ladder. A soft one costs a rung position, and only if you have written the handling code — “no streaming is fine, the UI shows a spinner” is a legitimate soft downgrade if and only if that spinner exists.

The Ladder, From Full Fidelity to Honest Refusal

A ranked ladder for one request class, ordered by how much of the original capability survives. Most stacks implement rungs 0 and 2 and nothing else, which is why their incidents end with a response that nobody can characterise.

Rung 0 — the primary

Full capability, and the reference every other rung is scored against. Its outputs on your golden set are the baseline, so it is the only rung whose quality figure is definitionally zero drop.

Rung 1 — same weights, different host

Available only for open-weights models, and the highest-fidelity substitution that exists. Qualify it by precision and serving stack: the same model name at a different quantisation is a different model for output purposes.

Rung 2 — different model, same capability set

A peer model that satisfies every hard requirement for this request class. This is where the measured degradation figure earns its keep, because the drop is real, bounded, and only knowable from the golden set.

Rung 3 — reduced capability, explicitly handled

A rung that drops a soft requirement you have code for — no streaming, so the UI switches to a spinner; no schema mode, so the repair loop is enabled. Only rank a rung here if the handling actually exists.

Rung 4 — cached previous result

Stale but real, and the first rung that preserves provenance rather than shape. Serve it with its age visible. For grounded or retrieval-backed products this outranks any ungrounded generation.

Rung 5 — deterministic non-AI path

Keyword search, a template, the pre-rerank order, the last known-good extraction. Worse output, zero correctness risk, and it does not depend on any inference provider being up.

Rung 6 — explicit refusal

The terminal rung, and the one that must exist for the ladder to be safe. Tell the caller the capability is unavailable. This is the only honest answer when every rung above it would have to violate a hard requirement to respond.

The ordering rule that does the most work here is the placement of rung 4 and below. A cached previous answer and a deterministic non-AI path both look like large downgrades because they abandon the model entirely — and for any product where the output must be grounded, current or verifiable, they outrank a fresh generation that has lost that property. The instinct to prefer whatever rung returns something shaped like the original response is what puts the most dangerous option at the top of the list.

Score the Ladder Offline, With a Golden Set

The ranking has to be a measurement, not an opinion, and it has to exist before the incident. Take a few hundred real inputs per request class and the reference outputs your primary produced for them. Run every candidate rung against the same inputs. Score with whatever your product can check automatically: schema-validity rate, exact match on the fields you extract, retrieval hit rate, a judged preference rate against the reference, plus p95 latency and cost per call.

What comes out is a degradation figure per rung per request class — a number you can put in a config file and defend in a review. It also gives you the control that turns ranking into a policy: a degradation budget. Decide, per request class, how much quality the feature can absorb before serving it is worse than not serving it. Any rung whose measured drop exceeds that budget does not get ranked lower. It gets removed. A ladder with three rungs inside budget is stronger than one with seven where the bottom four have never been quantified.

Re-run the golden set on a schedule and after any rung’s model identifier changes, because rungs drift under you: providers repoint aliases, ship silent point releases and change serving precision without a version bump. A ranking measured once is accurate on the day it was measured. This is also where the correlation check belongs — two rungs from different vendors that resolve to the same upstream host, region or capacity pool are one rung wearing two invoices, and no amount of quality scoring will reveal that. Look at where the inference actually runs.

Resolve the Ladder at Request Time, and Let It Refuse

At the call site, the request carries its class and its hard requirements. The resolver filters the ladder by those requirements, by any tenant-level policy constraints, and by the current breaker state of each rung, then takes the highest survivor. Two properties make the difference between a resolver that helps and one that hides things.

First, it must be able to return nothing. The common bug is a resolver that walks the list, finds no rung satisfying a hard requirement, and rather than failing falls through to whatever happens to be available — producing a response that violates the constraint the caller explicitly declared, with a success status attached. Make the terminal refusal rung real and reachable, and make callers handle it. An explicit refusal is a product decision you can design around; a silent constraint violation is a correctness bug that reaches users unannounced.

Second, every response has to carry which rung produced it, all the way through your system and into whatever you store. Without that field you cannot answer the question that matters after an incident — which outputs were degraded, by how much, for which customers — and you cannot re-run the affected work once the primary is back. Record the rung id, the model identifier as served, and the degradation figure for that rung and class at the time of the call.

Six Ways a Ranking Fails, and Every One Returns 200

Failure mode	What you observe	What is actually happening	Fix
Ladder chosen by uptime and price	Failover fires, response returns, 200	The rung ranked first cannot do what the request class needs; the parser or the feature breaks downstream, not at the call	Filter by hard capability requirements first, then order the survivors by latency and cost
One global ladder for all traffic	Every call has a fallback configured	An embedding write and a streaming chat turn get the same second rung; one of them is corrupting an index	Key the ladder on request class, defined at the call site with its hard requirements
Rung ranked but never exercised	Config is present and reviewed	Its first real traffic is 100% of primary volume during an incident, with auth, quota or region untested	Route a continuous share of live traffic down each rung and run the golden set on a schedule
Rungs share upstream infrastructure	Two vendor names, two invoices, two keys	Same underlying host, region or capacity pool — both rungs go down in the same incident	Rank on where inference actually runs, not on whose name is on the contract
No terminal refusal rung	Ladder always returns something	When no rung satisfies a hard requirement, the resolver falls through to whatever is merely available and violates the constraint silently	Make the last rung explicit and non-model: cached answer, deterministic path, or honest degraded state
Quality drop is never measured	Incident closed, availability restored	The feature ran for hours on a rung with a large measured gap and nobody can say how large; the cost lands as churn, not as an alert	Emit a degradation figure per fallback invocation from the golden-set delta for that rung and class

The Together AI Trap: The Same Model Somewhere Else Is Not the Same Model

Together AI serves open-weights models, which hands you the highest-fidelity fallback rung available anywhere in this comparison: the identical weights, served by a different host. No other provider in this list can offer that, and it is genuinely worth building around — a same-weights rung preserves prompt behaviour, formatting habits and tool-call style in a way no cross-family substitution does. The trap is assuming the name on the model card is sufficient evidence that it is the same thing.

It frequently is not. The same open model served at a different numeric precision or quantisation — fp8 against bf16, or a community quantisation against the reference weights — produces measurably different output under the same prompt and sampling settings, and the divergence concentrates in exactly the places you care about: long-context recall, strict format adherence, and tool-argument construction. Nothing in the response tells you which serve you got. So the same-weights rung has to be qualified by precision and serving stack, and it has to be validated against your golden set like any other rung, not waved through because the identifier matched. A rung that says “llama-family 70B, some host, some precision” is not a rung; it is a wildcard.

The third issue is that this catalogue makes it trivially easy to add rungs, which is how ladders end up eight deep with no one able to say what the bottom four do. Each additional rung has to be exercised in normal conditions or it is untested code that only ever runs during an incident, and the cost of keeping a rung honest — a golden-set run per model revision, a live share of traffic, an owner — is fixed per rung. Three rungs you can defend beat eight you have never measured, and a deep ladder is a common way to discover during an outage that rungs two through four all sit on the same upstream capacity.

What to Measure

Five numbers, and none of them is availability. The share of requests served by each rung, broken out by request class — this is the only honest statement of how often your users got the real product. The measured degradation figure carried by served responses, aggregated over the window, which converts “we failed over” into a quantity. Terminal-refusal counts, which tell you which request classes have no viable ladder and need one built. Golden-set drift per rung between scheduled runs, which is your early warning that a rung silently changed. And fallback failures — rungs that were ranked, selected, and then could not serve — because that is the incident behind the incident.

What none of them can tell you is whether api.together.xyz was genuinely degraded or whether your own client, network or retry behaviour triggered the ladder. Everything above is measured from inside your process, so a real provider incident and a self-inflicted one produce the same fallback-rate graph — and they call for opposite responses. One means wait and serve rung 2; the other means stop failing over and fix your own code, because every request you shed to a lower rung is quality you gave away for nothing.

Frequently Asked Questions

How is this different from the Together AI failover guide? Is this not the same thing?

Failover is the mechanism: how traffic moves off api.together.xyz once you have decided it should — health detection, cutover, connection handling, getting back. This guide is about the decision the mechanism assumes has already been made, which is WHICH destination traffic moves to and whether the answer you get from it is worth having. Failover treats candidates as interchangeable endpoints and measures success by whether a response arrived. Ranking treats them as unequal products and measures success by how much of the original capability survived. You can have flawless failover into a rung that quietly cannot do the thing your feature needs, and every dashboard will show that incident as handled.

Why not just rank fallbacks by uptime and price?

Because uptime tells you whether a rung will answer and price tells you what the answer costs, and neither tells you whether the answer is acceptable. The axes that actually decide a ranking are capability axes: usable context length, whether tool calling is supported and how faithfully, whether schema-constrained output is real or best-effort, streaming semantics, and for retrieval or embedding surfaces whether the output is even in the same space. A rung with better uptime that drops schema-constrained output is not a better rung for a pipeline that parses structured responses — it is a rung that will return 200 and break the parser downstream. Rank on preserved capability first, then use latency, cost and uptime to order the rungs that survive that filter.

Should there be one ranking for all of my Together AI traffic?

No, and a single global ladder is the most common structural mistake here. The right unit is the request class: the set of calls that share the same hard requirements. A user-facing streaming chat turn, a background extraction job that must emit valid JSON against a schema, and an embedding write into a vector index have almost nothing in common in what they can tolerate, and the rung that is second-best for one can be forbidden for another. Define the ladder alongside the call site, keyed on the requirements that call actually has, and expect some classes to have exactly one rung. A class with one rung is not a bug — it is a fact you want to know before the incident rather than during it.

How do I actually decide the order rather than guessing?

Offline, against a golden set, before anything is on fire. Take a few hundred real inputs per request class, keep the reference outputs from your primary, and run every candidate rung against the same set. Score with whatever check your product can automate — schema-validity rate, exact-match on extracted fields, retrieval hit rate, an LLM-judged preference rate against the reference, plus p95 latency and cost per call. That produces a measured degradation figure per rung per class instead of an opinion, and it lets you set a degradation budget: a rung whose quality drop exceeds what the feature can absorb is not ranked lower, it is removed from the ladder entirely. Re-run it on a schedule, because rungs drift underneath you when providers repoint model aliases.

What if no rung can meet a request’s hard requirements?

Then the correct behaviour is to refuse, and the ladder needs to be able to express that. This is the piece most implementations lack: the resolver walks the list, finds every rung unable to satisfy a hard requirement, and rather than returning nothing it falls through to whichever rung is merely available — producing an answer that violates the constraint the caller declared. Make the terminal rung explicit and non-model: serve a cached previous result with its age shown, serve a deterministic non-AI path, or return an honest degraded state to the caller. An explicit refusal is a recoverable event that your product can design around. A silent constraint violation is a correctness bug that ships to users wearing a success status code.

How do I keep the lower rungs from being untested code?

By running them on purpose in normal conditions. A rung that only executes during an incident is code whose first production traffic arrives on your worst day, at one hundred percent of primary volume, having only ever been exercised at zero. Route a small continuous share of real traffic down each rung and compare outputs against the primary; run the golden set per rung on a schedule and alert when the measured gap moves; and periodically force the ladder open in a controlled window to confirm the whole path — auth, quotas, region, response handling — still works end to end. This also surfaces the correlation problem, which is the one that ruins ladders: rungs that appear independent but share an upstream host, cloud region or capacity pool will fail together, and the only reliable way to discover that is to look at where each rung actually runs rather than at whose name is on the invoice.

Related Guides

Know Whether the Ladder Should Have Fired at All

Every fallback you serve is quality you traded for availability, and your own metrics cannot tell you whether the trade was necessary. API Status Check probes api.together.xyz independently of your traffic and alerts on errors and latency, so you know whether Together AI was really degraded or whether you shed traffic to a lower rung for nothing.

Start Your Free Trial →