Together AI Circuit Breakers: When to Stop Calling an API That Is Still Answering (2026)

Most resilience advice for AI APIs stops at the boundary of a single request. Set a timeout so a call cannot hang forever. Add backoff so a retry does not arrive instantly. Both are necessary and neither answers the question this guide is about: at what point should your system stop sending requests to api.together.xyz altogether, and what should it serve instead while it is not sending them?

That is a different decision with a different owner. A timeout is a property of one call and it is enforced by the caller. A circuit breaker is a property of the dependency and it is enforced across every caller in the process — it is stateful, it remembers what happened to the last few hundred requests, and it applies that memory to the next one before the network is touched. The distinction matters because the failure mode it exists to prevent is not “this request failed.” It is “this dependency degraded and took the rest of my service with it.”

This is also not the retry-budget problem, and confusing the two produces systems with neither control working properly. A retry budget limits how much additional load you add to Together AI after a failure; the beneficiary is the provider. A breaker limits how much of your own capacity you spend on a dependency that is not currently useful; the beneficiary is you. You want both, they trip on different signals, and one does not substitute for the other.

The Real Failure Is Resource Exhaustion, Not the Outage

Here is the sequence that takes down services that were, on paper, fully protected. api.together.xyz slows from a normal response time to several seconds. No errors are returned, so every error-rate alert stays green. But each in-flight call now occupies one of your workers — a thread, a connection, a slot in whatever bounds your real concurrency — for many times longer than the capacity plan assumed. Your pool fills. Requests to endpoints that have nothing to do with Together AI begin queueing behind calls to it. The service that goes down is yours, the cause is a dependency that never returned a single error, and the graph that shows it is saturation, not errors.

This is why the first control to build is not the breaker at all. It is the bulkhead: a hard cap on concurrent in-flight calls per dependency, set below your total worker count, so that Together AI can consume at most its allotted share of your capacity no matter how slow it gets. Calls beyond the cap are rejected immediately rather than queued. That single limit is what keeps a Together AI incident from becoming a site-wide incident, and it works even with no breaker present.

The breaker is the layer above. Once the bulkhead is rejecting calls, the rejection count becomes your earliest and cleanest degradation signal — well ahead of errors, usually ahead of your latency alert — and it is one of the three inputs the breaker should be watching. Build the bulkhead first, feed its rejections into the breaker, and the breaker suddenly has something honest to trip on.

What Should Actually Trip It

Error rate alone is the default and it is close to useless for an AI API, because the characteristic degradation is slow success. Watch three signals over a rolling window: the error rate, the share of calls exceeding a wall-clock latency ceiling, and the count of bulkhead rejections. Any of the three crossing its threshold should open the breaker.

Two details decide whether this behaves in production. First, require a minimum call volume in the window before the breaker is permitted to trip; without a floor, a low-traffic route opens on two failures out of three requests and stays open on essentially no evidence. Second, make the latency ceiling an absolute number derived from what the caller can wait for, not a multiple of your own recent baseline. A relative threshold ratchets: every slow period raises the baseline, which raises the threshold, which makes the next slow period look normal.

The inversion worth internalising is that the breaker’s job is not to detect whether Together AI is down. It cannot know that — everything it observes is measured from inside your own process, and a retry storm of your own making produces the same graphs as a genuine provider outage. Its job is narrower and entirely local: decide whether calls to this dependency are currently worth the resources they cost. Whether the provider is actually broken is a question only an independent external probe can answer, and it is the question that determines whether you wait or escalate.

The Three States, and the One Everyone Gets Wrong

Closed

Traffic flows normally. The breaker is only counting: errors, latency-ceiling breaches and rejections from the concurrency limiter, over a rolling window with a minimum-volume floor.

Open

Calls fail immediately without touching the network, and the fallback path serves. This is the state that gives you back your latency — without it, every request in your system still pays the full timeout before falling back.

Half-open

After the cooling period, a hard-capped number of concurrent real requests are allowed through. Several consecutive clean results close the breaker; one failure re-opens it and lengthens the next cooling period.

Half-open is where most implementations quietly fail. Two mistakes recur. Closing after a single successful probe means one lucky request restores full fleet traffic to a dependency that is still degraded, which re-trips the breaker immediately — that is the flapping loop. And allowing probes at a rate rather than a hard concurrent cap means every queued caller arrives the instant the cooling period ends, which is the recovery stampede in miniature. Require consecutive clean probes, cap concurrency during half-open at a small integer, and grow the cooling period each time the breaker re-opens instead of resetting it.

The third mistake is closing on the wrong evidence entirely: polling a status or health endpoint and closing when it returns healthy. Those endpoints are cheap, usually unauthenticated, and served from a different path than inference. They return healthy during real incidents on a routine basis. The only probe that tells you anything is a real request of the kind your product actually sends.

Breaker Scope: The Configuration That Decides Everything Else

A breaker covers one thing that fails as a unit. Get the key wrong and no amount of threshold tuning rescues it. Too broad — one breaker per provider — and a single broken model or endpoint is averaged into a healthy fleet, so the breaker either never trips or trips and sheds traffic that was working perfectly. Too narrow — one breaker per user or per request — and no single breaker ever sees enough calls to reach its volume floor, so none of them ever trip.

The workable default for an AI API is provider plus endpoint family plus model, with region added if you route across regions. Ask the diagnostic question for each candidate key: if this breaker opens, is every call it covers genuinely not worth making? If the answer is no for any of them, the key is too broad.

One more interaction to get right: hedged requests. Sending a duplicate request after a short delay and taking whichever returns first is an excellent latency tool and a terrible incident tool, because it doubles offered load at precisely the wrong moment. Gate hedging on breaker state — hedge only while the breaker is closed and the error rate is low, and disable it the moment either signal moves.

Choosing a Fallback That Is Not Secretly the Same Dependency

A breaker with no fallback is just a faster error, and a badly chosen fallback is worse than none because it hides the incident until something downstream breaks. Rank your options by how much of the original value they preserve: a genuinely independent second provider; a smaller or cheaper model at the same provider, which only helps if the failure is model-scoped rather than account-wide; a cached previous answer served with its age visible; a deterministic non-AI path such as keyword search or a template; and an explicit degraded state that tells the user the feature is unavailable.

Two properties separate a real fallback from a decorative one. It must not share infrastructure with the primary — same host, same key, same region means both are down in the same incident. And it must be capacity-planned for one hundred percent of primary traffic, not the one percent it normally sees, because full traffic is the only load it will ever be given when it matters. A fallback path that is never exercised in normal conditions is untested code on your most important day; route a real share of production traffic through it continuously so you find out now.

And be honest about which calls can degrade at all. Anything that writes durable state derived from the model — embeddings into an index, extracted fields into a database — must fail closed rather than substitute. A silently different model writing into the same store produces corruption that outlives the incident and is usually discovered weeks later.

Six Ways This Fails, and Most of Them Return 200

Failure mode	What you observe	What is actually happening	Fix
Dependency is slow, not failing	All calls return 200	Your worker pool fills with calls that will eventually succeed too late; unrelated routes queue behind them	Trip on a wall-clock latency ceiling, not just error rate
One breaker for the whole host	Aggregate error rate looks moderate	A broken model is averaged into healthy ones; the breaker either never trips or sheds healthy traffic	Key the breaker on the smallest unit that fails independently — endpoint family, then model
Breaker closes on a health-check endpoint	Health check 200, breaker closed	Health checks are cheap and served from a different path; the inference endpoint is still unusable	Half-open with a real request of the shape you actually send
Fallback shares the primary’s infrastructure	Fallback configured and tested	Same host, key or region — both fail in the same incident and the fallback path never fires usefully	Require the fallback to differ in provider or region; assert it in config review
Fallback only ever sees 1% of traffic	Fallback returns 200 in tests	It has never been load-tested at 100%, which is the only volume it will ever get	Route a real share of traffic to it continuously, or load-test it at primary volume
No concurrency cap behind the breaker	Breaker configured, metrics green	Before enough calls fail to trip it, in-flight slow calls have already exhausted the pool	Bulkhead first: a hard concurrent-call limit per dependency, sized below your worker count

The Together AI Trap: Health Is a Property of the Model, Not the Host

Together AI serves a very large catalogue of open models behind a single API, and those models do not share capacity in any way you can observe. One model can be saturated, cold, or temporarily withdrawn while every other model on the same host answers normally. This makes the default breaker key — the hostname — almost meaningless: it will either never trip, because most traffic is fine, or it will trip and shed traffic to models that were never broken.

Key the breaker on the model identifier. That produces more breakers with less traffic each, so you also have to lower the volume threshold at which a breaker is allowed to trip at all, and accept that a rarely-called model will spend most of its life with too little data to make a decision. That is the honest state, and it is better than one host-level breaker that averages a broken model into a healthy fleet and reports green.

The specifically Together-shaped trap is cold starts. A model that has not been called recently can take many seconds on its first request and then be fast for every subsequent one. That signature — one very slow call, then normal — is indistinguishable from the start of an incident to a latency breaker, and if the breaker opens on it, the model never gets warm, so the next probe is cold and slow too. The breaker holds the model in exactly the state it is punishing it for. Exclude the first call after an idle window from the trip statistics, or keep a low-rate warming call in place so the cold path is never the one your breaker is measuring.

What to Measure

Five numbers. Time spent with each breaker open, as a share of the window — this is your real availability for that dependency, and it is usually worse than any provider-reported figure. Open and close transitions per hour, which is your flapping detector. Bulkhead rejections, the earliest honest signal of degradation. Fallback invocations and, separately, fallback failures, because a fallback that fails under full traffic is the incident behind the incident. And the share of half-open probes that succeed, which tells you whether your cooling period is tuned or whether you are just guessing.

What none of them can tell you is whether api.together.xyz is actually degraded. All five are measured from inside your own process, so during an incident they describe your reaction rather than its cause — and your own retry storm produces graphs almost identical to a genuine provider outage. Independent external monitoring is what separates “they are down, wait it out” from “we are doing this to ourselves,” and those two situations call for opposite responses.

Frequently Asked Questions

How is this different from fixing my Together AI timeout errors? I already read that guide.

A timeout guide is about one call that took too long and what to change so it does not: the value itself, the connect-versus-read split, streaming, request size. This guide starts one level up, at the point where you accept that calls to api.together.xyz are going to fail for a while and the question becomes whether your system keeps trying. The timeout is what bounds a single request. The breaker is what decides that the next thousand requests should not be sent at all. You need both, and the ordering matters: a breaker on top of an unbounded timeout is close to useless, because the resource exhaustion it exists to prevent has already happened by the time enough calls have failed for it to trip.

A circuit breaker protects the provider from my traffic, right?

That is the common description and it is the wrong way round for a third-party API. You are one customer among many, and shedding your share is not what rescues Together AI. The breaker exists to protect YOU — specifically, to stop a slow dependency from consuming your own finite resources. Every in-flight call to api.together.xyz is holding one of your workers, one connection, one slot of whatever your concurrency limit actually is. When that dependency slows down, those resources are held longer, your pool fills with calls that are going to fail anyway, and requests to completely unrelated parts of your service start queueing behind them. The breaker stops that spread. Limiting your added load on the provider is what a retry budget does; these are two different controls with two different beneficiaries.

What should make the breaker trip — errors, or something else?

Errors alone will miss most real incidents with an AI API, because the characteristic failure is slow success rather than failure. Trip on three signals combined: the error rate over a rolling window, the share of calls exceeding a wall-clock latency ceiling, and the count of calls rejected by your own concurrency limiter, which is the earliest sign that the dependency is holding resources longer than planned. Use a rolling window rather than consecutive failures so one unlucky burst does not open the breaker, and require a minimum call volume in the window before the breaker is allowed to trip at all — without that floor, a low-traffic route trips on two failures out of three and stays open on essentially no evidence.

How does the breaker know when to close again?

By sending real traffic, in small amounts, and watching what happens — the half-open state. After the cooling period the breaker permits a strictly limited number of requests through, and it must be a hard concurrent limit rather than a rate, or the whole queued fleet arrives at once and you have rebuilt the stampede you were avoiding. If those probes succeed within the latency ceiling, close; if any fails, open again and extend the cooling period. What you should not do is close on the basis of a separate health-check endpoint. Health checks are cheap, unauthenticated and served from a different path than your actual workload, so they routinely return healthy while the model endpoint you care about is unusable. The only honest probe is a real request of the kind you actually send.

The breaker is open. What do I serve?

Decide this before you build the breaker, because a breaker with no fallback is just a faster error. In descending order of quality: a second provider if the outputs are genuinely interchangeable for your use case; a smaller or cheaper model at the same provider if the failure is model-scoped rather than account-wide; a cached previous answer with its age shown to the user; a deterministic non-AI path such as keyword search or a template; and finally an explicit, honest degraded state. Two traps to avoid. A fallback that shares infrastructure with the primary — same host, same key, same region — is not a fallback and will be down at the same time. And a fallback path that only ever sees one percent of traffic has never been load-tested at a hundred percent, which is the only volume it will ever receive when it matters. Exercise it on purpose in normal conditions.

My breaker keeps opening and closing every minute. What is wrong?

Flapping almost always means the breaker is keyed too broadly, or it is closing on too little evidence. Too broad: one hostname-wide breaker mixes a genuinely broken model or endpoint with several healthy ones, so the aggregate error rate hovers right at your threshold and crosses it repeatedly — split the key until each breaker covers one thing that fails as a unit. Too little evidence: closing after a single successful probe means one lucky request restores full traffic to a dependency that is still degraded, which immediately re-trips it. Require several consecutive clean probes, and make the cooling period grow each time the breaker re-opens rather than resetting to its initial value. Flapping is also expensive in its own right when your fallback is slower or costlier than the primary, so treat open/close transitions per hour as a metric worth alerting on.

Related Guides

Your Breaker Cannot Tell You Whose Fault It Is

Every signal a circuit breaker uses is measured inside your own process, so a real Together AI incident and a self-inflicted one look the same from there. API Status Check probes api.together.xyz independently of your traffic and alerts on errors and latency, so you know whether to wait it out or to look at your own code.

Start Your Free Trial →