Together AI API 503 Service Unavailable: What It Means and How to Respond (2026)

Most API errors are an argument about the request. A 503 is not. The JSON was valid, the key was accepted, the model existed — and the service still said no, because at that instant it had nowhere to run the work. Reading it as a bug sends teams into a debugging session that cannot possibly find anything, because there is nothing in the code to find.

30-second triage: run the same call with curl from outside your deployment. 503 there too? It is Together AI — check live Together AI status and skip to the retry policy below. Succeeds in curl? The 503 is being generated by something between your app and api.together.xyz, and the fix is on your side.

Which 5xx you actually received

The status code narrows the cause far more than the message body does, and the four you will see from an AI endpoint mean genuinely different things. Getting this wrong is expensive: three of the four are worth retrying and one usually is not.

Code	What it means	Retry?
`429`	Your key exceeded its own allowance	Yes, after backoff — but fix your pacing
`500`	Something broke while handling the request	Once or twice; a persistent 500 is a bug report
`502`	A gateway got a bad response from upstream	Yes — usually brief and self-clearing
`503`	No capacity available for the request	Yes, with jitter and a budget
`504`	Upstream took longer than the gateway allowed	Only if the call is idempotent

One caveat that catches everyone: a 503 in your logs did not necessarily come from api.together.xyz. Load balancers, service meshes and your own web framework all emit 503 when they are shedding load, and the code is identical. Log the response headers alongside the status — the provider's responses carry identifying headers that your own gateway's do not.

📡

Recommended

Know Which Side Returned the 503

External checks against your AI endpoints run from outside your infrastructure, so you can tell a provider capacity refusal from your own gateway shedding load — in seconds, not after twenty minutes of arguing.

Try Better Stack Free →

Why Together AI returns 503 specifically

Together AI is a marketplace, not a single model service, and that changes what a 503 means. Capacity is allocated per model, so the platform can be entirely healthy while one endpoint has none. The practical consequence: never test liveness with a different model than the one that failed. A 200 from a popular flagship tells you the API is up and tells you nothing about the fine-tuned or long-tail model your job depends on.

Together AI exposes an OpenAI-compatible surface, so most teams reach it through an OpenAI SDK with a swapped base URL and inherit whatever that SDK does on 5xx. Verify the retry defaults; a client that retries a 503 immediately against a model with no capacity simply spends your latency budget three times.

curl -i -s -o /dev/null -w "%{http_code} %{time_total}s\n" \
  https://api.together.xyz/v1/chat/completions \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.3-70B-Instruct-Turbo","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'

Run that on a short loop during an incident rather than once. A single sample cannot distinguish a total outage from partial capacity, and partial is the far more common state: ten samples returning seven 200s and three 503s is a completely different operational situation from ten out of ten failing, and it calls for a different response.

The Together AI trap: 503s are per-model, and the tail is where they live

Serverless endpoints for less-popular models are the ones that go unavailable first and stay that way longest, because there is less standing capacity behind them. A model your team picked eighteen months ago on a benchmark can quietly become the least-served entry in the catalog. Poll the model list on a schedule and diff it, and record which model each error came from — an error rate averaged across models hides the one endpoint that is actually failing.

A retry policy that helps instead of hurting

The default reflex — three immediate retries — is the worst possible response to a capacity error. It triples your offered load at the exact moment the service has none to give, and every other client on the platform is doing the same thing at the same time. Four rules turn retries from an amplifier into a recovery mechanism:

Honour Retry-After. When the response carries one, it is the only number in the exchange that reflects what the service actually knows. Waiting less is not clever.
Back off with full jitter. Randomise the delay across the whole window rather than adding a small wobble. Deterministic backoff makes every client wake at the same instant and hit the first sliver of recovered capacity together.
Budget by ratio, not by count. Allow retries as a fraction of recent successes. When almost everything is failing the allowance collapses on its own, without a deploy.
Retry against a deadline. If the user-facing request gave up ten seconds ago, attempt three is pure load on behalf of nobody. Propagate the deadline and skip any attempt that cannot finish inside it.

The full treatment, including how to size the budget and ramp concurrency back after recovery, is in the Together AI retry budget guide.

What your product does while it lasts

Retries buy seconds. A capacity event that runs for twenty minutes needs a product answer, and there are only three honest ones. Decide which applies to each call site before an incident, because the decision is a product decision and nobody makes good ones at 2am.

Fail over

Route to a second provider behind the same interface. Best for calls where any competent model will do. Needs the fallback kept warm and tested, not merely configured. See the Together AI failover guide.

Degrade visibly

Serve a cached or simpler result and say so. Far better than a spinner: users forgive a labelled downgrade and do not forgive silence.

Queue and defer

For work nobody is watching, accept it, persist it, and drain when capacity returns. Turns an outage into latency instead of an error.

What none of the three tolerate is finding out late. The gap between the first 503 and the moment someone notices is where the damage happens, and it is the one part of this that is entirely within your control — external monitoring closes it to seconds regardless of which of the three you chose.

Frequently Asked Questions

What does a 503 from the Together AI API actually mean?

It means the request was well-formed, your key was accepted, and the service declined to run it because it had no capacity to spare at that moment. Nothing about the request needs to change for it to succeed later — which is precisely what separates a 503 from a 400 or a 401, where retrying the identical payload can only produce the identical error. Read it as a scheduling outcome rather than a defect: the same JSON that failed at 14:02 will very often succeed at 14:03, and the engineering question is not what is wrong with the call but what your system does during the minute in between.

Is a 503 the same as a 429 rate limit?

No, and conflating them sends you to the wrong fix. A 429 is about you: your key exceeded an allowance that is written down somewhere, and the remedy is your own pacing, a higher tier, or a queue. A 503 is about the service: capacity was not available for anyone in that position, and your own consumption may have been entirely modest. The tell is what happens when you slow down. Reduce concurrency and 429s disappear almost immediately; 503s often continue at the same rate, because you were never the cause.

Should I retry a 503, and how?

Yes, but with a policy rather than a loop. Honour a Retry-After header when one is present, and otherwise back off exponentially with full jitter so that a fleet of clients does not synchronise and hit the service in waves. Cap total attempts against a deadline rather than a fixed count, so a request nobody is waiting for any more is abandoned instead of retried. And keep a retry budget expressed as a ratio of retries to successes, which collapses on its own when most calls are failing — the case where naive retries do the most damage.

What should I never retry after a 503?

Anything that may have already had an effect. A 503 usually means the work never started, but a 503 returned from an intermediary after the request was accepted is indistinguishable from one returned before, and a retried batch submission or long-running job can quietly become two. Any call that creates durable server-side state needs an idempotency key or a ledger you check before re-sending. For plain completions the risk is cost rather than correctness, but a retried streaming call that had already emitted tokens still bills for them.

How do I tell a Together AI 503 apart from a failure in my own infrastructure?

Take your application out of the path. Run one curl against the exact endpoint your code calls, from a machine outside your deployment, with a valid key. A 503 there is the provider. A success there while your app still fails means the failure is between your app and api.together.xyz — your proxy, your egress, your DNS, or a gateway of your own returning its own 503, which is a genuinely common confusion because the status code is identical and the body is not.

One Together AI model returns 503 while another works. Which is broken?

Neither the platform nor your integration — the specific model endpoint has no capacity available, which is the normal failure mode on a marketplace. The correct response depends on the model’s importance. For a long-tail model, hold a named second choice with the same interface and fail over to it, accepting the quality difference for the duration; for a model you cannot substitute, dedicated capacity is the only structural fix, since serverless allocation is by definition shared. Either way, record the model identifier in the error so the pattern is visible: teams routinely spend a week debugging a client that was fine, because their dashboard aggregated a healthy flagship and a starved long-tail endpoint into one error rate.

Related Together AI Guides

Find Out About the 503 Before Your Users Do

API Status Check watches Together AI and the rest of your stack from outside your infrastructure and alerts you the moment capacity errors start — so failover happens on your schedule, not after the first support ticket.

Start Your Free Trial →

Together AI API 503 Service Unavailable