Together AI API 504 Gateway Timeout: Which Timer Expired, and Whose (2026)

A 504 is a statement about a clock, not about a failure. Some intermediary decided in advance how long it was willing to wait, the answer did not arrive inside that window, and the intermediary wrote an error of its own. Nothing in the status code tells you that Together AI failed. It tells you that a deadline expired, and deadlines are configuration — which means the fault is findable by reading a number rather than by guessing.

Whose clock is the whole question. It might be Together AI's edge. It might just as easily be your ingress controller, a service-mesh sidecar, a corporate egress proxy, a platform request limit, or the timeout argument you passed to your own SDK three releases ago and have not looked at since. These produce an identical status code and entirely different remediation.

30-second triage: read the elapsed time, not the status code. If your 504s cluster on a round number — 30s, 60s, 120s — that number is somebody's configured constant and your job is to find whose. If the elapsed times are scattered, something variable is being waited on. Either way, checklive Together AI statusfirst so you know whether any of this analysis is worth doing.

504 against the rest of the 5xx family

These five get filed together as “the API is broken” and then handed one retry policy, which guarantees the policy is wrong for at least three of them. The last column is the one that matters, because it is the only one that changes what you do next.

Code	Who authored it	Correct response
`429`	Together AI, deliberately, about your allowance	Slow down. Never fail over — you export your own pacing bug
`500`	The service, about an unhandled fault	Retry once or twice; a persistent 500 is a bug report
`502`	An intermediary, about an unusable reply	Identify the hop from the body, then one jittered retry
`503`	The service, about having no capacity	Retry with a budget, then fail over or degrade
`504`	An intermediary, about a deadline it set	Find the timer from elapsed time; retry only if idempotent

502 and 504 are the two gateway-authored codes and they are routinely conflated. The difference is simple and it changes your diagnosis: a 502 means the gateway got an answer it could not use, so the evidence is in the response body; a 504 means it got nothing and stopped waiting, so the evidence is in the elapsed time. The unusable-reply case is covered in theTogether AI 502 guide, the capacity case in theTogether AI 503 guide, and throttling in theTogether AI 429 guide.

📡

Recommended

Find Out Whose Timer Expired

External checks run from outside your own network on a known deadline, so a timeout inside your infrastructure looks different from one at the provider edge — instead of identical, which is what your application logs show you.

Try Better Stack Free →

Why Together AI returns 504 specifically: cold starts are per-model

Together AI's defining characteristic is catalogue breadth — hundreds of models available behind one API surface. That breadth is only economical because not every model is resident on hardware at every moment. A request for a model that is not currently loaded has to wait for it to be loaded, and loading a large model is measured in tens of seconds, not milliseconds.

This produces a failure signature that looks nothing like an outage and is constantly mistaken for one. It is per-model: the popular model your colleague is testing answers instantly while the specialised one your job depends on times out repeatedly. It is bursty in an unusual direction — the first request after a quiet period fails, and requests immediately afterwards succeed, because the first one paid the loading cost. And it is completely invisible on a status page, because nothing is broken.

The corollary is a diagnostic rule that is specific to this provider and worth internalising: never probe liveness with a different model id than the one that failed. A curl against a popular model proving the API is up proves nothing whatsoever about the model your production traffic uses. Capacity, residency and therefore latency are per-model here, and a probe that changes the model id has changed the experiment.

Sample rather than probe once, and record all three columns. One request cannot tell a total failure apart from the far more common partial one:

for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s %{content_type}\n" \
    --max-time 90 \
    https://api.together.xyz/v1/chat/completions \
    -H "Authorization: Bearer $TOGETHER_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"meta-llama/Llama-3.3-70B-Instruct-Turbo","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'
done

The status code says whether it failed, the elapsed time says which timer expired, and the content type says who wrote the error. Set --max-time deliberately higher than the deadline you are investigating, otherwise curl becomes the shortest timer in the path and you measure your own probe instead of the system.

The Together AI trap: your health check tests a model you do not use

The standard health check — hit the API with a cheap known-good model and assert a 200 — is actively misleading on Together AI. It will stay green through an incident that is destroying your production path, because the model it probes is popular enough to be permanently resident while the one you depend on is not. Point the probe at the exact model id your application calls, and keep it warm as a side effect.

Prove the cold-start hypothesis with two requests rather than one. Send the failing request, let it time out, then send it again immediately. If the second attempt returns quickly, the first one paid for a load and you have your answer: this is residency, not an outage, and no amount of provider failover addresses it. If both time out identically, look elsewhere.

The two real fixes are keeping the model warm with a low-rate heartbeat so it never falls out of residency, and setting a deadline for the first request of a cold path that is long enough to absorb a load. Dedicated endpoints remove the problem entirely by keeping capacity reserved, which is the honest answer when a model is genuinely load-bearing for you.

The streaming case, where the 504 stops happening at all

Streaming is the most underrated fix for 504s, and it works for a reason that is easy to miss: most gateway timeouts are idle timeouts rather than total-duration limits. They fire when no bytes have moved for N seconds, not when the request has been open for N seconds. A streamed response puts bytes on the wire as soon as the first tokens exist, which resets the idle timer continuously and keeps a long generation alive under a deadline that would have killed the buffered equivalent.

Check which kind of timeout you have. An idle timeout is defeated by streaming; a hard request-duration cap is not. They are configured separately and confusing them wastes a day.
Assert on the terminal event. An iterator finishing is not the same as a stream completing. If the terminal marker never arrived, the response was truncated no matter what the status code said.
Count partial responses as their own class. Not a success, not a 5xx. On a streamed path they are the metric that moves first, and a status-code-only dashboard stays flat through the entire incident.
Decide the resume policy in advance. Re-issuing the whole prompt is the honest default and it bills twice; showing the partial output with a visible label is often the better product answer.

A retry policy that fits a 504 specifically

504 is the 5xx where the standard retry advice is most likely to hurt you, because the one thing a 504 does not tell you is whether the work happened. Four rules keep the retry from becoming its own incident:

Assume the request may have succeeded. Your gateway stopped listening; the upstream did not necessarily stop working. Send an idempotency key for anything with a side effect, and expect a retried completion to be billable twice.
Do not retry into the same deadline. If the timeout was too short for this request, a second identical attempt will expire identically. Change something — stream it, shrink it, or route it to a path with a longer deadline — or do not retry at all.
Budget by ratio, not by count. Allow retries as a fraction of recent successes so the allowance collapses on its own when almost everything is failing, with no deploy required. See the Together AI retry budget guide.
Retry against a deadline, not a counter. If the user-facing request gave up eight seconds ago, attempt three is load on behalf of nobody. Propagate the deadline and skip attempts that cannot finish inside it.

If 504s persist past a couple of attempts they have stopped being transient, and the failover machinery should take over — which only works if it was kept warm. TheTogether AI failover guidecovers keeping a second path tested rather than merely configured, and thecircuit breaker guidecovers stopping the retries automatically.

Frequently Asked Questions

What does a 504 from the Together AI API actually mean?

It means an intermediary between your process and the model set a deadline, that deadline expired before a usable response arrived, and the intermediary wrote its own error rather than waiting longer. Two things follow, and both are frequently missed. First, the 504 was authored by whichever hop was holding the timer — which may be Together AI's edge, but may equally be your own load balancer, service mesh, egress proxy or SDK client. Second, a 504 says nothing about whether the upstream completed the work; it says only that the answer did not come back inside the window. That distinction is what makes the retry decision non-obvious.

How is a 504 different from a 502 on Together AI?

They are siblings — both are written by intermediaries, neither is authored by the component that actually failed — but they describe opposite events. A 502 means the gateway received something it could not use: an empty body, a reset connection, a malformed reply. A 504 means the gateway received nothing at all and gave up waiting. The practical difference is what you measure next. For a 502 the diagnostic signal is the response body and its content type, because that identifies the author. For a 504 the diagnostic signal is elapsed time, because the elapsed time tells you which timer expired, and timers are configuration you can find and read.

Is a 504 the same as Together AI being down?

Usually not, and the two call for opposite responses. A real outage is broad, sustained and consistent — every endpoint, every key, every request shape. A 504 is characteristically narrow: it hits the slow requests, the large requests, or the requests that traverse one particular hop, while everything else continues to work. Sample rather than infer from a single failure. Ten identical probes failing over several minutes is an incident and failover is correct; three of ten failing while the fast requests sail through is a deadline set too tight for part of your traffic, and failing over exports the problem rather than solving it.

Is it safe to retry a Together AI 504?

Less safe than a 502, and the reason is worth understanding rather than memorising. A 504 means the answer did not arrive in time — it does not mean the upstream never did the work. The request may have completed a moment after your gateway stopped listening, which for a completion is a billing question and for anything with a side effect is a correctness one. Retry only what is idempotent, or propagate an idempotency key so the second attempt is provably the same operation rather than a second one. And retry against a deadline rather than a fixed count: if the user-facing request gave up already, attempt three is load on behalf of nobody.

Why do my Together AI 504s show up as client-side timeout exceptions instead of status codes?

Because whichever timer is shortest wins, and it is often yours. If your SDK's client timeout is below the gateway's, your process abandons the request first and raises a local exception — the 504 that the gateway would eventually have written never reaches you, and your status-code dashboard stays clean through the whole incident. This is why the elapsed-time distribution matters more than the error taxonomy here. Log total elapsed time on every failure, timeouts included, and count local timeouts in the same bucket as 504s; otherwise you are measuring which timer fired rather than how often requests failed.

Why does the same Together AI request fail once and then succeed immediately after?

That is the cold-start signature, and it is close to diagnostic. Together AI serves a very large catalogue, which means not every model is resident on hardware at all times. The first request for a model that is not loaded triggers a load that takes tens of seconds; your deadline expires partway through and you get a 504. By the time you retry, the load has completed and the model answers normally. Nothing was down, and nothing on a status page will ever reflect it. If a retry-immediately test reproduces this pattern, the fix is a warming heartbeat or a dedicated endpoint, not failover.

Can I use a different model to check whether Together AI is up?

No, and this is the single most common diagnostic mistake made against this provider. Capacity and residency are per-model, so a 200 from a popular model tells you the API surface is reachable and tells you nothing at all about the model your traffic uses. A health check written that way will report green for the entire duration of an incident that is failing every production request. Always probe with the exact model id that failed — if the probe passes with a substitute and fails with the real one, you have already learned the answer.

Related Together AI Guides

Know Whose Deadline Expired — Before Your Users Do

API Status Check watches Together AI and the rest of your stack from outside your own network on a known deadline, so you find out whether the timer that fired was theirs or yours — in seconds, not after twenty minutes of reading a status page that was never going to mention it.

Start Your Free Trial →

Together AI API 504 Gateway Timeout

Live Together AI status right now