Together AI API 502 Bad Gateway

A 502 is the only 5xx that is definitionally written by someone other than the component that failed. A gateway asked an upstream for a response, got something it could not use — an empty body, a reset connection, a malformed reply — and substituted an error of its own. Everything useful about a 502 follows from that one fact: your request was fine, the model may never have seen it, and the error you are reading was authored by a middleman.

Which middleman is the entire question. It might be Together AI's own edge. It might equally be your ingress controller, a service-mesh sidecar, a corporate egress proxy, a CDN, or a routing library sitting between your code and the provider. These produce an identical status code and wildly different remediation, and teams routinely spend an incident refreshing a status page for a failure their own infrastructure emitted.

502 against the rest of the 5xx family

These five get filed together as “the API is broken” and then handed one retry policy, which guarantees that the policy is wrong for at least three of them. The column that matters is the last one, because it is the only one that changes what you do next.

502 and 504 are siblings and both are written by gateways — the difference is that a 504 means the intermediary waited and gave up, while a 502 means it got an answer it could not use. Neither tells you the upstream failed. The capacity case is covered separately in the Together AI 503 guide, and the throttling case in the Together AI 429 guide.

Why Together AI returns 502 specifically

One hostname, hundreds of models, and each model is served by its own pool behind a shared gateway. That architecture produces a 502 shape you will not see at a single-model provider: the gateway is perfectly healthy, your credential is fine, the flagship model returns 200 in eighty milliseconds — and the one long-tail model your feature depends on returns 502 because its pool has no healthy backend to route to right now.

Cold and scaled-to-zero pools are the usual mechanism. A rarely-called model may have no warm backend at the instant you ask, and depending on where the failure lands you get a 502 rather than a queue. This is why "is Together AI down" is nearly always the wrong question. The right one is whether the exact model id you asked for is serving.

Sample rather than probe once. One request cannot distinguish a total failure from the far more common partial one, and the two call for different responses:

for i in $(seq 1 10); do
  curl -s -o /tmp/body -w "%{http_code} %{time_total}s %{content_type}\n" \
    https://api.together.xyz/v1/chat/completions \
    -H "Authorization: Bearer $TOGETHER_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"meta-llama/Llama-3.3-70B-Instruct-Turbo","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'
done

Three columns, three answers. The status code tells you whether it failed, the elapsed time tells you whether something waited before failing, and the content type tells you who wrote the error. Ten out of ten failing with a JSON body is an incident. Two out of ten failing with an HTML body is your own infrastructure having a moment, and no amount of provider failover will fix it.

The Together AI trap: never probe liveness with a different model than the one that failed

The single most common misdiagnosis on this platform is a health check that calls a cheap flagship model, gets 200, and reports Together AI healthy while the model your product actually uses has been 502ing for twenty minutes. Capacity is per-model. A probe against a different model id is not a probe of your dependency, it is a probe of an unrelated one that happens to share a hostname.

Pin the health check to the exact model string in your production config, and label every metric and alert by model id rather than by provider. If you run a capability-ordered fallback chain, each tier needs its own probe too — otherwise a chain that looks three-deep on paper can quietly be one-deep in reality, because tiers two and three have not served a successful request in a week and nobody has been watching.

The streaming case, where the 502 never reaches your status-code metrics

If you stream, a large share of your gateway failures will never be counted as 502s at all. The response headers arrived long ago with a 200; the failure happens partway through the token stream, and what your client receives is a truncated event or an HTML error fragment spliced into a channel that was expecting server-sent events. Most SDKs raise that as a decoding error, so it lands in your logs as a parse failure in the JSON layer and your Together AI error-rate panel stays flat through the entire incident.

• Assert on the terminal event. An iterator finishing is not the same as a stream completing. If the terminal marker never arrived, the response was truncated, whatever the status code said.
• Count partial responses as their own class. Not a success, not a 5xx. They behave differently and they are the metric that moves first.
• Record bytes and tokens received on failure. A stream that died at token three is a different problem from one that died at token three hundred, and only one of them is worth showing the user.
• Decide the resume policy in advance. Re-issuing the whole prompt is the honest default and it bills twice; showing the partial output with a visible label is often the better product answer.

A retry policy that fits a 502 specifically

502s are the most retry-friendly 5xx there is — they are usually brief, usually one hop, and usually gone by the time you look. That makes blanket “never retry 5xx” advice actively wrong here. Four rules keep the retry from becoming its own incident:

• One quick jittered retry first. Randomise across the full window rather than adding a small wobble, so a fleet of clients does not resynchronise and re-burst against the first recovered backend.
• Assume the work may have happened. The gateway could not read the response; that does not mean the upstream never produced one. Send an idempotency key for anything with a side effect, and expect that a retried completion can bill twice.
• Budget by ratio, not by count. Allow retries as a fraction of recent successes so the allowance collapses on its own when almost everything is failing — no deploy required. See the Together AI retry budget guide.
• Retry against a deadline. If the user-facing request gave up eight seconds ago, attempt three is load on behalf of nobody. Propagate the deadline and skip attempts that cannot finish inside it.

If 502s persist past a handful of retries, they have stopped being transient and the failover machinery should take over — which only works if it was kept warm. The Together AI failover guide covers keeping a second path tested rather than merely configured, and the circuit breaker guide covers stopping the retries automatically.

Frequently Asked Questions

Related Together AI Guides

• Together AI API 503 Service Unavailable
• Together AI API 429 Too Many Requests
• Together AI API 500 Internal Server Error
• Together AI API Timeout Errors
• Together AI API Error Codes Reference
• Together AI Streaming Errors Guide
• Together AI API Monitoring Guide