Together AI Batch Inference: The 50% Discount and the Rows It Quietly Costs You (2026)

Together AI runs a large catalogue of open models, and for anything that does not need an answer inside a request timeout it offers those models at roughly half price through batch inference. You submit a file of requests, receive a job id, and the whole set completes within a window measured in hours. For embedding a corpus, classifying a backlog, or re-scoring a table overnight, that is an obvious win and the savings are genuine.

The trap is that batch fails in a register your instincts are not tuned to. The synchronous API fails loudly — an exception, a status code, a line to point at. Batch fails quietly. A job reaches completed, you download the output, and it has fewer rows than you submitted. Nothing raised an error. No metric moved. The missing rows are not an incident; they are the ordinary behaviour of a system where every request inside the batch succeeds or fails on its own, independent of the job that carried it.

This guide is about the distance between the job finished and every input got an answer. That distance is where the batch discount is kept or silently handed back, and closing it is a matter of how you submit and reconcile, not how you call the model.

What Batch Actually Guarantees

Read these guarantees literally. The gap between what batch promises and what teams assume it promises is precisely the set of failures you will otherwise spend a quarter chasing.

Property	What you actually get	What that forces you to build
Completeness	The job finishes; rows can fail independently	Reconcile returned ids against submitted ids every job. The status is not the receipt.
Latency	A window in hours, no position inside it	Completion is an event, never a deadline. Keep a synchronous path for urgent rows.
Ordering	Keyed by your custom id, not submission order	Stable id on every request. Join on it; never match by row position.
Pricing	~50% off successfully processed tokens	Measure delivered-token cost. Re-runs and synchronous retries erode the discount.
Result window	Output retained only for a limited period	Fetch and persist promptly. The provider is not your archive.

None of this is unique to Together AI — it is the standard contract of batch inference everywhere — but the synchronous API trained you to expect guarantees this one does not make.

The Lifecycle, and Where Each Stage Loses Rows

A batch passes through four stages, and every one has a signature failure that reports success while dropping data.

Upload. You send a file of requests, one per line, each carrying a custom id. The failure is one malformed line taking down more than itself, or an encoding slip that misaligns your ids. Validate locally first: count the lines, assert every id is present and unique. A bad input file is far cheaper to catch on your side than to reverse-engineer from a job that processed most of the rows and errored on the rest.

Submit. You create the job against the uploaded file and get a job id back. The failure is losing that id — a crash, or a submit that succeeds while its response never reaches you — leaving a job running that nothing is waiting for. Persist the job id transactionally the instant you have it, and store the submitted id set beside it so you know exactly what this job owes you.

Wait. The job queues where you cannot see it. The failure is treating this as bounded: blocking a request on it, or giving up and re-submitting a job that was only slow, which doubles the tokens you pay for. Poll on a schedule, list your non-terminal jobs, and advance state from the authoritative job object rather than from a push you may never receive.

Retrieve and reconcile. The job is done and you pull the output. This is the stage teams under-build every time. Key the results by your custom id and compute the set difference against the ids you stored at submit. The rows in that gap are your failures — send them to a synchronous retry, a dead-letter table, or a human, but send them somewhere. An unreconciled output on a completed job is exactly how rows vanish without a trace.

The Custom ID Is Non-Negotiable

The one habit that separates a batch pipeline that loses data from one that does not: every request carries a custom id you can join straight back to your own records, and every downstream decision keys off it.

Outputs do not return in submission order. They interleave, some rows are absent, and a partial job hands back a subset. Match by line position and the first missing row shifts every match after it by one — you will write plausible-looking outputs against the wrong records for the whole tail of the file, which is worse than losing them because it is silent and wrong rather than merely absent. With a stable id, matching is a join, missing rows announce themselves, duplicates collapse, and a re-run of only the failures slots back into the same keyspace.

When Not to Use Batch

The discount tempts teams into pushing the wrong work through batch. It is wrong whenever a person or a request is waiting, because you cannot promise when the answer lands. It is wrong for small volumes, where upload, poll and reconcile overhead swamps the savings on a few hundred rows. And it is wrong as the only path, because a Together AI incident on the batch queue is nearly invisible from inside your own telemetry — no errors, no latency spike, just a queue that stops draining and a graph that flattens into what reads as a slow afternoon. Keep a synchronous path for the rows that cannot wait, and monitor api.together.xyz independently so a stuck queue reaches you as an alert.

What to Instrument

Five numbers make batch legible: the age of your oldest non-terminal job against the promised window; the submitted-versus-returned row count per job; the count of rows routed to synchronous retry because batch dropped them; the delivered-token cost per batch, so you know the real discount rather than the quoted one; and the count of jobs re-submitted because a slow one was mistaken for a dead one. The first two catch silent loss. The last three tell you whether the discount is money in the bank or an accounting fiction your retries already spent.

What none of them can tell you is whether api.together.xyz was degraded while your queue sat quiet — that failure is invisible from inside a system that was waiting rather than calling, and it is exactly the gap external monitoring closes.

Frequently Asked Questions

How much cheaper is Together AI batch inference?

The batch path prices tokens at roughly half the synchronous rate, and that discount is genuine, but it is the ceiling rather than the outcome. You keep the full 50% only when a job runs once, returns every submitted row, and needs no synchronous cleanup. Rows that error still get re-served at full price, jobs you re-submit because you thought they had stalled double their tokens, and results you fetch twice cost you the fetch. Budget against delivered-token cost -- what you actually paid to get usable answers -- not against the quoted per-token line, because the two diverge exactly as much as your failure and retry rate.

How long until a Together AI batch job completes?

You get a completion window measured in hours, not a position inside it, and that is the whole trade you accepted for the discount. The identical batch can finish quickly one afternoon and near its deadline the next, driven by queue depth you have no visibility into. Every consumer of the results must treat completion as an event that lands whenever it lands, never as a scheduled deadline. Any row that a user or a request timeout is waiting on does not belong in a batch -- send those synchronously and pay full price for the handful that genuinely cannot wait.

My Together AI batch says completed but some outputs are missing. Why?

Because completed describes the job, not every request inside it. Rows fail independently -- a malformed line, an input over a length limit, a transient error on one request -- and the job still reaches a terminal success state with those rows absent from the output or shunted to an error file. Code that assumes a completed job means every input has a matching output will drop those failures silently, and the loss surfaces weeks later as a customer asking where their record went. The fix is structural: reconcile the returned ids against the submitted ids on every job and route the difference somewhere deliberate.

Do batch results come back in the order I submitted them?

No, and building on the assumption that they do is the fastest way to corrupt your data quietly. Outputs are keyed by the custom id you attach to each request, and they can arrive interleaved or with rows missing entirely. If you match answers to inputs by line position, the first absent row shifts every subsequent match by one and you write correct-looking outputs against the wrong records for the entire tail of the file. Attach a stable id to every request and join on it; position is not a key you can rely on.

Should I poll for batch completion or rely on a webhook?

Make polling the source of truth and treat any push as an accelerator. A scheduled sweep that lists your non-terminal jobs and reads their current state from the API keeps working when a callback is dropped, when your receiver was mid-deploy, or when the submitting process is not the one waiting. Build the poll first, make advancing a job to done idempotent so a duplicate signal is harmless, and let a webhook merely mean the poll notices sooner. A system that only learns of completion through a push has a single point of silent failure.

Is a slow batch the same as a provider outage?

From inside your own metrics they are indistinguishable, which is exactly why batch outages are dangerous. A job stuck behind a Together AI incident and a job merely far back in a healthy queue both look the same: not done, no error raised, no latency percentile moving, because you are waiting rather than calling. The only internal signal is the age of your oldest non-terminal job against the window you were promised. When that crosses the line, treat it as stuck, fail the time-sensitive portion over to synchronous calls, and lean on independent monitoring of api.together.xyz to tell you which of the two it actually is.

Related Guides

A Stalled Batch Queue Looks Exactly Like a Quiet Day

When the batch path degrades, your telemetry goes quiet rather than red — no errors, no latency spike, just jobs that never reach done. API Status Check probes api.together.xyz independently and alerts on errors and latency, so you find out from a monitor instead of from a customer asking where their results went.

Start Your Free Trial →