Together AI Context Window Limits: Token Budgeting and History Trimming (2026)

Context bugs are the least dramatic failures in an LLM integration and the hardest to attribute. There is no stack trace, no non-2xx status, and often no error at all — just an answer that is subtly worse than it was last month, in a system where nobody expects the output to be deterministic in the first place. By the time someone files it, the change that caused it is dozens of deploys back.

The short version

The window holds your input and the model's output together, so reserve room for the answer before you decide how much history fits. Check the finish reason on every response, because a truncated answer arrives as a 200. And do not treat the advertised window as a target — filling it costs money, adds latency, and makes answers worse.

Where the Tokens Actually Go

Most teams account for one of these and are surprised by the rest. The window is shared by everything in the list below, and the items you did not write are usually the ones that grow.

Consumer	Grows with	Usually forgotten?
System prompt	Every feature request, forever	No — but its growth rate is
Conversation history	Session length	No
Retrieved documents	Corpus size and top-k	Yes — the top-k tuned on short docs
Tool and function definitions	Number of tools, schema verbosity	Yes — sent on every call, always
Tool results	Whatever your API returned	Yes — an unpaginated list is a bomb
The model's answer	max output tokens	Yes — and this is the expensive one

The last row is the one that causes production incidents. Input and output share the window, so a request that fits perfectly on the way in can leave the model a few hundred tokens to answer in. It will use them, stop mid-sentence, and return a 200.

Reserve the Answer First

The correct order of operations is not the intuitive one. Do not fill the window with context and then ask how much room is left for the answer — decide the answer's budget first, subtract it, and let history compete for what remains.

// Order matters. Output budget is a requirement, not a leftover.
const WINDOW = windowForModel('meta-llama/Llama-3.3-70B-Instruct-Turbo');
const MAX_OUTPUT = 1_500;        // what a complete answer actually needs
const SAFETY_MARGIN = Math.ceil(WINDOW * 0.05); // estimation error, chat scaffolding

const systemTokens = estimateTokens(SYSTEM_PROMPT);
const toolTokens = estimateTokens(JSON.stringify(TOOL_DEFINITIONS));

const historyBudget = WINDOW - MAX_OUTPUT - SAFETY_MARGIN - systemTokens - toolTokens;

if (historyBudget < MIN_VIABLE_HISTORY) {
  // The fixed costs no longer leave room for a conversation. That is a
  // design problem, not a runtime one — fail loudly rather than sending
  // a request that cannot produce a good answer.
  throw new ContextBudgetExhausted({ systemTokens, toolTokens, WINDOW });
}

const messages = trimToFit(history, historyBudget);

The explicit throw matters more than the arithmetic. Without it, a system prompt that grew past the point of viability degrades silently: every request still succeeds, every answer is slightly worse, and the graph that would have shown you is one nobody built.

Four Trimming Strategies and What Each One Loses

Strategy	What it loses	Use when
Sliding window (drop oldest)	The setup — constraints stated once at the start	Short, self-contained exchanges
Pinned head + sliding tail	The middle, abruptly and visibly	Default choice for most chat products
Rolling summary of the middle	Precision — a summary can quietly alter facts	Long sessions where continuity is the product
Retrieval over history	Recency — relevant-but-old beats recent	Sessions long enough that summarising also fails

Two rules survive across all four. Never trim to a message boundary that separates a tool call from its result — most APIs reject the orphan, and the ones that do not will produce something stranger. And never summarise text the user needs reproduced verbatim: identifiers, code, exact quotes. Those get pinned or dropped, never paraphrased.

Reading a Context Failure

Symptom	Most likely cause	Fix
Explicit context-length error	Total input exceeds the window	Trim before sending; do not retry unchanged
Answer stops mid-sentence, status 200	Output budget exhausted	Raise max output, or reserve it up front
Model ignores an early instruction	History trimmed away the system context	Pin the system prompt outside the trim window
Quality drops as sessions get longer	Signal diluted by accumulated filler	Trim more aggressively, not less
Works in dev, fails for one customer	Their documents or history are much larger	Test with p99 input size, not median
Cost per call rising without a traffic change	Prompt or top-k grew; nobody measured	Alert on p95 input tokens per request

Rows two through four are the reason this family of bug survives so long: all three return a200 with a well-formed body. Nothing in a conventional error budget or uptime monitor will ever fire on them. The only instrument that catches them is a per-request record of input tokens, output tokens, and finish reason.

The Together AI Trap: The Window Is a Property of the Model String

Together AI's appeal is the model zoo — you change one string and you are running a different model from a different family. The context window changes with it, and nothing in your code notices. The model identifier is usually a config value or an environment variable, which means the effective capacity of your system is set by a string that no test asserts on.

The resulting bug is seasonal. A prompt that has been comfortable for months starts failing the week someone swaps the default model for a cheaper one with a smaller window, and the error surfaces in a completely different part of the product to the change that caused it. If the swap happened via a config push rather than a deploy, there is no diff to find.

Bind the limit to the model rather than to the codebase. Keep a table mapping model identifier to window size, look the value up at call time, and fail loudly on an unknown identifier instead of falling back to a default that happens to be whatever the last model needed.

Make the window a lookup, and fail loudly on unknown models:

// The window belongs to the model, not to the codebase. Keep them together.
const MODEL_WINDOWS = {
  'meta-llama/Llama-3.3-70B-Instruct-Turbo': 128_000,
  // ...one entry per model you actually ship. Verify each against the
  // model's page on Together before adding it.
};

function windowFor(model) {
  const limit = MODEL_WINDOWS[model];
  if (!limit) {
    // Do NOT fall back to a default. An unknown model is a config bug,
    // and a silent default turns it into a truncation bug months later.
    throw new Error(`No context window registered for model: ${model}`);
  }
  return limit;
}

const budget = windowFor(process.env.TOGETHER_MODEL) - MAX_OUTPUT_TOKENS;

When the Problem Is Not Your Budget

There is a class of incident that looks exactly like a context regression and is not one. During upstream degradation, answers get shorter and vaguer, requests time out before completing, and long prompts fail more often than short ones because they take longer to process. Every symptom points at the prompt change that shipped that morning.

The question that separates the two cases cannot be answered from your own logs, because your logs only contain your traffic. It needs something outside the application making real requests to api.together.xyz on a fixed interval and recording latency and error rate independently — otherwise the first hour of every incident is spent bisecting a prompt that was never the problem.

Frequently Asked Questions

What happens when I exceed the Together AI context window?

One of two things, and they look nothing alike. If the whole request is over the limit, you get an explicit error before any generation happens — a fast, loud, honest failure that is easy to handle. The dangerous case is the request that fits on the way in but leaves too little room for the answer: generation starts, runs out of budget, and stops mid-sentence with a length finish reason. That comes back as a 200 with a plausible-looking body, so nothing in your error handling fires. Check the finish reason on every Together AI response, not just the status code.

How do I count tokens before sending a request to Together AI?

Exactly, using the tokenizer for the specific model, or approximately, using characters divided by a deliberately pessimistic ratio. What you should not do is count with a tokenizer borrowed from a different model family and treat the result as precise — the divergence is worst on code and non-English text, which is where prompts are longest. If the number drives a trimming decision, being wrong in the over-reserving direction costs you a little unused window; being wrong the other way ships a request that fails or silently truncates.

Should I summarise old conversation turns or just drop them?

Summarising preserves more of the thread but adds a call, adds latency, and adds a place for facts to be quietly altered — a summary is a lossy paraphrase generated by the same class of model that gets things wrong. Dropping is instant and honest about what it lost. The pragmatic split is to keep the system prompt and the most recent turns verbatim, summarise the middle only when the conversation is long enough that dropping it visibly breaks continuity, and never summarise anything the user typed that has to be reproduced exactly, such as an identifier or a snippet of code.

Why does a request that fits in the Together AI window still get a truncated answer?

Because the window holds the input and the output together. A prompt that consumes almost the entire window leaves the model a few hundred tokens to answer in, and it will use them and stop — mid-sentence, mid-object, or mid-list. This is the single most common context bug in production, and it does not look like a context bug: it looks like the model got lazy. Reserve the maximum output length up front, subtract it from the window, and only then decide how much history fits.

Does a bigger context window mean I should use it?

No, and treating the advertised number as a target is expensive in three separate ways. You pay for input tokens, so a full window costs more per call. Time-to-first-token grows with input length, so a full window is slower. And retrieval quality does not scale with volume — padding a prompt with marginally relevant material reliably makes answers worse, not better, because the relevant part is now a smaller fraction of what the model is attending to. The window is a ceiling to stay under, not a budget to spend.

Do all Together AI models have the same context window?

No — the window is a property of the individual model, and Together serves models from many families with very different limits. This matters more on Together than on a single-model provider because swapping models is a one-string change that is often made for cost or speed reasons by someone who is not thinking about prompt capacity. Keep an explicit map from model identifier to window size in your code, look it up at call time, and throw on an unregistered identifier so that adding a model forces a conscious decision about how much context it can hold.

Related Guides

A Truncated Answer and an Outage Both Return 200

Your monitoring cannot tell a context regression from Together AI having a bad hour, because neither one changes a status code. API Status Check probes api.together.xyz and the rest of your stack independently and alerts on errors and latency, so the first question of every incident already has an answer.

Start Your Free Trial →