Groq Context Window Limits: Token Budgeting and History Trimming (2026)

Context bugs are the least dramatic failures in an LLM integration and the hardest to attribute. There is no stack trace, no non-2xx status, and often no error at all — just an answer that is subtly worse than it was last month, in a system where nobody expects the output to be deterministic in the first place. By the time someone files it, the change that caused it is dozens of deploys back.

The short version

The window holds your input and the model's output together, so reserve room for the answer before you decide how much history fits. Check the finish reason on every response, because a truncated answer arrives as a 200. And do not treat the advertised window as a target — filling it costs money, adds latency, and makes answers worse.

Where the Tokens Actually Go

Most teams account for one of these and are surprised by the rest. The window is shared by everything in the list below, and the items you did not write are usually the ones that grow.

Consumer	Grows with	Usually forgotten?
System prompt	Every feature request, forever	No — but its growth rate is
Conversation history	Session length	No
Retrieved documents	Corpus size and top-k	Yes — the top-k tuned on short docs
Tool and function definitions	Number of tools, schema verbosity	Yes — sent on every call, always
Tool results	Whatever your API returned	Yes — an unpaginated list is a bomb
The model's answer	max output tokens	Yes — and this is the expensive one

The last row is the one that causes production incidents. Input and output share the window, so a request that fits perfectly on the way in can leave the model a few hundred tokens to answer in. It will use them, stop mid-sentence, and return a 200.

Reserve the Answer First

The correct order of operations is not the intuitive one. Do not fill the window with context and then ask how much room is left for the answer — decide the answer's budget first, subtract it, and let history compete for what remains.

// Order matters. Output budget is a requirement, not a leftover.
const WINDOW = windowForModel('llama-3.3-70b-versatile');
const MAX_OUTPUT = 1_500;        // what a complete answer actually needs
const SAFETY_MARGIN = Math.ceil(WINDOW * 0.05); // estimation error, chat scaffolding

const systemTokens = estimateTokens(SYSTEM_PROMPT);
const toolTokens = estimateTokens(JSON.stringify(TOOL_DEFINITIONS));

const historyBudget = WINDOW - MAX_OUTPUT - SAFETY_MARGIN - systemTokens - toolTokens;

if (historyBudget < MIN_VIABLE_HISTORY) {
  // The fixed costs no longer leave room for a conversation. That is a
  // design problem, not a runtime one — fail loudly rather than sending
  // a request that cannot produce a good answer.
  throw new ContextBudgetExhausted({ systemTokens, toolTokens, WINDOW });
}

const messages = trimToFit(history, historyBudget);

The explicit throw matters more than the arithmetic. Without it, a system prompt that grew past the point of viability degrades silently: every request still succeeds, every answer is slightly worse, and the graph that would have shown you is one nobody built.

Four Trimming Strategies and What Each One Loses

Strategy	What it loses	Use when
Sliding window (drop oldest)	The setup — constraints stated once at the start	Short, self-contained exchanges
Pinned head + sliding tail	The middle, abruptly and visibly	Default choice for most chat products
Rolling summary of the middle	Precision — a summary can quietly alter facts	Long sessions where continuity is the product
Retrieval over history	Recency — relevant-but-old beats recent	Sessions long enough that summarising also fails

Two rules survive across all four. Never trim to a message boundary that separates a tool call from its result — most APIs reject the orphan, and the ones that do not will produce something stranger. And never summarise text the user needs reproduced verbatim: identifiers, code, exact quotes. Those get pinned or dropped, never paraphrased.

Reading a Context Failure

Symptom	Most likely cause	Fix
Explicit context-length error	Total input exceeds the window	Trim before sending; do not retry unchanged
Answer stops mid-sentence, status 200	Output budget exhausted	Raise max output, or reserve it up front
Model ignores an early instruction	History trimmed away the system context	Pin the system prompt outside the trim window
Quality drops as sessions get longer	Signal diluted by accumulated filler	Trim more aggressively, not less
Works in dev, fails for one customer	Their documents or history are much larger	Test with p99 input size, not median
Cost per call rising without a traffic change	Prompt or top-k grew; nobody measured	Alert on p95 input tokens per request

Rows two through four are the reason this family of bug survives so long: all three return a200 with a well-formed body. Nothing in a conventional error budget or uptime monitor will ever fire on them. The only instrument that catches them is a per-request record of input tokens, output tokens, and finish reason.

The Groq Trap: Context Length Is a Rate Limit, Not Just a Ceiling

Everywhere else, the context window is a wall you hit at the end of a long conversation. On Groq it is also a throttle you hit at the start of a busy minute, because Groq meters tokens per minute alongside requests per minute. Doubling the size of your system prompt does not just move you closer to the model's ceiling — it halves the number of requests you can make before the account 429s.

That coupling produces an incident shape that reads as an outage and is not one. Someone adds four few-shot examples to a prompt on Tuesday. Nothing breaks in review, because a single request with a bigger prompt works fine. On Thursday, at the hour when traffic peaks, the same code starts returning 429s in bursts — and the change that caused it was a prompt edit, not a traffic increase, so nobody looks at it.

The defence is to treat prompt size as a capacity change requiring the same scrutiny as a concurrency change. Log input tokens per request, alert on the p95 rather than the mean, and when a prompt grows, work out what it does to your requests-per-minute headroom before it ships rather than after.

Track prompt growth as a capacity metric, not a prompt detail:

const res = await client.chat.completions.create({
  model: 'llama-3.3-70b-versatile',
  messages,
});

// usage.prompt_tokens is the number that governs your TPM headroom.
// Emit it on EVERY call, not just when something fails.
metrics.histogram('groq.prompt_tokens', res.usage.prompt_tokens);
metrics.histogram('groq.completion_tokens', res.usage.completion_tokens);

// A prompt-size regression is a capacity regression. Catch it in CI, once.
if (res.usage.prompt_tokens > PROMPT_TOKEN_BUDGET) {
  logger.warn('prompt budget exceeded', {
    actual: res.usage.prompt_tokens,
    budget: PROMPT_TOKEN_BUDGET,
  });
}

When the Problem Is Not Your Budget

There is a class of incident that looks exactly like a context regression and is not one. During upstream degradation, answers get shorter and vaguer, requests time out before completing, and long prompts fail more often than short ones because they take longer to process. Every symptom points at the prompt change that shipped that morning.

The question that separates the two cases cannot be answered from your own logs, because your logs only contain your traffic. It needs something outside the application making real requests to api.groq.com on a fixed interval and recording latency and error rate independently — otherwise the first hour of every incident is spent bisecting a prompt that was never the problem.

Frequently Asked Questions

What happens when I exceed the Groq context window?

One of two things, and they look nothing alike. If the whole request is over the limit, you get an explicit error before any generation happens — a fast, loud, honest failure that is easy to handle. The dangerous case is the request that fits on the way in but leaves too little room for the answer: generation starts, runs out of budget, and stops mid-sentence with a length finish reason. That comes back as a 200 with a plausible-looking body, so nothing in your error handling fires. Check the finish reason on every Groq response, not just the status code.

How do I count tokens before sending a request to Groq?

Exactly, using the tokenizer for the specific model, or approximately, using characters divided by a deliberately pessimistic ratio. What you should not do is count with a tokenizer borrowed from a different model family and treat the result as precise — the divergence is worst on code and non-English text, which is where prompts are longest. If the number drives a trimming decision, being wrong in the over-reserving direction costs you a little unused window; being wrong the other way ships a request that fails or silently truncates.

Should I summarise old conversation turns or just drop them?

Summarising preserves more of the thread but adds a call, adds latency, and adds a place for facts to be quietly altered — a summary is a lossy paraphrase generated by the same class of model that gets things wrong. Dropping is instant and honest about what it lost. The pragmatic split is to keep the system prompt and the most recent turns verbatim, summarise the middle only when the conversation is long enough that dropping it visibly breaks continuity, and never summarise anything the user typed that has to be reproduced exactly, such as an identifier or a snippet of code.

Why does a request that fits in the Groq window still get a truncated answer?

Because the window holds the input and the output together. A prompt that consumes almost the entire window leaves the model a few hundred tokens to answer in, and it will use them and stop — mid-sentence, mid-object, or mid-list. This is the single most common context bug in production, and it does not look like a context bug: it looks like the model got lazy. Reserve the maximum output length up front, subtract it from the window, and only then decide how much history fits.

Does a bigger context window mean I should use it?

No, and treating the advertised number as a target is expensive in three separate ways. You pay for input tokens, so a full window costs more per call. Time-to-first-token grows with input length, so a full window is slower. And retrieval quality does not scale with volume — padding a prompt with marginally relevant material reliably makes answers worse, not better, because the relevant part is now a smaller fraction of what the model is attending to. The window is a ceiling to stay under, not a budget to spend.

Why did my Groq requests start rate limiting after a prompt change?

Because Groq meters tokens per minute in addition to requests per minute, and a longer prompt spends the token budget faster at identical request volume. If a prompt grew and 429s appeared without any traffic increase, that is the most likely cause and it is easy to confirm: compare prompt_tokens on the usage object before and after the change, multiply by your peak requests per minute, and see whether the product crossed your tier's limit. The fix is either a shorter prompt, a higher tier, or client-side concurrency limiting — not a retry loop, which makes the contention worse.

Related Guides

A Truncated Answer and an Outage Both Return 200

Your monitoring cannot tell a context regression from Groq having a bad hour, because neither one changes a status code. API Status Check probes api.groq.com and the rest of your stack independently and alerts on errors and latency, so the first question of every incident already has an answer.

Start Your Free Trial →