Counting Groq Tokens Before You Send (2026): TPM Ceilings, Tokenizers and Usage

Q: Can I use tiktoken to count Groq tokens?

No, and this is the most common mistake because the API is OpenAI-shaped. Groq serves open-weight models, each of which ships its own tokenizer vocabulary, and none of those vocabularies is the one tiktoken implements. Using tiktoken against a Llama or Qwen model produces a number that is wrong in a direction and magnitude that varies with the content, and it fails silently because nothing rejects a bad estimate. Load the model repository tokenizer instead.

Q: Why does my Groq token count drift after a model update?

Because your count is tied to a tokenizer, and Groq rotates its hosted model catalogue faster than most inference providers. When a model version is retired and you repoint to its successor, the successor may carry a different vocabulary, so identical text now tokenizes to a different number. Pin the tokenizer to the exact model string you send, and treat a model swap as a change that invalidates every cached token count you hold.

Q: Are Groq rate limits based on requests or tokens?

Both, and the token ceiling is the one that usually binds first for real workloads. Groq publishes per-minute limits on requests and on tokens, metered separately per model, and long-context traffic exhausts the token budget long before it approaches the request budget. This is why pre-send counting matters more on Groq than on request-metered providers: a single oversized prompt can consume a large share of a minute allowance that dozens of small requests would not have touched.

Q: Do output tokens count against the Groq TPM limit?

Plan on the assumption that they do, because the part you cannot count in advance is precisely the part you cannot control. You know the prompt exactly before you send it; you know the completion only by the ceiling you set with max_tokens. Budget the sum of the counted prompt and the max_tokens you allow, and set max_tokens deliberately rather than leaving it unbounded — an unbounded completion is an unbounded claim on your own minute.

Every team that scales on Groq eventually discovers that the token count is not a billing curiosity but a control input. Groq’s allowance is expressed per minute and denominated in tokens, which means the question “how many tokens is this request?” has to be answered before the request leaves your process, not after it comes back. The API answers it afterwards. That gap — between when you need the number and when Groq gives it to you — is the entire subject of this guide.

It is a gap that is easy to miss, because Groq speaks the OpenAI dialect fluently. The request shape is familiar, the response carries a usage object in the same place, and an existing OpenAI client points at Groq with a base-URL change. That fluency is exactly what makes the token problem sneaky: the code that counted tokens correctly against OpenAI keeps running against Groq, keeps returning numbers, and those numbers are now wrong.

Why tiktoken Is the Wrong Tool Here

Tokenization is not a general property of text. It is a property of a specific model’s vocabulary, learned during that model’s training, and two models with different vocabularies split the same sentence into different numbers of pieces. OpenAI’s tokenizers are published as tiktoken encodings; the open-weight models Groq serves are published with their own tokenizer files in their model repositories. These are different artefacts producing different answers.

The practical consequence is a silent, content-dependent error. Plain English prose tends to be the case where the mismatch looks smallest, which is why the bug survives testing. It widens on exactly the content that matters operationally: source code, JSON payloads, non-Latin scripts, long identifiers, base64 blobs, and heavily punctuated structured text. A budget calibrated on English chat that is then applied to a JSON-heavy extraction pipeline can be off by enough to blow through a minute allowance you believed you were comfortably inside.

The fix is unglamorous: load the tokenizer that belongs to the exact model string you are sending, from that model’s own repository, and use it for the count. Do it once per model at process start and cache the loaded tokenizer — instantiating it per request is a real and avoidable cost. If your service routes across several Groq-hosted models, you need one tokenizer per model, keyed by the model string, not one tokenizer for “Groq”.

The Moving-Catalogue Problem

Groq retires and replaces hosted models faster than most inference providers, and that cadence interacts badly with token counting. A pinned model string that stops resolving is a loud failure — you get a 404 and you notice. A model string you update to a successor is a quiet one: the calls keep succeeding, and if the successor carries a different vocabulary, every token count you compute afterwards changes meaning without anything in your system announcing it.

Treat the tokenizer as versioned state that belongs to the model, not to the provider. When you change a model string, invalidate any cached counts keyed on the old one, re-measure your representative prompts, and re-check the assumptions you built on top of the old numbers: your chunk sizes, your context-fitting logic, your per-minute pacing budget. The failure mode here is not an error; it is a system that has been quietly over- or under-estimating for a week.

Model retirement is covered in more depth in our Groq model deprecation guide. The point specific to counting is that a deprecation event is also a tokenizer event, and most teams only plan for the first half of that.

Counting Is How You Stay Under a TPM Ceiling

Groq meters requests per minute and tokens per minute separately, and for any workload with meaningful context the token ceiling binds first. This is the structural reason counting matters more on Groq than on a provider that meters requests: your allowance is not consumed evenly by your calls. One long-context request can take a slice of the minute that a hundred short ones would not, so a rate limiter that counts requests will let you sail past a token limit while reporting that you are well within budget.

A token-aware limiter is the answer, and it needs a pre-send number to work: before dispatching, compute the prompt tokens with the model’s tokenizer, add the max_tokens you are willing to allow for the completion, and admit the request only if that sum fits in the remaining budget for the current window. Requests that do not fit wait rather than fail. This turns the ceiling from a wall of 429s into a queue with predictable latency, which is nearly always the better user experience.

Reconcile against reality rather than trusting your estimate forever. Groq returns rate-limit headers alongside responses that report your remaining allowance, and the response usage object reports what the request actually cost. Compare your pre-send estimate to the reported usage on a sample of live traffic and track the drift as a metric. A widening gap is your early warning that a tokenizer has changed under you, that a code path is injecting content you did not count, or that a template grew.

What Your Count Will Miss

A naive count of the user’s message is always an underestimate, because it is not the only thing you send. The system prompt is charged on every single call, and it is the piece most likely to have grown through accretion — a line added here, an example added there, until a template nobody has read end-to-end in months is riding along on every request in production. Count the fully assembled payload, not the variable part of it.

Chat history is the second omission and the one that scales badly, because the conversation grows monotonically while your per-request budget does not. A session that is comfortable at turn three can be several times the size at turn thirty, and if your counting happens on the new message rather than the assembled thread, the cost curve is invisible to you right up until the ceiling.

Tool and function definitions are the third, and they are the most consistently forgotten because they live in a separate field from the prose. Schemas are serialised into the model’s context and charged like any other input, and a rich tool catalogue can be a substantial fixed cost paid on every request — including the many requests where no tool ends up being called. See the Groq tool calling guide for how that surface behaves in practice.

Finally, the completion. You cannot count it in advance because it has not been generated, and the only lever you hold is max_tokens. Leaving it unbounded is equivalent to declining to budget: your pacing arithmetic then rests on a number nobody has bounded. Set it to something you would actually accept and treat it as the completion’s reserved share of the minute.

Fitting the Context Window Is a Different Question

Two distinct constraints are both measured in tokens, and conflating them causes real bugs. The context window is a per-request limit: the assembled prompt plus the completion must fit inside the model’s window or the call is rejected outright. The TPM allowance is a per-minute limit across all your requests. A prompt can fit the window perfectly and still be the thing that exhausts your minute, and a set of tiny prompts can exhaust the minute while every one of them sits comfortably inside the window.

You need the same count for both, but the decisions differ. Overflowing the window is handled by truncating or summarising history, chunking documents, or moving to a longer-context model — see the Groq context window guide. Exhausting the minute is handled by pacing, caching, and routing overflow elsewhere. Reaching for the wrong remedy is common: teams shrink prompts to fix 429s that pacing would have fixed, degrading answer quality to solve a scheduling problem.

Counting Is Not an Outage Detector

Careful token accounting tells you when you are the reason calls are failing. It says nothing about whether api.groq.com is healthy, and during an incident that distinction determines your response. A token-metered 429 wants you to back off and let the window refill. A provider incident wants you to route traffic away immediately, and backing off is precisely the wrong move because the queue you are building will hit the same broken endpoint later.

Everything in this guide is measured inside your own client, which is exactly where those two situations look identical: errors up, latency up, throughput down. External monitoring that probes Groq on its own schedule, independent of your traffic, is what separates them. Our Is Groq Down? guide covers the outage-side playbook, and the Groq rate limits guide covers the ceilings themselves in detail.

Frequently Asked Questions

Does Groq have a token counting endpoint?

No. Groq exposes an OpenAI-compatible surface but there is no tokenize call, so the only authoritative count Groq gives you is the usage object attached to a completed response. That count arrives after you have already spent the tokens, which is the wrong side of the decision if you are trying to stay under a tokens-per-minute ceiling. Counting before you send means running the tokenizer yourself, in your own process, against the exact model you are about to call.

Can I use tiktoken to count Groq tokens?

No, and this is the most common mistake precisely because the API is OpenAI-shaped, so the code compiles, runs and returns plausible numbers. Groq serves open-weight models, each shipping its own tokenizer vocabulary, and none of those is the vocabulary tiktoken implements. The resulting error is content-dependent: smallest on plain prose, largest on code, JSON, non-Latin scripts and structured text, which is exactly the traffic where an accurate budget matters most. Load the tokenizer from the model’s own repository, cache it per model string, and count with that.

Why does my Groq token count drift after a model update?

Because the count belongs to a tokenizer, and Groq rotates its hosted catalogue quickly. Repointing to a successor model is a silent change from the counting perspective: calls keep succeeding, but if the successor carries a different vocabulary then identical text now produces a different number and every downstream assumption built on the old one is subtly wrong. Treat a model swap as invalidating cached counts, re-measure representative prompts, and re-check your chunk sizes and pacing budget against the new numbers.

Are Groq rate limits based on requests or tokens?

Both, metered separately and per model, and for realistic workloads the token ceiling is the one that binds first. This is why a request-counting limiter gives false comfort on Groq: it reports plenty of headroom while a handful of long-context calls quietly consume the minute’s token allowance. A token-aware limiter needs a pre-send number, which is the whole reason to count before dispatch rather than reading usage afterwards.

Do output tokens count against the Groq TPM limit?

Budget as though they do, because the completion is the part you cannot count and therefore the part most likely to break your arithmetic. You know the prompt exactly; you know the completion only by the ceiling you impose with max_tokens. Reserve the sum of counted prompt plus allowed max_tokens against your window, and set max_tokens to a value you would genuinely accept — an unbounded completion is an unbounded claim on a minute you are trying to schedule.

What is the difference between the context window and the TPM limit?

The context window is per request: prompt plus completion must fit or the call is rejected. The TPM allowance is per minute across every request you make. They are measured in the same unit and fixed by opposite remedies, which is why teams confuse them and reach for the wrong one. A prompt that fits the window comfortably can still exhaust your minute; a set of small prompts can exhaust the minute while each fits trivially. Window overflow is solved by truncation, chunking or a longer-context model. Minute exhaustion is solved by pacing, caching and routing overflow to a second provider.

My estimate does not match the usage Groq reports. Where is the gap?

Almost always in something you send but did not count. The four usual suspects, in order of frequency: the system prompt, which rides on every call and grows by accretion; the chat history, which grows monotonically while your budget does not; tool and function schemas, which are serialised into context and charged even on calls where no tool fires; and the completion itself, bounded only by max_tokens. Track estimate-versus-actual drift as a live metric on sampled traffic rather than validating once — a widening gap is the earliest signal that a template grew or a tokenizer changed under you.

Related Guides

Over Your Token Budget, or Is Groq Down?

Both look the same from inside your client — errors up, latency up — and they call for opposite responses. API Status Check probes api.groq.com independently of your traffic, so you know whether to pace or to route around it.

Start Your Free Trial →