Cohere Structured Output and JSON Mode: A Practical Guide (2026)

The first version of every LLM-to-JSON pipeline is a prompt that ends with "respond only with valid JSON, no other text" and a bare JSON.parse. It works in development, because you tested it on five clean inputs. It breaks in production on the input where the model opened with "Sure! Here is the JSON:", or fenced the object in a markdown block, or returned a number as a quoted string, or ran out of tokens two keys from the end.

The short version

JSON mode guarantees the response parses. It does not guarantee the response matches your schema — that is a separate mode, and even that only constrains shape, never truth. Keep a validation layer after the parse no matter which mode you used, treat a truncated response as a budget bug rather than a retryable error, and cap the repair loop at one attempt so a regression shows up as an alert instead of a bill.

Three Ways to Constrain Cohere Output, and What Each Actually Promises

These get discussed as if they were three settings of one dial. They are not — they make different guarantees, fail differently, and cost differently. Picking the wrong one is why a pipeline can have a 98% success rate that nobody can explain.

Approach	Guarantees	Does not guarantee	Typical failure
Prompt instruction only	Nothing	Syntax, keys, types	Preamble text or a markdown fence around a perfectly good object
JSON mode (`json_object`)	The response parses	Your keys, your types, required fields	Valid JSON with a renamed key, or a number returned as a string
Schema-constrained decoding	Parses and matches the declared shape	That the values are correct or grounded	A confidently wrong value in a perfectly shaped object
Tool / function calling	Arguments match the declared parameters	That the model calls the tool at all	A plain text answer where your code assumed a tool call

The row that catches people is the second one. "JSON mode" sounds like it means "the JSON I described", and it means "JSON". A response of {"result": "ok"} when you asked for three specific keys is a complete success by that contract and a production incident by yours.

What Cohere Supports

Cohere ships its own SDK with a chat-shaped surface, and response_format is supplied alongside the message rather than as a nested completion parameter.

Cohere supports a JSON response format on the Command R family, including a schema-constrained variant that accepts a JSON Schema subset. The subset is the part worth reading carefully: constructs that a general JSON Schema validator accepts are not all supported by a constrained decoder, and an unsupported keyword is more likely to be ignored than to raise.

import { CohereClient } from 'cohere-ai';

// The parameter, not the prompt, is what makes this reliable.
const res = await client.chat.completions.create({
  model: 'command-r-plus',
  response_format: { type: 'json_object' },
  max_tokens: 1024,
  messages: [
    { role: 'system', content: 'Return a JSON object with keys: title (string), tags (array of strings), sentiment (one of positive, neutral, negative).' },
    { role: 'user', content: input },
  ],
});

// Still not safe. This parses, but nothing above promised these keys exist.
const raw = JSON.parse(res.choices[0].message.content);

Note the comment on the last line. That call is correct and the response will parse, and the variable is still any-shaped data from a third party. Everything in the next section exists because of the gap between those two facts.

The Validation Layer You Still Need

Treat the Cohere response as untrusted input, because that is exactly what it is. Three distinct checks belong here and they fail for different reasons, so collapsing them into one try/catch destroys the information you need to fix anything.

import { z } from 'zod';

const Result = z.object({
  title: z.string().min(1),
  tags: z.array(z.string()).max(10),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
});

function parseCohereApiResponse(content, finishReason) {
  // 1. Truncation is not a parse problem and must not be retried blindly.
  if (finishReason === 'length') {
    throw new TruncatedResponseError('max_tokens too small for this schema');
  }

  // 2. Syntactic layer.
  let obj;
  try {
    obj = JSON.parse(content);
  } catch {
    throw new MalformedJsonError(content.slice(0, 200));
  }

  // 3. Structural layer — the one JSON mode never covered.
  const parsed = Result.safeParse(obj);
  if (!parsed.success) {
    throw new SchemaMismatchError(parsed.error.issues);
  }
  return parsed.data;
}

The ordering matters more than the library. Checking the finish reason before parsing is what stops a token-budget problem from being misfiled as a model-quality problem — a truncated object and a badly-formatted object both throw at JSON.parse, and only one of them gets better if you retry.

Structured Output and Streaming

These two features compose badly, and the reason is structural rather than a bug in anyone's implementation. A stream delivers bytes; a JSON object is only meaningful once it closes. Every chunk before the last one is, by definition, invalid JSON.

That leaves two honest options. Buffer the entire stream and parse once at the end — in which case streaming has bought you a progress spinner and nothing else, which is often the right trade. Or run an incremental parser that can interpret a partial object and render fields as they complete, which is genuinely nice for a form-filling UI and carries one non-obvious hazard: a value that appeared at 40% of the stream is not final. Nothing prevents a model from producing a later key that changes what an earlier one meant, and a UI that has already committed to the first render will show the user a value your backend never accepted.

If you do stream, keep the authoritative parse at the end and treat the incremental one as presentation only. Two parses is not duplication — it is the difference between what the user sees and what you write to a database.

The Failure Modes Worth Writing Tests For

Symptom	Real cause	Right response
Unexpected end of JSON input	`max_tokens` hit mid-object	Raise the budget or shrink the schema. Do not retry.
Unexpected token 'S' at position 0	Conversational preamble — JSON mode was not actually on	Set `response_format`; verify the model supports it
Parses, but a field is missing	JSON mode without schema constraints	Move to constrained decoding, or validate and repair once
A number arrives as `"42"`	Type coercion the syntax check cannot see	Coerce deliberately in the validator, not implicitly downstream
Enum value drifts to a synonym	Enum expressed in the prompt rather than the schema	Declare the enum in the schema; reject rather than fuzzy-match
Object wrapped in a markdown fence	Prompt-only constraint on a chat-tuned model	Use the API parameter; strip fences only as a stopgap

Every one of these is cheap to reproduce with a mocked response and expensive to diagnose at 03:00 from a stack trace that says SyntaxError. Write them as fixtures once.

The Trap: Schema Subsets and RAG-Shaped Answers

Cohere's constrained decoding accepts a subset of JSON Schema, not the whole specification. That sounds like a footnote until you paste in a schema generated from a Zod or Pydantic model, which will cheerfully emit oneOf, pattern constraints, tuple-typed arrays and recursive $refs. The dangerous outcome is not rejection — it is a keyword that is quietly not enforced, so the field you believed was constrained to three enum values is in practice constrained to being a string.

The second Cohere-specific wrinkle is that Command R is built for retrieval-augmented answering with document citations. Same problem as any grounded model: a schema that has no field for provenance instructs the model to throw the citations away, and you end up with a clean object whose claims cannot be traced back to the documents you supplied.

Write the schema by hand for the constrained call rather than generating it from your application types, keep it to objects, plain-typed fields, enums and simple arrays, and add explicit fields for the source documents behind each claim. Validate against your richer application schema after parsing — that is where oneOf and regex belong.

Bound the Repair Loop

Feeding the validation error back to the model and asking it to try again works, and it is the most commonly over-applied fix in this whole area. Every extra attempt doubles cost and latency for that request, and an unbounded loop turns a schema regression into a quiet budget leak that nobody notices until the invoice.

const MAX_ATTEMPTS = 2; // one call, one repair. Not three, not five.

async function structured(input) {
  let lastError;
  for (let attempt = 0; attempt &lt; MAX_ATTEMPTS; attempt++) {
    const res = await callCohereApi(input, lastError);
    try {
      const value = parseCohereApiResponse(
        res.choices[0].message.content,
        res.choices[0].finish_reason,
      );
      metrics.increment('cohere.structured.ok', { attempt });
      return value;
    } catch (err) {
      if (err instanceof TruncatedResponseError) throw err; // retrying will truncate identically
      metrics.increment('cohere.structured.repair', { reason: err.name });
      lastError = err;
    }
  }
  // Fail loudly. A silent fallback here is how a broken schema survives a quarter.
  throw new StructuredOutputFailed('cohere', lastError);
}

Two attempts, a metric on every repair, and a hard failure at the end. The metric is the important part: a repair rate that climbs from 2% to 20% after a prompt change is the earliest signal you will get that something broke, and it is invisible to any monitor that only watches HTTP status codes — because every one of those repairs is a 200.

When the Problem Is Not Your Schema

There is a category of structured-output incident that no amount of validation will fix, and it looks identical from inside your code: elevated latency pushing requests into timeouts, intermittent 5xx responses, or degraded output quality during an upstream incident. Your parse failures spike, your repair rate spikes, and every instinct says the prompt regressed.

The question that separates the two is not answerable from your own logs, and it is always the same question: is it us or is it Cohere? Answering it requires something outside your application making real requests to api.cohere.ai on an interval and recording error rate and latency independently of your traffic — otherwise the first thirty minutes of every incident are spent re-reading a prompt that never changed.

Frequently Asked Questions

Does JSON mode guarantee the Cohere response matches my schema?

No, and conflating the two is the most expensive misunderstanding in this area. JSON mode guarantees syntactic validity — the response will parse. It says nothing about whether the keys are the ones you asked for, whether a number came back as a string, or whether a required field is present at all. Schema-constrained decoding is the mode that constrains structure, and even that only constrains shape, never correctness of the values. Validate after parsing regardless of which mode you used.

Why does my Cohere response contain valid JSON wrapped in explanatory text?

Because the model was asked in the prompt for JSON rather than constrained by the API to produce it. Left to itself an instruct model will often preface an object with a sentence, or fence it in a markdown code block, and both of those make JSON.parse throw on a response that a human would call correct. The fix is to use the Cohere response-format parameter rather than a stricter prompt. If you cannot, extract the first balanced brace span before parsing — but treat that as a workaround, not a design.

What happens if the Cohere response is cut off mid-object?

You get a truncated string that fails to parse, and the cause is almost always max_tokens rather than anything to do with structured output. Constrained decoding will not let the model close the object early to fit your budget — it will simply stop mid-token when the limit lands. Check the finish reason on every response: if it is a length stop, the right response is a larger budget or a smaller schema, not a retry, because a retry with the same budget will truncate again in exactly the same place.

Can I stream structured output from Cohere?

Yes, but the stream is a byte stream, not an object stream — you receive a partial, unparseable fragment on every chunk until the last one. If you only need the finished object, buffer the whole stream and parse once at the end; streaming then buys you nothing except a progress indicator. If you genuinely want to render fields as they arrive, you need an incremental or repair-tolerant JSON parser, and you must be certain that a field which appears mid-stream cannot be revised before the object closes.

Can I pass a Zod or Pydantic generated schema straight to Cohere?

You can pass it, but do not assume all of it is enforced. Cohere's constrained decoding supports a subset of JSON Schema, and generated schemas routinely include constructs outside that subset — oneOf, regex patterns, tuple arrays, recursive refs. An unsupported keyword tends to be ignored rather than rejected, which produces the worst possible outcome: a schema you believe is enforcing an enum that is really only enforcing a string. Hand-write a simple schema for the API call, then validate the parsed result against your full application schema locally.

Related Guides

A Parse Failure and an Outage Look Identical

Your validator cannot tell the difference between a schema regression and Cohere having a bad hour. API Status Check probes api.cohere.ai and the rest of your stack independently and alerts on errors and latency, so the first question of every incident already has an answer.

Start Your Free Trial →