Together AI Structured Output and JSON Mode: A Practical Guide (2026)

The first version of every LLM-to-JSON pipeline is a prompt that ends with "respond only with valid JSON, no other text" and a bare JSON.parse. It works in development, because you tested it on five clean inputs. It breaks in production on the input where the model opened with "Sure! Here is the JSON:", or fenced the object in a markdown block, or returned a number as a quoted string, or ran out of tokens two keys from the end.

The short version

JSON mode guarantees the response parses. It does not guarantee the response matches your schema — that is a separate mode, and even that only constrains shape, never truth. Keep a validation layer after the parse no matter which mode you used, treat a truncated response as a budget bug rather than a retryable error, and cap the repair loop at one attempt so a regression shows up as an alert instead of a bill.

Three Ways to Constrain Together AI Output, and What Each Actually Promises

These get discussed as if they were three settings of one dial. They are not — they make different guarantees, fail differently, and cost differently. Picking the wrong one is why a pipeline can have a 98% success rate that nobody can explain.

Approach	Guarantees	Does not guarantee	Typical failure
Prompt instruction only	Nothing	Syntax, keys, types	Preamble text or a markdown fence around a perfectly good object
JSON mode (`json_object`)	The response parses	Your keys, your types, required fields	Valid JSON with a renamed key, or a number returned as a string
Schema-constrained decoding	Parses and matches the declared shape	That the values are correct or grounded	A confidently wrong value in a perfectly shaped object
Tool / function calling	Arguments match the declared parameters	That the model calls the tool at all	A plain text answer where your code assumed a tool call

The row that catches people is the second one. "JSON mode" sounds like it means "the JSON I described", and it means "JSON". A response of {"result": "ok"} when you asked for three specific keys is a complete success by that contract and a production incident by yours.

What Together AI Supports

Together AI presents an OpenAI-compatible endpoint, so response_format arrives in the request body exactly where an OpenAI client would put it.

Together supports JSON mode and schema-constrained decoding, but availability is per-model rather than account-wide because it depends on the serving stack behind that particular checkpoint. Together's catalogue is the largest of the five providers here, which makes this the most important thing to check: the same request that works against one Llama variant can be rejected or silently unconstrained against another.

import { Together } from 'together-ai';

// The parameter, not the prompt, is what makes this reliable.
const res = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.3-70B-Instruct-Turbo',
  response_format: { type: 'json_object' },
  max_tokens: 1024,
  messages: [
    { role: 'system', content: 'Return a JSON object with keys: title (string), tags (array of strings), sentiment (one of positive, neutral, negative).' },
    { role: 'user', content: input },
  ],
});

// Still not safe. This parses, but nothing above promised these keys exist.
const raw = JSON.parse(res.choices[0].message.content);

Note the comment on the last line. That call is correct and the response will parse, and the variable is still any-shaped data from a third party. Everything in the next section exists because of the gap between those two facts.

The Validation Layer You Still Need

Treat the Together AI response as untrusted input, because that is exactly what it is. Three distinct checks belong here and they fail for different reasons, so collapsing them into one try/catch destroys the information you need to fix anything.

import { z } from 'zod';

const Result = z.object({
  title: z.string().min(1),
  tags: z.array(z.string()).max(10),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
});

function parseTogetherAiResponse(content, finishReason) {
  // 1. Truncation is not a parse problem and must not be retried blindly.
  if (finishReason === 'length') {
    throw new TruncatedResponseError('max_tokens too small for this schema');
  }

  // 2. Syntactic layer.
  let obj;
  try {
    obj = JSON.parse(content);
  } catch {
    throw new MalformedJsonError(content.slice(0, 200));
  }

  // 3. Structural layer — the one JSON mode never covered.
  const parsed = Result.safeParse(obj);
  if (!parsed.success) {
    throw new SchemaMismatchError(parsed.error.issues);
  }
  return parsed.data;
}

The ordering matters more than the library. Checking the finish reason before parsing is what stops a token-budget problem from being misfiled as a model-quality problem — a truncated object and a badly-formatted object both throw at JSON.parse, and only one of them gets better if you retry.

Structured Output and Streaming

These two features compose badly, and the reason is structural rather than a bug in anyone's implementation. A stream delivers bytes; a JSON object is only meaningful once it closes. Every chunk before the last one is, by definition, invalid JSON.

That leaves two honest options. Buffer the entire stream and parse once at the end — in which case streaming has bought you a progress spinner and nothing else, which is often the right trade. Or run an incremental parser that can interpret a partial object and render fields as they complete, which is genuinely nice for a form-filling UI and carries one non-obvious hazard: a value that appeared at 40% of the stream is not final. Nothing prevents a model from producing a later key that changes what an earlier one meant, and a UI that has already committed to the first render will show the user a value your backend never accepted.

If you do stream, keep the authoritative parse at the end and treat the incremental one as presentation only. Two parses is not duplication — it is the difference between what the user sees and what you write to a database.

The Failure Modes Worth Writing Tests For

Symptom	Real cause	Right response
Unexpected end of JSON input	`max_tokens` hit mid-object	Raise the budget or shrink the schema. Do not retry.
Unexpected token 'S' at position 0	Conversational preamble — JSON mode was not actually on	Set `response_format`; verify the model supports it
Parses, but a field is missing	JSON mode without schema constraints	Move to constrained decoding, or validate and repair once
A number arrives as `"42"`	Type coercion the syntax check cannot see	Coerce deliberately in the validator, not implicitly downstream
Enum value drifts to a synonym	Enum expressed in the prompt rather than the schema	Declare the enum in the schema; reject rather than fuzzy-match
Object wrapped in a markdown fence	Prompt-only constraint on a chat-tuned model	Use the API parameter; strip fences only as a stopgap

Every one of these is cheap to reproduce with a mocked response and expensive to diagnose at 03:00 from a stack trace that says SyntaxError. Write them as fixtures once.

The Trap: Per-Model Capability Across a Huge Catalogue

The reason people choose Together is breadth — dozens of open models behind one key, so you can move a workload to whatever is cheapest or best this quarter. Structured output is where that breadth turns into a liability, because constrained decoding is a property of the serving stack behind a specific checkpoint, not of the account or the endpoint. Two model strings that accept identical request bodies can behave differently on the same response_format.

The failure that costs teams the most is not a rejection. A rejection is loud and you fix it in ten minutes. The expensive case is the request that is accepted and returns unconstrained text that happens to look like JSON most of the time, because then your parse-success rate is 90-something percent and the remainder is written off as model flakiness for months.

Before you route production traffic to a new Together model, send it a deliberately hard schema — nested objects, an enum, an optional field — twenty times and count exact-match parses. If the number is not effectively 100%, that model is not doing constrained decoding for you regardless of what the request accepted.

Bound the Repair Loop

Feeding the validation error back to the model and asking it to try again works, and it is the most commonly over-applied fix in this whole area. Every extra attempt doubles cost and latency for that request, and an unbounded loop turns a schema regression into a quiet budget leak that nobody notices until the invoice.

const MAX_ATTEMPTS = 2; // one call, one repair. Not three, not five.

async function structured(input) {
  let lastError;
  for (let attempt = 0; attempt &lt; MAX_ATTEMPTS; attempt++) {
    const res = await callTogetherAi(input, lastError);
    try {
      const value = parseTogetherAiResponse(
        res.choices[0].message.content,
        res.choices[0].finish_reason,
      );
      metrics.increment('together-ai.structured.ok', { attempt });
      return value;
    } catch (err) {
      if (err instanceof TruncatedResponseError) throw err; // retrying will truncate identically
      metrics.increment('together-ai.structured.repair', { reason: err.name });
      lastError = err;
    }
  }
  // Fail loudly. A silent fallback here is how a broken schema survives a quarter.
  throw new StructuredOutputFailed('together-ai', lastError);
}

Two attempts, a metric on every repair, and a hard failure at the end. The metric is the important part: a repair rate that climbs from 2% to 20% after a prompt change is the earliest signal you will get that something broke, and it is invisible to any monitor that only watches HTTP status codes — because every one of those repairs is a 200.

When the Problem Is Not Your Schema

There is a category of structured-output incident that no amount of validation will fix, and it looks identical from inside your code: elevated latency pushing requests into timeouts, intermittent 5xx responses, or degraded output quality during an upstream incident. Your parse failures spike, your repair rate spikes, and every instinct says the prompt regressed.

The question that separates the two is not answerable from your own logs, and it is always the same question: is it us or is it Together AI? Answering it requires something outside your application making real requests to api.together.xyz on an interval and recording error rate and latency independently of your traffic — otherwise the first thirty minutes of every incident are spent re-reading a prompt that never changed.

Frequently Asked Questions

Does JSON mode guarantee the Together AI response matches my schema?

No, and conflating the two is the most expensive misunderstanding in this area. JSON mode guarantees syntactic validity — the response will parse. It says nothing about whether the keys are the ones you asked for, whether a number came back as a string, or whether a required field is present at all. Schema-constrained decoding is the mode that constrains structure, and even that only constrains shape, never correctness of the values. Validate after parsing regardless of which mode you used.

Why does my Together AI response contain valid JSON wrapped in explanatory text?

Because the model was asked in the prompt for JSON rather than constrained by the API to produce it. Left to itself an instruct model will often preface an object with a sentence, or fence it in a markdown code block, and both of those make JSON.parse throw on a response that a human would call correct. The fix is to use the Together AI response-format parameter rather than a stricter prompt. If you cannot, extract the first balanced brace span before parsing — but treat that as a workaround, not a design.

What happens if the Together AI response is cut off mid-object?

You get a truncated string that fails to parse, and the cause is almost always max_tokens rather than anything to do with structured output. Constrained decoding will not let the model close the object early to fit your budget — it will simply stop mid-token when the limit lands. Check the finish reason on every response: if it is a length stop, the right response is a larger budget or a smaller schema, not a retry, because a retry with the same budget will truncate again in exactly the same place.

Can I stream structured output from Together AI?

Yes, but the stream is a byte stream, not an object stream — you receive a partial, unparseable fragment on every chunk until the last one. If you only need the finished object, buffer the whole stream and parse once at the end; streaming then buys you nothing except a progress indicator. If you genuinely want to render fields as they arrive, you need an incremental or repair-tolerant JSON parser, and you must be certain that a field which appears mid-stream cannot be revised before the object closes.

Do all Together AI models support JSON mode?

No, and this is the single most common source of surprise on Together specifically. Constrained decoding depends on the serving stack behind a given checkpoint, so support is per-model rather than per-account, and the catalogue is large enough that assuming parity will eventually bite. Worse, an unsupported model does not always reject the request — it can accept it and return unconstrained text, giving you a high-but-not-perfect parse rate that reads as model flakiness. Verify each model string against Together's documentation and with a twenty-call hard-schema test before it takes production traffic.

Related Guides

A Parse Failure and an Outage Look Identical

Your validator cannot tell the difference between a schema regression and Together AI having a bad hour. API Status Check probes api.together.xyz and the rest of your stack independently and alerts on errors and latency, so the first question of every incident already has an answer.

Start Your Free Trial →