How to Test and Mock the Together AI API (2026 Guide)

The first integration test anyone writes against Together AI calls the real endpoint, asserts the model said something sensible, and passes. It keeps passing for about three weeks. Then it fails on a Sunday because of a rate limit, fails again on a fork's pull request because the key is not there, fails a third time because the model phrased its answer differently — and someone adds `retry: 3` and stops reading the output.

The short version

Mock the network, not the SDK. Record fixtures from one real call rather than hand-writing them. Spend most of your testing effort on the failure responses, because those are the paths that only ever run during an incident. And keep one live check outside CI — a suite full of mocks cannot tell you Together AI is down.

Why CI Should Not Call Together AI Directly

None of these are theoretical. Each one is a specific way a live-calling test suite degrades into a suite nobody trusts.

Your build inherits their uptime. A Together AI incident becomes a red build across every branch, and the team learns that red means "probably upstream" — which is exactly the habit that lets a real regression through.
Every pull request costs money. Token spend scales with contributor activity rather than with anything you would choose to pay for, and it is invisible until the invoice.
Secrets have to exist everywhere CI runs. That includes fork pull requests, where a real TOGETHER_API_KEY either fails to exist (so external contributors can never get a green build) or does exist (which is a far worse problem).
Model output is not deterministic. Asserting on generated prose produces failures that are not bugs, and the standard fix — loosening the assertion until it always passes — leaves a test that costs money and proves nothing.
Rate limits look exactly like flakiness. A parallelised suite trips Together AI's limits precisely because it is fast, and the 429 arrives as an intermittent failure rather than as a clear message about concurrency.

The Four Layers, and What Each One Proves

Most teams have layer one and layer four missing, then wonder why a green suite did not prevent the outage. Each layer answers a different question, and no layer answers another layer's.

Layer	Runs	Proves	Cannot tell you
Unit / mocked	Every push	Your prompt construction, routing, parsing, retry and error-handling logic	Nothing about the provider itself
Recorded fixtures	Every push	That your code survives real response bodies, including error bodies	Whether those bodies are still current
Live smoke test	Pre-deploy, or nightly	Credentials work, model ids resolve, the contract still holds	Behaviour under real production load
Synthetic monitoring	Continuously	That the provider is up and fast right now, from outside your app	Nothing — this is the layer that pages you

The line worth internalising: layers one and two are about your code, layer three is about your assumptions, and layer four is about Together AI. Adding more of the first two will never substitute for the last one.

A Mock You Can Paste In

Together AI is OpenAI-compatible, and most codebases reach it through the OpenAI SDK pointed at Together, or the `together-ai` package. Intercept at the HTTP boundary so the real client still runs — its validation, its error classes, its stream parsing all stay in the test.

import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';

const server = setupServer(
  http.post('https://api.together.xyz/v1/chat/completions', () =>
    HttpResponse.json(SUCCESS_FIXTURE)),
);

beforeAll(() => server.listen({ onUnhandledRequest: 'error' }));
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

// onUnhandledRequest: 'error' is the important line. Without it, a call your
// code makes to an endpoint you forgot to stub goes to the real api.together.xyz
// with whatever key is in the environment.

Get the success fixture by making one real call and saving the response verbatim, not by writing out what you think the body looks like. A hand-authored fixture tests your code against your own assumptions, which it will always pass.

Simulate the Four Failures That Actually Happen

This is where mocking earns its cost. Every one of these paths runs for the first time during an incident unless a test made it run earlier.

// 1. Rate limited — assert the backoff reads retry-after.
http.post('https://api.together.xyz/v1/chat/completions', () =>
  new HttpResponse(null, { status: 429, headers: { 'retry-after': '2' } }));

// 2. Server error — assert a bounded retry, then a clean surfaced failure.
http.post('https://api.together.xyz/v1/chat/completions', () =>
  new HttpResponse(null, { status: 500 }));

// 3. Timeout — assert your own deadline fires before the user's patience does.
http.post('https://api.together.xyz/v1/chat/completions', async () => {
  await new Promise((r) => setTimeout(r, 30_000));
  return HttpResponse.json(SUCCESS_FIXTURE);
});

// 4. Truncated stream — the one nobody tests, and the one that
//    quietly writes half an answer into your database as if it were whole.
http.post('https://api.together.xyz/v1/chat/completions', () => {
  const stream = new ReadableStream({
    start(controller) {
      controller.enqueue(new TextEncoder().encode(
        'data: {"choices":[{"delta":{"content":"The ans"}}]}\n\n'));
      controller.close(); // no [DONE], no finish_reason
    },
  });
  return new HttpResponse(stream, {
    headers: { 'Content-Type': 'text/event-stream' },
  });
});

The truncated stream is the expensive one

A stream that ends without a terminator is not an error at the HTTP layer — the status was 200 and bytes arrived. Code that accumulates deltas and writes the result on stream-close will happily store half an answer and mark it complete. Assert that a stream ending without a finish signal is treated as a failure, not as a short response.

The Together AI Trap: Fixtures Outlive the Model They Were Recorded Against

Together AI's advantage is catalogue breadth — hundreds of open models behind one OpenAI-shaped endpoint. The cost of that breadth is that individual model ids come and go far more often than they do at a provider serving three of its own models. A fixture recorded against a model id that has since been retired keeps passing, because a recorded response does not care whether the endpoint would still serve it.

This is the failure mode worth designing against: the suite is green, the fixture is stale, and the first real request in production returns a 404 or an 'model not found' error for a model your tests are still happily replaying. Nothing in a purely mocked suite can catch it, because the mock *is* the stale thing.

The fix is cheap and it is not more mocking. Add one scheduled job that does nothing but confirm every model id your code references still resolves — a single minimal completion per id, run daily, outside the pull-request path. It costs a few cents a month and it converts a silent production 404 into a mail in your inbox.

A model-id liveness check to run on a schedule

// Every model id the codebase can send to Together AI.
const MODELS = [
  'meta-llama/Llama-3.3-70B-Instruct-Turbo',
  // ...add each one your routing layer can select
];

for (const model of MODELS) {
  const res = await fetch('https://api.together.xyz/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.TOGETHER_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model,
      messages: [{ role: 'user', content: 'ok' }],
      max_tokens: 1,
    }),
  });

  // 404 / model-not-found here means a fixture in the suite is lying to you.
  if (!res.ok) {
    console.error(`STALE MODEL ID: ${model} -> ${res.status}`);
    process.exitCode = 1;
  }
}

The One Test That Should Hit the Real API

Keep exactly one, keep it out of the pull-request path, and give it a narrow job: prove that the credentials work, that the model id still resolves, and that the response still has the shape your fixtures claim. It is the only thing standing between a fully mocked suite and a fixture that has quietly stopped describing reality.

// One test. Runs pre-deploy or nightly, never on a pull request.
// Its job is to prove the assumptions the mocks are built on.
const res = await fetch('https://api.together.xyz/v1/chat/completions', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.TOGETHER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'meta-llama/Llama-3.3-70B-Instruct-Turbo',
    messages: [{ role: 'user', content: 'Reply with the single word: ok' }],
    max_tokens: 5,
  }),
});

if (!res.ok) {
  throw new Error(`Together AI smoke test failed: ${res.status}`);
}

// Assert the SHAPE the fixtures encode, not the words that came back.
const body = await res.json();
assertMatchesFixtureShape(body);

Run it on a schedule and before deploy. Do not run it on pull requests, because that reintroduces every problem in the first section — the secret exposure, the fork failures, the upstream-outage red builds — in exchange for a signal you already get nightly.

Where Testing Stops and Monitoring Starts

A test suite runs when you push. An outage happens when it happens. That gap is the entire reason a green pipeline and a broken product coexist so comfortably: nothing in CI is watching Together AI at 02:00 on a Saturday, and a nightly smoke test tells you about a failure up to twenty-four hours after your users found it.

The layer that closes the gap is independent probing — real requests to api.together.xyz on an interval, from outside your infrastructure, alerting on both errors and latency. It is also what answers the question your team will actually ask during an incident, which is never "did the tests pass" but always "is it us or is it them?"

Frequently Asked Questions

Should my CI pipeline call the live Together AI API?

Almost never on every push. Live calls in CI make your build depend on someone else's uptime, cost money on every pull request, leak a real TOGETHER_API_KEY into every fork and pull-request runner, and produce non-deterministic failures that teams eventually silence by retrying. Keep the pull-request path fully mocked, and run a small live smoke test on a schedule or before deploy where a failure means something and a secret can be handled properly.

What is the difference between mocking the Together AI SDK and mocking the HTTP layer?

Mocking the SDK replaces the client object, so your test never exercises the client's own validation, error classes or stream parsing — the parts most likely to change under you. Mocking the HTTP layer (msw, nock, a local server, or a base-URL override) intercepts the request to api.together.xyz and lets the real client run against a fake network. The second is more work to set up once and catches an entire category of bug the first cannot see, including SDK upgrades that change behaviour silently.

How do I test what my code does when Together AI returns a 429 or a 500?

Make the mock return them. This is the single highest-value thing a mock does, because the failure paths are the ones that are never exercised by hand and are always the ones that break in an incident. Return a 429 with a retry-after header and assert the backoff honours it rather than sleeping a hardcoded amount; return a 500 and assert the request is retried a bounded number of times; return a stream that ends mid-message and assert the partial output is not silently persisted as complete.

How often should Together AI fixtures be re-recorded?

Re-record on a trigger, not on a calendar. The triggers that matter are: you changed the request your code sends, you changed the model id, or the scheduled liveness check told you an id stopped resolving. A monthly 'refresh all fixtures' ritual mostly produces large diffs nobody reads, which is worse than stale fixtures because it trains the team to approve fixture churn without looking. Small, caused, reviewable re-records beat bulk refreshes.

If everything is mocked, how do I know when Together AI is actually down?

You do not, and that is the honest limit of a test suite: mocks tell you your code is correct, not that the provider is available. Availability is a monitoring question, not a testing one. Something outside your application has to be making real requests to api.together.xyz on an interval and alerting when they fail or slow down — otherwise the first signal of an outage is a user, and the first thing your team will do is spend twenty minutes deciding whether the bug is yours.

Related Guides

Your Tests Cannot Page You at 2am

Mocks prove your code is right. They cannot tell you Together AI stopped answering an hour ago. API Status Check probes Together AI and the rest of your stack independently and alerts on errors and latency, so "is it us or is it them?" has an answer before anyone opens a terminal.

Start Your Free Trial →