Together AI Connection Reset: The Marketplace Routing Trap

The instinct with a connection reset is to ask whether the provider is up. On a marketplace that question is malformed. Together AI routes a single endpoint to hundreds of separately provisioned model deployments, so “is Together AI up” and “is the thing serving my model id up” are different questions with different answers.

A green platform status page next to a 100% failure rate on one model id is the expected reading, not a contradiction. The first thing to establish is therefore not whether Together AI is healthy but whether the failure follows the model id, the endpoint, or your own connection pool.

Reset after connect is a routing signal

Read the timing, because it separates the two dominant causes cleanly. A reset that lands within a millisecond or two of the write, before any routing could have happened, is a dead pooled socket on your side. A reset that lands hundreds of milliseconds or seconds later — after the TCP handshake and TLS completed successfully — means the connection was accepted, routing began, and something downstream gave up.

That second shape is the marketplace one. The edge terminates your connection immediately; only afterwards does the request get matched to a serving replica for your model id. If no warm replica exists, the path involves a cold start, and a serverless model that has not been called recently can take long enough to provision that an intermediary in the chain severs the connection rather than continuing to wait.

The practical consequence is that this failure mode is worst on exactly the models you care least about keeping warm: niche or newly released ids with little shared traffic. High-traffic models are effectively always warm, so the same code against Llama 3.3 70B looks flawless while a rarely-used fine-tune resets constantly.

Dedicated endpoints add a reset that shared serving cannot produce

If you have moved to a dedicated endpoint, you have taken ownership of a lifecycle that shared serving hides. Dedicated deployments scale down when idle and are replaced during updates, and during those transitions connections are refused or reset outright. This failure is total for your tenant and completely invisible to every shared signal — the status page, other customers, and any monitoring pointed at the public API will all look perfectly healthy while every one of your requests fails.

The tell is that the resets are bounded in time and correlate with your own traffic pattern rather than with anyone else’s: a burst after a quiet overnight window, or a cluster immediately after a configuration change. Autoscaling with a warm minimum replica removes the first case; retrying with backoff across a short window absorbs the second.

This is also the case where retries genuinely help rather than just adding load, because the underlying condition resolves in seconds on its own. A capped retry with jitter across roughly thirty seconds converts a hard failure into a slow success.

Tag every reset with the model id, then tune

The instrumentation matters more than the pool settings here, because the pool settings only address one of the two causes. If your logs cannot answer “which model id, and was the socket reused?”, you cannot tell the two apart.

// Scope every connection failure to a model id — otherwise the two causes look identical.
import { Agent, setGlobalDispatcher } from 'undici';

setGlobalDispatcher(new Agent({ keepAliveTimeout: 4_000, keepAliveMaxTimeout: 10_000 }));

export async function callTogether(model: string, body: object) {
  const t0 = Date.now();
  try {
    const res = await fetch('https://api.together.xyz/v1/chat/completions', {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.TOGETHER_API_KEY}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({ ...body, model }),
    });
    if (!res.ok) throw new Error(`upstream ${res.status}`);
    return res;
  } catch (err) {
    // ms near zero  => stale pooled socket (your side)
    // ms in the hundreds/thousands => accepted, then routing/cold-start gave up
    console.error('together_conn_reset', { model, ms: Date.now() - t0, cause: String((err as Error).cause ?? err) });
    throw err;
  }
}

Then handle the failure itself. The rule that keeps retries honest is the same everywhere: retry only when nothing was read back, cap the attempts, and always add jitter — a reset that hits many sockets at once will otherwise synchronise every client into a single reconnect wave that re-fails together.

// Cold-start resets resolve on their own — retry across a short window, with jitter.
async function withColdStartRetry<T>(fn: () => Promise<T>, budgetMs = 30_000) {
  const deadline = Date.now() + budgetMs;
  let wait = 400;
  for (;;) {
    try {
      return await fn();
    } catch (err) {
      const reset = String((err as Error).cause ?? err).includes('ECONNRESET');
      if (!reset || Date.now() + wait > deadline) throw err;
      await new Promise((r) => setTimeout(r, wait + Math.random() * wait));
      wait = Math.min(wait * 2, 5_000);   // jitter, or every client reconnects together
    }
  }
}

The stale-socket half, which Together shares with everyone

Underneath the marketplace-specific causes sits the ordinary one. Your HTTP client keeps connections open between requests; the edge has its own idle threshold; when your client hands out a socket the edge already closed, the RST comes back on write. This is the near-zero-millisecond shape from the timing section.

It is fixed the same way it is fixed everywhere: set the client’s idle timeout strictly below the server’s, cap the maximum lifetime of any connection, and give the pool enough connections that a burst does not force reuse of marginal sockets. Four seconds idle and a ten second ceiling is a safe starting point.

Fix this one first even if you suspect routing, because it is cheap and it removes the noise that makes the model-scoped signal hard to read.

Frequently Asked Questions

Related Together AI Guides

• Is Together AI Down? Outage Checking Guide
• Together AI API Timeout Error Explained
• Together AI API Error Codes Reference
• Why Your Together AI Stream Breaks
• Together AI API Retry Budget Guide
• Together AI API Failover Guide
• Together AI API Monitoring Guide