Mistral AI Retry Budgets and Backoff

Almost every guide about calling an AI API treats retries as the answer. Wrap the call, set a max attempt count, add exponential backoff, move on. That advice is correct for a single client talking to an infinite service, and it is actively dangerous for a fleet of clients talking to api.mistral.ai, because it optimises the wrong thing. It makes each individual call more likely to succeed while making the system as a whole more likely to fail, and the mechanism is arithmetic rather than subtle: retries are new requests, and they are issued in the exact window when the dependency has the least capacity to serve them.

This is a different problem from the one the rate-limit guide covers. Rate limits are the provider’s side of the contract — the ceiling api.mistral.ai publishes and enforces. This guide is about your side of the wire: the policy your own client applies after something goes wrong, and whether that policy is helping the dependency recover or holding it down. It is also distinct from failover, which is about where the traffic goes when a provider is genuinely unavailable. Here the provider is usually still available. The question is how much of what it is serving is work you already asked for.

The shape to keep in mind: a service degrades enough that ten percent of calls fail. A policy of three attempts per call turns that ten percent into roughly a thirty percent increase in offered load. The higher load pushes the failure rate up, which triggers more retries, which pushes it higher. Nothing in the system is misconfigured and every component is behaving exactly as designed. This is the failure mode a retry budget exists to break.

First: Decide What Is Even Retriable

Most retry damage comes from retrying things that were never going to succeed. Before tuning any timing, classify the failures — roughly half the rows below should never be retried at all, and the two timeout rows are the ones teams consistently get wrong.

The distinction between the two timeout rows is the one worth internalising. A connect timeout means no bytes reached the server, so nothing happened and a retry is free. A read timeout means your request arrived, the server may well still be working on it, and it is very likely billing you for tokens it is generating right now. Retrying that is not a second chance at the same operation — it is a second operation, running concurrently with the first.

The Budget: A Ratio, Not a Count

This is the central idea, and it is the one thing on this page that changes behaviour during a real incident. An attempt count is a per-call property, so it gives you no control over aggregate load; a budget is a fleet property expressed as the share of extra traffic retries are permitted to add on top of successful traffic.

// A retry budget is a RATIO against successful traffic, not a per-call count.
// Per-call counts are unbounded in aggregate: 10k concurrent callers x 3 tries
// = 30k requests at exactly the moment the provider is least able to serve them.
const BUDGET_RATIO = 0.1;        // retries may add at most 10% on top of successes
const MIN_RETRIES_PER_SEC = 5;   // floor, so low-traffic services can still retry

const window = new SlidingWindow(60_000);  // successes + retries, last 60s

function mayRetry(): boolean {
  const allowed = Math.max(
    MIN_RETRIES_PER_SEC * 60,
    window.successes() * BUDGET_RATIO
  );
  if (window.retries() >= allowed) {
    metrics.increment('retry.budget_exhausted');   // <-- the alert that matters
    return false;                                   // shed, do not amplify
  }
  return true;
}

// Retry is now conditional on GLOBAL health, not just on this call's history.
// When the provider is broadly down, successes fall, the budget shrinks with
// them, and the system stops retrying automatically. That is the property a
// per-call attempt count can never have.
async function call(req: Request, deadline: number) {
  for (let attempt = 0; ; attempt++) {
    if (Date.now() > deadline) throw new DeadlineExceeded();  // caller's clock wins
    try {
      const res = await send(req, { timeout: deadline - Date.now() });
      window.recordSuccess();
      return res;
    } catch (e) {
      if (!isRetriable(e)) throw e;              // 400/401/403/404: never retry
      if (attempt >= 2 || !mayRetry()) throw e;  // budget or ceiling reached
      window.recordRetry();
      await sleep(backoffWithJitter(attempt, e));
    }
  }
}

Read what happens at the two extremes, because that is where the design earns its keep. When a single request fails against an otherwise healthy dependency, successes are plentiful, the allowance is large, and the retry proceeds — exactly the behaviour you wanted from a retry policy in the first place. When the dependency is broadly degraded, successes collapse, the allowance collapses proportionally, and the client stops retrying on its own. No deploy, no feature flag, no engineer awake at 3am. The policy that protects the dependency is the same policy that was helping individual calls a minute earlier.

A ten percent ratio is a reasonable starting point and the floor matters as much as the ratio: a low-traffic service with a pure-ratio budget can compute an allowance below one and never retry anything, so keep a small absolute minimum. The number to watch after shipping this is not the ratio itself but how often the budget is hit — a counter that is regularly non-zero means the dependency is unhealthy often enough that you are relying on retries to hide it.

Backoff, Jitter, and the Recovery Stampede

Exponential backoff without jitter does not solve the problem it is usually credited with solving. If a thousand clients fail within the same second and each computes the same doubling delay, they all wake at the same instant — you have not spread the load, you have merely postponed a spike and made it larger. Full jitter, where the delay is drawn uniformly from zero to the capped exponential value, is what actually decorrelates them.

// Two rules do almost all the work here.
// 1. The provider's own hint beats your formula, always.
// 2. Full jitter, not "exponential + a little noise". Synchronised clients
//    that all failed at the same instant must NOT wake at the same instant.
function backoffWithJitter(attempt: number, err: HttpError): number {
  const hinted = err.headers?.['retry-after'];
  if (hinted) {
    // Honour it exactly, plus a small random spread so N clients handed the
    // same value do not reconverge into one synchronised wave.
    return Number(hinted) * 1000 + Math.random() * 1000;
  }
  const capped = Math.min(BASE_MS * 2 ** attempt, MAX_BACKOFF_MS);
  return Math.random() * capped;            // full jitter: uniform over [0, cap]
}

// The recovery stampede is the failure people forget. When a provider comes
// back, every client that was backing off retries at once and knocks it over
// again. Ramp concurrency back up instead of restoring it in one step.
function onProviderRecovered() {
  concurrencyLimit = Math.max(1, Math.floor(steadyStateLimit * 0.1));
  const ramp = setInterval(() => {
    concurrencyLimit = Math.min(steadyStateLimit, Math.ceil(concurrencyLimit * 1.5));
    if (concurrencyLimit >= steadyStateLimit) clearInterval(ramp);
  }, 5_000);
}

The second half of that snippet addresses the failure that catches teams after they think they have finished: the recovery stampede. A provider comes back, every client that was backing off resumes at once, the first moment of restored capacity absorbs the fleet’s entire queued backlog, and it falls over again. From the outside this looks like a provider that keeps flapping. From the inside it is your own traffic pattern. Ramp concurrency back up over tens of seconds instead of restoring it in a single step, and the flapping stops.

One more rule that costs nothing: when the response carries a reset hint, honour it exactly rather than applying your formula. Your backoff is a guess about when capacity returns; the header is the answer. Add a small random spread on top so that a thousand clients handed the same value do not immediately re-synchronise around it.

Deadlines Beat Timeouts, and Retry at Exactly One Layer

A per-attempt timeout answers “how long do I wait for this call” and nobody is answering the question that matters: how long has the person waiting actually been waiting. Set the deadline once at the entry point — the moment after which the answer is worthless — and pass it down through every layer, deriving each attempt’s timeout from the time that remains. Any retry that cannot complete before the deadline should never be issued. That single check eliminates most of the wasted load a system generates during an incident, because it stops work on behalf of callers who have already gone.

The companion rule is to retry at exactly one layer. Retry logic is easy to add and invisible once added, so it accumulates: the SDK retries, your service wrapper retries, the API gateway retries, and the client app retries. Each policy is defensible in isolation and they multiply — three layers of three attempts is twenty-seven requests from one user action. Pick the layer that knows enough to decide (usually the one nearest the provider, which can see the status code and the reset hint), disable retries everywhere else, and write it down, because the next person to add a “resilience” wrapper will not know.

Six Ways a Retry Policy Causes the Incident

In every row below the retry logic did exactly what it was configured to do, and in most of them the wire reports success at the moment the damage is done.

The Mistral Trap: Two Deployment Targets, One Retry Policy, and Only One of Them Has Elastic Capacity

Mistral is consumed two ways that demand opposite retry behaviour, and teams routinely ship one policy for both. Against the managed platform, a 5xx is usually transient and a retry is a reasonable bet on someone else’s spare capacity. Against a self-hosted or dedicated deployment of the open-weight models, there is no spare capacity — the pool is exactly the GPUs you are paying for, and a 5xx generally means that pool is already saturated. Retrying into it is not a bet on recovery, it is additional load applied to the precise resource that is failing, and it converts a queue that would have drained into one that will not.

So the retry budget has to be set per target, not per client library. On elastic managed capacity a modest budget is defensible. On fixed capacity the correct budget is close to zero and the correct behaviour is to shed: return a fast, honest failure to the caller and let the queue drain. If you run both — managed for burst, self-hosted for baseline, which is the most common Mistral topology — the failover path between them must not inherit the retry count of the path it came from, or a single request quietly becomes nine.

Third: Mistral’s smaller and larger models are frequently used behind the same abstraction with the same timeout, and a retry policy tuned against the small model is far too aggressive for the large one. A timeout set just above the small model’s p99 will fire mid-generation on the large model under load, cancelling work that was going to succeed and immediately re-requesting it — the worst possible pattern, because you pay for the abandoned tokens and add load at the same moment. Timeouts and retry budgets belong to the model, not to the SDK client.

What to Measure

Five numbers, and the first one is the one almost nobody has. Track attempts per successful call — not a success rate, a ratio — because it is the only metric that makes amplification visible while everything else still looks green. Then the count of budget exhaustions, the count of attempts skipped because the deadline had already passed, the distribution of backoff delays actually slept (if it is not spread, your jitter is not working), and the share of retries that eventually succeeded. That last one is the honest test of whether the policy earns its cost: if retries almost never turn into successes, you are paying load and latency for nothing and the budget should be smaller.

What none of those five can tell you is whether api.mistral.ai was actually degraded. Every one of them is measured from inside your own client, which means during an incident they describe your reaction rather than the cause — and a retry storm and a genuine provider outage produce almost identical internal graphs. Independent external monitoring is what separates “they are down” from “we are hitting them too hard,” and those two situations call for opposite responses.

Frequently Asked Questions

Related Guides

• Mistral AI API Rate Limits Explained
• Mistral AI API Error Codes Explained
• Mistral AI API Failover Patterns
• Fixing Mistral AI API Timeout Errors
• Mistral AI API Latency and Performance
• Mistral AI Webhooks and Async Job Delivery
• Live Mistral AI Status Check