Mistral API Capacity Planning: Sizing Before the Traffic Arrives

There is a date on a calendar — a launch, a campaign, a seasonal peak, a customer going live — and someone has asked whether the Mistral integration will hold. Every reliability control you have already built is the wrong tool for answering that, because all of them are reactive by construction. A retry budget decides what to do after a failure. A circuit breaker decides when to stop calling. A failover route decides where traffic goes instead. None of them can tell you whether the traffic you are about to create fits in the space you are allowed to use.

That is the inversion that makes capacity planning a separate discipline rather than a harder version of monitoring. Everything else in your reliability stack is evaluated against requests that happened. This is evaluated against requests that have not been sent yet, using a number you have to write down in advance, and the feedback loop is a single event that either goes fine or becomes an incident you scheduled yourself.

Four Ceilings, and Only One of Them Binds First

“Do we have enough capacity” is not one question. Your Mistral account has several independent allowances, they are consumed at different rates by the same traffic, and the plan is only as good as its identification of which one runs out first. Teams almost always forecast the one that is easiest to count — requests — and it is almost never the one that binds.

The concurrency row is the one worth sitting with, because it is the only ceiling that is a product rather than a sum. In-flight requests equal arrival rate times average duration — so a release that adds nothing to your traffic but increases output length by 40 percent has increased your concurrency requirement by 40 percent. This is also why a capacity plan built during a fast week quietly under-provisions for a slow one: the provider getting slower consumes your headroom exactly as if your users had multiplied.

Forecast From Product Events, Not From Last Month's Requests

The instinct is to extrapolate the request graph. It is the wrong input for a launch, because a launch is precisely the event that breaks the relationship between last month's traffic and next month's. The durable input is the product event — a user doing a thing — and a conversion factor from that event to tokens, which you measure once and re-measure whenever the prompt changes.

// Capacity worksheet. The input is a product event, not a request count,
// because a launch is exactly the event that breaks the relationship
// between last month's requests and next month's.

const PLAN = {
  // 1. Demand, in product terms. Marketing owns this number, not you.
  activeUsersAtPeakWeek: 40000,
  aiActionsPerUserPerDay: 3.2,

  // 2. Conversion factor: MEASURE this from production, never estimate it.
  //    Re-measure whenever the prompt template or retrieval config changes.
  inputTokensPerAction: 2400,   // system + retrieved context + user text
  outputTokensPerAction: 500,

  // 3. Shape. Traffic does not arrive uniformly and daily averages are
  //    useless for sizing. 4x is typical for consumer traffic concentrated
  //    in waking hours; an email blast or push notification is far spikier.
  peakToMeanRatio: 4.0,

  // 4. Your own amplification. Retries are traffic, and they arrive when
  //    the system is least able to absorb them.
  retryInflation: 1.2,

  // 5. Latency, from YOUR measurements at p95 - not the mean, and not the
  //    number on the marketing page.
  p95LatencySeconds: 3.5,
};

const actionsPerDay = PLAN.activeUsersAtPeakWeek * PLAN.aiActionsPerUserPerDay;
const meanActionsPerSec = actionsPerDay / 86400;
const peakActionsPerSec = meanActionsPerSec * PLAN.peakToMeanRatio * PLAN.retryInflation;

const tokensPerAction = PLAN.inputTokensPerAction + PLAN.outputTokensPerAction;

const peakRPM = peakActionsPerSec * 60;
const peakTPM = peakActionsPerSec * tokensPerAction * 60;

// Little's Law. This is the number nobody forecasts, and it is a PRODUCT:
// latency doubling has the same effect on it as traffic doubling.
const concurrency = peakActionsPerSec * PLAN.p95LatencySeconds;

// Compare each against the corresponding account ceiling at 70%, and the
// row with the smallest ratio is your real constraint - the only one worth
// spending a quota request on.
const need = { peakRPM, peakTPM, concurrency };
const ceilings = { peakRPM: 6000, peakTPM: 2000000, concurrency: 200 };

for (const key of Object.keys(need)) {
  const utilisation = need[key] / ceilings[key];
  const verdict = utilisation > 0.70 ? 'ACTION REQUIRED' : 'ok';
  console.log(key, Math.round(need[key]), (utilisation * 100).toFixed(0) + '% of ceiling', verdict);
}

Three things in that worksheet are the ones that get skipped. The peak-to-mean ratio is the first: daily totals are useless for sizing because nothing arrives uniformly, and a plan built on a daily average is wrong by whatever your real peak multiple is — typically three to six for consumer traffic concentrated in waking hours, and far higher for anything triggered by an email send or a push notification. The second is that retries are traffic. A 20 percent retry rate is 20 percent more load arriving at exactly the moment the system is least able to absorb it. The third is that concurrency is derived, not assumed: the number of simultaneous requests you must support falls out of arrival rate and latency, and no amount of counting requests per minute will produce it.

Why 70 Percent Is the Target, Not 95

Planning to consume 95 percent of a ceiling feels efficient and is the single most reliable way to turn a successful launch into an incident. The reason is not caution, it is arithmetic: utilisation and queueing are not linearly related. As a shared resource approaches saturation, waiting time rises super-linearly, so the last few percent of a ceiling costs vastly more latency than the first few. At 70 percent, a 40 percent surprise is absorbed. At 95 percent, a 6 percent surprise is an outage.

That last row is the expensive one, and it is expensive in a way that does not show up in a capacity review. A system tuned to the edge of its allowance generates throttling that is indistinguishable, from inside your own logs, from a provider degradation — so every burst starts with an hour of investigating the wrong system. Independent status data collapses that hour to a glance, which is a capacity argument as much as a monitoring one. For the mechanics of what to do once you are being throttled, see the Mistral rate limits guideand the retry budget guide.

Load Testing a Provider You Do Not Own

The natural next step after a forecast is to prove it with a load test, and this is where capacity planning against a hosted API diverges sharply from capacity planning against your own infrastructure. You can hammer a service you own until it breaks and learn exactly where the break is. You cannot do that to Mistral, for three reasons that are worth separating because they have different remedies.

• The result is not repeatable. Serverless inference capacity is shared. A test at 2am on a Sunday and the same test on a Tuesday afternoon measure two different systems, and neither is the one your launch will meet.
• It costs real money at real prices. A load test that generates production-scale tokens generates production-scale spend, and unlike a test against your own hardware there is no marginal-cost-of-zero option.
• Sustained synthetic load may breach the terms you agreed to. Ask the vendor before you generate it, especially if the plan is to discover the ceiling by exceeding it.

What is worth testing is everything on your side of the wire, which is also where your unknowns actually live. Point your load generator at a local mock that returns realistic token counts and realistic latency distributions, then verify that your admission control refuses the right requests, that your queue depth stays bounded, that your connection pool does not become the constraint before the provider does, that a p99 latency doubling does not cascade into timeouts everywhere, and that the degradation path activates when it should. Reserve real traffic for a short, agreed, small-multiple smoke test that confirms the shape of the numbers rather than finding the wall.

// The load test that is actually worth running: everything on YOUR side
// of the wire, against a mock with realistic token counts and a realistic
// latency DISTRIBUTION. A mock that always answers in 200ms proves nothing,
// because the failure you are hunting is a queue forming behind a slow tail.

// 1. Latency profile taken from production percentiles, not a constant.
function syntheticLatencyMs() {
  const r = Math.random();
  if (r < 0.50) return 900;    // p50
  if (r < 0.95) return 3500;   // p95
  if (r < 0.99) return 9000;   // p99
  return 30000;                // the tail that fills your connection pool
}

// 2. Assertions that must hold at 1.4x the forecast peak:
//    - admission control refuses the correct requests, and refuses FAST
//    - in-flight count never exceeds the configured ceiling
//    - queue depth is bounded; it must not grow without limit
//    - connection pool is not the constraint before the provider is
//    - the degradation path fires at the utilisation threshold, on its own
//    - p99 user-visible latency stays inside the deadline, or times out
//      cleanly rather than piling up

// 3. Then, and only then, a small agreed smoke test against the REAL API
//    at a low multiple. Its job is to confirm the SHAPE of your numbers -
//    tokens per request, latency percentile, headers reporting remaining
//    allowance - not to discover the wall by hitting it. Sustained
//    synthetic load against a shared provider may breach your agreement,
//    costs production money at production prices, and is not repeatable
//    because the capacity is shared with everyone else's Tuesday.

The Quota Request Is a Calendar Item

If the forecast exceeds the allowance, the remedy is not an engineering task, and treating it as one is how launches slip. A limit increase is a request into someone else's queue, subject to their review and their working week, and there is no amount of urgency on your side that shortens it once the date is close. Put it on the plan the week the forecast is finished, not the week before launch.

Requests are approved faster when they answer the reviewer's questions without a round trip. That means: the specific limit and model you want raised, stated in the provider's own units; the number you are asking for and the arithmetic behind it, not a round figure; the date it must be in place and the event driving it; your current utilisation as evidence you are already using what you have efficiently; and a sentence on what you are doing to be a good tenant — backoff, caching, budgets. The full mechanics live in the Mistral quota increase guide. Assume the answer might be no, or smaller than you asked, and plan the launch so that outcome is survivable rather than fatal.

The other lever, available immediately and requiring nobody's approval, is to need less. Caching identical and near-identical requests, trimming context that is not earning its tokens, and routing the easy majority of requests to a smaller model all reduce the forecast rather than the ceiling — and a forecast reduced by 30 percent is worth exactly as much as a limit raised by 43 percent, without the lead time. See the response caching guideand the cost routing guide.

The Mistral Trap: The Forecast Becomes a Purchase Order

On pure pay-as-you-go pricing a capacity forecast is a planning artefact: if it is wrong you spend more or less than you expected and the system keeps working. The moment any part of your traffic sits on reserved or provisioned throughput — which is the normal shape of a serious Mistral deployment, and the whole point of a cloud-hosted or self-deployed model — the forecast stops being a planning artefact and becomes a commercial commitment for a term.

That changes the cost of being wrong in both directions, asymmetrically. Under-forecast and you do not get a bill you did not expect; you get a hard throughput wall at the size you bought, and the remedy is a commercial conversation on the vendor's calendar rather than an autoscaler on yours. Over-forecast and you have bought capacity for the term whether or not the traffic arrives, and unused reserved throughput does not refund itself the way an unspent pay-as-you-go budget does.

The plan that survives this is a split one: size the reserved tier at the load you are confident about — the steady-state floor, not the peak — and route everything above that line to a pay-as-you-go path or a second provider. That way the expensive commitment is made against the part of the forecast that is nearly certain, and the uncertain part rides on the pricing model designed for uncertainty. Note also that a self-deployed or cloud-hosted Mistral deployment may not appear on the first-party status page at all, so capacity incidents on that path are yours to observe.

Six Ways a Capacity Plan Fails Quietly

Capacity failures are unusually hard to see because most of them do not produce an error. Five of the six below leave your Mistral error rate flat, which is why they are typically discovered by a user complaint or an invoice rather than by an alert.

The pattern across all six: the metric that would have caught it is a rate against a known ceiling, and almost every default dashboard plots absolute values with no ceiling line drawn. Adding the limit as a horizontal reference to four graphs — requests, tokens, concurrency, spend — converts them from descriptions into predictions, which is the entire job of a capacity plan.

Ship the Degradation Path, Because the Forecast Is Wrong

Every forecast in this guide is an estimate with an error bar, and the deliverable that determines whether being wrong is survivable is not the spreadsheet. It is the behaviour of your system at 110 percent of plan, decided in advance and written in code, because the alternative is deciding it during the event with the whole company watching.

Rank your traffic by what has to survive: interactive user-facing requests first, then background enrichment, then batch and analytical work, then anything speculative or prefetched. Then define the rungs the system steps down through as pressure rises — pause the lowest tier of work, shorten context, drop to a smaller model, serve cached or partial answers, and only at the end refuse with an honest message and a retry hint. Each rung should be a flag that can be pulled by an operator in seconds and, better, an automatic response to a signal you already have: sustained utilisation above your target, or a rising 429 rate.

The overflow option is the other half. Routing surplus traffic to a second provider on a separate account gives you a ceiling that is genuinely independent of this one, which is worth more than any amount of headroom on a single account — provided you have decided in advance whether the second destination's answer is good enough to send to a user. That decision is its own problem; see the fallback ranking guideand the failover guide.

After the event, the plan is only worth keeping if you close the loop: compare forecast to actual on every unit, write down which conversion factor was wrong and by how much, and carry that ratio into the next forecast. A capacity plan that is never reconciled is a document; one that is reconciled twice is a model.

Frequently Asked Questions

Related Mistral Guides

• Mistral API Rate Limits
• Mistral API Quota Increase Guide
• Mistral API Retry Budget Guide
• Mistral API Latency Guide
• Mistral API Cost Attribution Guide
• Mistral API Response Caching Guide
• Mistral API Multi-Tenant Guide