Mistral API Eval and Regression Testing Guide: Catching Silent Quality Drift (2026)

Availability monitoring answers one question: did the call come back. Everything a team builds around the Mistral API — retries, failover, circuit breakers, timeouts, schema validation — is an answer to some version of that question. None of it addresses the case where the call came back perfectly and the answer was worse than last week's, because there is no error to catch, no shape to reject and no latency to alert on. The response is valid. It is just not as good.

30-second triage: before assuming quality moved, rule out the loud failures — check live Mistral status and your error rate. Clean on both? Then you need a scored, frozen input set, not another log search. Everything below is how to build one that will not lie to you.

Four places the change can come from

“The model got worse” is a conclusion, not an observation, and it is wrong more often than it is right. The output of a Mistral call is a function of four inputs, three of which are yours, and an eval suite is worth building largely because it tells you which one moved.

Source of change	Owner	Evidence that identifies it
Prompt or template edit	You	Git history; score moves on the commit
Context or retrieval change	You	Same prompt, different inputs logged per item
Parameter or SDK default drift	You, unintentionally	Recorded request body differs between runs
Served model or serving stack	Mistral	Nothing of yours changed and the score still moved

Note what the last row requires: to conclude that the vendor changed something, you need a run from before, produced by a configuration you can prove was identical. That artefact does not exist unless you created it on purpose, which is the whole argument for doing this before you need it. Identifier-level changes are covered separately in the Mistral model deprecation guide; this page is about the case where even the identifier stayed the same.

Building a golden set that survives contact with production

A golden set is a frozen list of inputs paired with what a good answer must satisfy. Three rules separate the ones that catch real regressions from the ones that generate noise nobody reads.

Draw from production, not imagination. Hand-written test prompts encode what you expected users to send. Sample real traffic, strip anything sensitive, and keep the awkward inputs — the truncated ones, the wrong-language ones, the ones with an injection attempt in them.
Stratify, then over-sample the hard cases. If eighty per cent of production traffic is one easy task, an unweighted sample is eighty per cent blind to everything else. Give each task class a floor, and add every input that has ever caused an incident.
Store the expectation, not the output. Freezing last week's response as the correct answer bakes in whatever was wrong with it and fails on any harmless rewording. Store what must be true: fields present, value within tolerance, claim supported, format respected, forbidden content absent.

Version the set in the repository beside the code, and treat additions as a re-baselining event — adding twenty hard items will drop the aggregate score, and if that drop is not labelled it will be investigated as a regression.

📡

Recommended

Rule Out the Loud Failure First

External checks against your AI endpoints tell you within seconds whether an availability event is in play — so a quality investigation starts from a clean baseline instead of chasing a provider incident nobody logged.

Try Better Stack Free →

Graders, in the order you should reach for them

Most teams start with a model judge because it feels general, and pay for it in flaky scores. The cheap deterministic checks catch the majority of real regressions and never drift, so they go first.

1. Programmatic assertions

Schema validity, required fields, numeric tolerance, label in the allowed set, length ceiling, forbidden strings absent. Deterministic, free, no drift. Cover everything you can here.

2. Reference comparison

For tasks with a defensible right answer: exact match on extracted values, set overlap on entities, or a similarity threshold. Cheap, but only as honest as the reference.

3. Model judge, pinned

For faithfulness, tone and helpfulness. Pin the judge to a dated version, give it a rubric with explicit anchors, and keep a calibration set it must score correctly.

If you run some traffic through a self-deployed Mistral build and some through the hosted API, they are two systems and need two baselines — the same weights under different serving configurations do not produce comparable scores.

# One eval run. Record the configuration, not just the score.
#   host:  api.mistral.ai
#   model: mistral-large-latest   <- pinned, never an alias
#
# result row written per item:
#   item_id, task_class, model_id, sdk_version, base_url,
#   temperature, max_tokens, prompt_sha, context_sha,
#   assertion_pass, judge_score, judge_model_id, run_id, run_ts
#
# The four fields teams leave out, and why each one costs a day:
#   prompt_sha     -> "nothing changed" is unprovable without it
#   context_sha    -> retrieval moved, the prompt did not
#   sdk_version    -> a client upgrade changed a default
#   judge_model_id -> the ruler changed, not the thing measured

A results row that contains only the item id and a score is close to useless six weeks later. The configuration fields are what turn a red number into a two-minute diff, and they cost nothing to emit while you are already making the call.

The Mistral trap: never run an eval against a -latest alias

Mistral publishes friendly aliases such as mistral-large-latest that point at whichever dated build is current, and they are the single most common reason a Mistral eval suite drifts without a code change. Grading through an alias means the identifier in your results file is not a fact about the model — it is a pointer that resolved to one thing in March and another in June, and your stored history silently compares two different systems. The fix is mechanical and takes minutes: pin the dated version string in the eval harness, keep the alias in production if you want the upgrades, and treat the difference between them as a scheduled comparison you run deliberately. That inverts the failure into a feature — the alias becomes a candidate you evaluate against the pinned baseline before promoting, instead of a change that arrives in your users' results first and your dashboards second.

Six ways quality degrades while every signal reads green

Each of these returns HTTP 200 with a well-formed body. That is the point: not one of them is visible to error-rate, latency or spend monitoring, and each has a different fix.

What happened	What you see	What finds it
Served build changed under a stable identifier	Nothing; all metrics flat	Scheduled run of a pinned suite
Refusal rate crept up on borderline inputs	Slightly shorter responses	Refusal-detection assertion per item
Output truncated at the token ceiling	Valid JSON, missing tail	Finish-reason check before scoring
Retrieval or context assembly regressed	Vaguer, less specific answers	Grade retrieval separately from generation
SDK upgrade changed a sampling default	More variance run to run	Recorded request body diff between runs
The judge model was upgraded	Uniform score shift across every task	Judge calibration set fails first

The last row is the one that wastes the most engineering time, because it looks exactly like a broad model regression and points at the wrong system entirely. A uniform shift across unrelated task classes is almost always the ruler, not the thing being measured.

Running it: gates, schedules and what to do with red

Two triggers, deliberately different in scope. On pull requests that touch prompts, retrieval or parameters, run the full suite and block the merge on a drop beyond the noise floor. On a schedule that has nothing to do with your deploys, run a cheaper subset against a pinned configuration — this is the one that catches a change on Mistral's side, and it only works because the configuration is frozen.

When a scheduled run goes red, resist the urge to change the prompt first. Re-run to confirm it is outside the noise floor, diff the recorded configuration against the last green run, and check the per-item flips before the aggregate. If nothing on your side moved and the flips cluster in one task class, you have a vendor-side change with evidence attached — which is exactly the artefact a support conversation needs, and the input to the Mistral incident postmortem process. Keep every run's results; the value of the suite compounds with history, and a single green run proves nothing on its own.

Frequently Asked Questions

What is a quality regression on the Mistral API, if nothing errors?

It is a change in how good the answers are, with every operational signal unchanged. The call returns 200, the body parses, latency sits where it always does, and the invoice looks normal — and yet extractions start missing a field, summaries get longer, refusals appear where they never used to, or a classifier begins favouring one label. No monitor built on status codes, latency percentiles or spend can see it, because none of those numbers moved. The only instrument that detects it is a fixed set of inputs you re-run and score, which is why teams that skip eval infrastructure invariably learn about these changes from a customer rather than from a dashboard.

How large does a golden set need to be before the scores mean anything?

Smaller than most teams assume and more carefully chosen than most teams manage. Forty to a hundred and twenty items, stratified across the task types you actually serve, will detect the size of change that matters far more reliably than a thousand items scraped at random from a log — because the random thousand is dominated by your most common easy case and says almost nothing about the edges. Build it from real production inputs, deliberately over-sample the hard and previously-broken cases, and freeze it. The number itself matters less than the discipline: a small suite you re-run every night beats a large one you run twice a year, since only the first tells you when something changed.

Can we use an LLM as the grader, or does that just move the problem?

You can, with two conditions, and without them it genuinely does move the problem. First, prefer programmatic assertions wherever they apply — schema validity, required fields present, a numeric answer within tolerance, forbidden content absent, length within a ceiling. These are free, deterministic and never drift. Reserve the model judge for what code cannot check, such as whether a summary is faithful to the source. Second, pin the judge model to a dated version and keep its own calibration set, because an unpinned judge is a measuring instrument that changes shape between runs. A judge upgrade that shifts scores by two points across the board will be read as a regression in the system under test, and teams have spent weeks debugging a model that never changed.

How often should the Mistral eval suite run?

On every change to your prompts, retrieval or parameters, and on a schedule independent of your deploys. The second half is the one teams skip and the one that matters most here: the system under test is partly somebody else's, and it can change on a day when your repository does not. A nightly run against a pinned configuration turns a vendor-side shift into an alert with a date attached, which is the difference between a two-hour investigation and a two-week argument about whether anything actually changed. Keep the scheduled run cheap enough that nobody is tempted to disable it — a subset of the golden set is fine — and run the full suite before releases.

Our scores moved by three points. Is that a regression or noise?

Assume noise until a comparison says otherwise. Sampling makes single-run scores wobble, and a small suite makes the wobble larger. Establish the noise floor before you interpret anything: run the identical configuration several times, record the spread, and treat that spread as the threshold below which no action is taken. Then compare candidate against baseline as a paired difference on the same items rather than as two aggregate numbers, and look at which items flipped. Five specific items flipping in the same direction is a real signal even when the aggregate barely moves; a two-point aggregate change with different items flipping each way is usually nothing.

Our Mistral scores dropped but nothing in our repo changed. Where do we start?

Start by establishing whether you were grading a pinned version or an alias, because that one question resolves most of these cases immediately. If the harness used a -latest alias, capture the currently resolved version, re-run the same golden set against the previously pinned dated build, and compare. A gap between them is a vendor-side change and the work is a prompt or parameter adjustment against the new build. No gap means the model is exonerated and the change is on your side of the wire — most often a prompt template edit, a retrieval or context change that altered what the model was given, or a parameter default that moved when an SDK was upgraded.

Related Mistral Guides

Separate a Quality Problem From an Outage in Seconds

An eval suite tells you the answers got worse. API Status Check tells you whether Mistral was degraded while they did — monitored from outside your infrastructure, with the timeline your investigation needs already recorded.

Start Your Free Trial →

Mistral API Eval and Regression Testing