Did That Prompt Change Help? A Method, Not a Benchmark

How do you know the prompt change made things better? Not felt better in the three examples you tried, actually better across the traffic you serve. Most teams cannot answer that question, and the reason is rarely laziness. It is that the obvious methods, eyeballing a few outputs and checking a benchmark score, both fail in ways that are not visible from inside them. Medicine has been learning the same lesson with higher stakes, which is why it is worth reading how AI in healthcare claims hold up when graded by study design rather than by headline accuracy.

LLM evaluation is a method problem rather than a metric problem. Translation is the oldest worked example of that, and the lesson generalises: the metrics that correlate best with human judgement miss the errors that cost the most. This piece sets out the four methods that exist, what each one genuinely catches, what each one structurally cannot catch, and roughly what each costs to run and to keep alive.

Why does a single test run tell you almost nothing?

Because the variance between identical runs is larger than most of the improvements you are trying to detect.

The clearest measurement of this comes from Bjarni Haukur Bjarnason, André Silva and Martin Monperrus, who collected 60,000 agentic trajectories on SWE-bench Verified across three models and two scaffolds. Their paper on randomness in agentic evaluations reports that single run pass@1 estimates varied by 2.2 to 6.0 percentage points depending purely on which run happened to be selected. Standard deviations exceeded 1.5 percentage points even at temperature zero, because trajectories diverge early, often within the first few percent of generated tokens.

Read that against how prompt changes are usually justified. A tweak that moves your score by three points is, on that evidence, indistinguishable from noise if you measured it once, which is worth remembering before concluding that a sixth example in the prompt helped. The authors recommend estimating from multiple independent runs, doing a power analysis to work out how many runs you need for the effect size you expect, and reporting pass@k or pass^k rather than a single optimistic number.

Nobody wants to hear that the answer to "did this help" involves statistical power. But the alternative is shipping changes on the strength of a number that would have come out differently if you had run it again on the same input.

What are the four methods, and what does each miss?

There are four, they are not alternatives to each other, and a working evaluation setup usually runs two or three at different cadences.

The pairing that works for most teams is a golden set plus sampled real traffic. The golden set is your regression gate, small and fast and run on every change. The traffic sample is your discovery mechanism, run weekly, and it is where the cases you never imagined come from. Human review sits on top of both, applied to a handful of disagreements rather than to everything.

How reliable is a model acting as judge?

Useful, cheap, and biased in a specific direction you have to correct for rather than hope away.

The direction is over validation. Work by Moises Andrade and colleagues on agreement bias in models used as verifiers, revised in March 2026, found a strong tendency to over validate agent behaviour, persistent across models and not fixed by test time scaling. Their self grounded verification method, which has the model write down what good behaviour looks like before it sees the trajectory, improved failure detection by 25 percentage points and overall accuracy by 14. The size of that improvement is also the size of the problem: a judge that misses a quarter of failures is not measuring what you think it is measuring.

The second problem is that judges disagree. Fiona Lau's February 2026 study, on the inconsistency of LLM judges, ran five models over real enterprise question and answer pairs from a retrieval augmented system. It found substantial variability within models even at temperature zero, with completeness scoring the most unstable dimension, and systematic differences in strictness between models. Lowering the temperature stabilised some models and had limited effect on others.

Two practical rules fall out of that. Use a different model to judge than the one that generated the output, which is also what Anthropic's own testing guidance recommends. And treat a judge score as a comparison instrument rather than an absolute one: the number is only meaningful against the same judge, same prompt and same rubric, so pin the judge model version and change it deliberately.

How big should a golden set be?

Here the sources genuinely disagree, and the disagreement is instructive rather than confusing.

Anthropic's evaluation guidance says to prioritise volume over quality, arguing that more questions with slightly lower signal automated grading beats fewer questions graded carefully by hand. Its worked examples are large: 1,000 tweets for sentiment analysis, 500 queries for a privacy check, 200 articles for summarisation, 100 inquiries for customer service.

That advice is correct, and it is correct for a specific shape of task: narrow, categorical, automatically gradable. Sentiment has a right answer. Whether a response contains protected health information has a right answer. When the grader is a string comparison, more cases cost nothing and noise averages out.

It stops being correct the moment grading requires judgement. If a person has to read each case to decide whether the output was good, a 500 item set is not a test suite, it is a project. It will be built once, run twice and abandoned, which is the most common fate of evaluation work.

So the honest rule is conditional. Automatic grading available: go large, hundreds of cases, the advice above applies. Human or judge grading required: build 30 to 50 cases and defend that number fiercely. Thirty cases you actually run on every change beat five hundred you ran in March.

What does a maintainable golden set look like?

Concretely, for a support reply drafting feature, a 40 case set breaks down roughly like this.

Three properties make that set survive contact with a real roadmap. More than half the cases are graded by code rather than by judgement, so most of the suite runs free and instantly. The known failures slice grows by itself, because adding a case is part of fixing a bug rather than a separate task. And every case is traceable to something that happened, which means nobody has to argue about whether it belongs.

Run it three times per change, not once, given what the variance research says. Forty cases times three runs is 120 calls, which on a mid tier model is small change. The arithmetic for pricing that, and why the output column dominates it, is in our breakdown of what a real feature costs once retries are counted.

What do you actually measure, when the output is text?

This is the question that stalls most evaluation projects, because "is the answer good" is not a measurement.

The move that unblocks it is to stop scoring quality and start scoring properties. A property is a claim about the output that a person would agree or disagree with in under five seconds, and that can usually be checked without reading the whole thing. Did it answer the question that was asked. Did it invent a policy that does not exist. Did it stay inside the JSON envelope. Did it escalate when it should have. Did it use the customer's name correctly.

Anthropic's testing guidance makes the same point in a different vocabulary, arguing that most use cases need multidimensional evaluation along several success criteria at once, and giving an example that combines an F1 threshold, a safety rate, a severity distribution for the errors that remain, and a latency target. Four different measurements, none of which is "quality".

Splitting quality into properties has a second benefit that matters more over time. A single quality score cannot tell you what regressed. Five property scores can, because when a change moves one of them and leaves the other four alone, you know exactly what to look at. A composite number gives you a direction and no explanation.

The four grading mechanics available to score a property are worth knowing by name, because picking the wrong one is the usual reason an eval feels like more work than it is worth. Exact match handles categorical answers. Similarity scoring using embeddings handles consistency across paraphrases, which is the right tool for checking that a bot answers the same question the same way. Reference metrics such as ROUGE-L handle summarisation against a known good summary. A model judge handles the genuinely subjective residue. Most properties in a real feature fall into the first mechanic, and teams reach for the last one by default.

What breaks an evaluation setup after month three?

Rot, in three specific places, and all three are predictable enough to plan around.

The first is the judge prompt. It was written against a version of the feature that has since changed, so it now penalises behaviour you deliberately introduced. Symptom: scores drift downward while user complaints do not. Fix: version the rubric alongside the feature prompt and review both in the same pull request.

The second is the golden set becoming unrepresentative. Traffic shifts, a new customer segment arrives, an intent that was 2% becomes 30%, and your 40 cases still reflect last quarter. Symptom: the suite is green and production is not, which is the same silent decay that drift monitoring at the first MLOps level exists to catch. Fix: the weekly traffic sample, which is the only mechanism that notices this on its own.

The third is the model underneath the judge being silently upgraded. If you pinned a model alias rather than a version, your measuring instrument changed without a commit. Every historical score becomes incomparable and nobody notices for weeks. Fix: pin explicit versions for evaluation, always, even where you float them in production.

None of these are exotic failures. They are the standard fate of a test suite that measures something fuzzy, and the whole reason to keep the set small is that a small set survives the maintenance these three problems require.

Why do public benchmark scores not predict your task?

Three reasons, and each one alone would be enough.

The first is that a benchmark measures a model inside a harness, not a model. A June 2026 position paper by Maria I. Gorinova and colleagues argues that coding benchmarks conflate the model with the rest of the system, and that components other than the model can shift performance by amounts equivalent to the gap between model generations. Your harness is not theirs, so their number is not transferable.

The second is provenance. Leaderboard rows are submitted by the teams being measured. The SWE-bench process runs through pull requests to a public experiments repository, where submitters supply their own predictions and traces; independent verification is a separate optional step in which maintainers re run a submission on a random subset. That is a reasonable process, and it is not the same as a third party having reproduced the number.

The third is distribution. A benchmark samples tasks that someone found interesting to collect. Your traffic samples what your customers happen to send, which is skewed differently, usually much more repetitive, and contains failure modes no public set contains. We went through how far one popular benchmark's task distribution sits from ordinary engineering work in a closer look at what SWE-bench Verified measures.

Use public scores the way you would use a job title: enough to shortlist, useless for the decision. The attempt to build a test that resists this is its own subject, which we covered in a look at the benchmark built to measure real reasoning.

What should you build first?

In this order, and the first two are a day of work between them.

Start by logging inputs and outputs with a stable identifier and the version of every prompt and model involved. Without this you cannot build any of the other three methods later, and you cannot answer questions about last week at all.

Then build the format and safety slice of a golden set, because it is graded by code, it never needs a human, and it catches the class of failure that embarrasses you publicly. Ten assertions is a real gate.

Then add a weekly sample of real traffic reviewed by a person for twenty minutes. This is the discovery loop, and every case it surfaces becomes a golden set entry, so the suite grows from evidence rather than imagination.

Add a model judge last, once you have enough human labelled examples to measure the judge against. Doing it in the other order gives you a number early and no way to know whether the number means anything.

The same discipline applies to whatever sits between your model and your tools, because an evaluation that only grades the final text will not tell you that the routing was wrong. That surface belongs to the layer we described in the piece on what an orchestration layer has to own, and if your tools are reached over a documented protocol you get most of the trace for free, which is one of the reasons we run ours as an MCP server with the tool list written down.

What does good look like after a quarter?

Modest, and worth naming so you can tell whether you got there.

You can answer whether last Tuesday's prompt change helped, with a number and a run count. Your golden set has grown mostly from real incidents rather than from a planning session. You know your judge's agreement rate with a human on your own task, and you re measure it when you change judge models. And you have at least once decided not to ship something because the evaluation said no, which is the only real proof that the evaluation has authority rather than decoration.

None of that requires a platform, a vendor or a framework. It requires logs, about 40 cases, and the willingness to run things three times instead of once.