Evaluating AI output

You don't need a benchmark, you need five probes

Public evals measure the average task. Yours has awkward inputs and one definition of wrong. Building the small, specific eval that catches your failures in an afternoon.

2026-07-28 · 4 min read

Someone ships a summariser. It works on the six documents they tried. Two weeks later a colleague mentions it dropped the deadline from a contract summary, and there is no way to know whether that was always happening, whether last Tuesday's prompt edit caused it, or whether it will happen again tomorrow.

The reflex is to look for a benchmark. There are dozens, they are free, and they will tell you how some model performed on a summarisation exam written by researchers. None of them contain your contracts, and none of them share your definition of a bad summary — which for you is very specific: it silently dropped a number.

What you need is smaller than a benchmark and takes an afternoon.

A probe is an assertion, not a score

A probe is one property you can check automatically on any output. Not "is this summary good" — that needs a human, or a judge model, and both are a later problem. Something mechanical:

// Probe: every date in the source survives into the summary.
const dates = (s) => s.match(/\b\d{1,2} \w+ \d{4}\b/g) ?? [];
const missing = dates(source).filter((d) => !summary.includes(d));
// pass when missing.length === 0

That is the whole idea. Five of those, run over twenty documents, and you have replaced a feeling with a percentage you can watch move.

The five that earn their place in almost any extraction or summarisation task:

1. Nothing invented. Every number in the output appears in the input. Catches the most damaging failure mode there is, and it is a set difference.
2. Nothing dropped. Every date, total, or named party in the input appears in the output. The contract-deadline bug, caught by four lines.
3. Shape. It is valid JSON, it has the six keys, the amount parses as a number. Half of all production failures are a model deciding to be conversational inside a field.
4. Length. Within the band you asked for. A model that starts returning 800 words where it used to return 200 has changed behaviour whether or not the content is right.
5. Refusal. The output is not an apology. "I'm sorry, I can't help with that" passes every other probe you write, which is exactly why it needs its own.

None of these tell you the summary is good. All of them tell you when it has become wrong, which is the thing you actually need to know at 6pm on a Friday.

Twenty documents, chosen badly on purpose

The instinct is to grab twenty representative documents. Do not. Representative documents pass.

Pick these instead:

• The three that broke it before. Every one of them, forever. This is the whole regression suite, and it is the highest-value thing in the file.
• The two longest, because truncation failures only appear near the context limit.
• The two ugliest — the scanned one, the one with a table, the one where someone typed the total in the wrong column.
• The empty one. And the near-empty one, with a header and no body.
• Two in the wrong language, if that can happen to you.
• The rest ordinary, so the number means something.

A set weighted towards awkwardness is the point. You are not estimating average quality; you are trying to make it fail in the office rather than at the customer.

The number is a ratchet, not a grade

Run the probes, get a figure — 82% of checks passing across 20 documents — and write it down with the date and the prompt version. That figure is close to meaningless in absolute terms. Its entire value is comparative:

• Change the prompt, rerun, see 91%: keep the change.
• Change the prompt, rerun, see 74%: you have just been saved from shipping a regression that six ad-hoc tries in the playground would have missed.
• Swap the model, rerun, see 83% instead of 82%: the upgrade everyone was excited about did nothing for your task, and you now know that for the cost of one command.

That last case is the one that pays for the afternoon. Model releases arrive with impressive benchmark deltas, and the honest answer to "should we switch" is almost always "our number moved by one point, so no."

Where this stops working

Be clear about the limits, because an eval you trust too far is worse than none:

• Probes cannot tell you the output is good. They tell you it is not obviously broken. A summary that keeps every date and invents nothing can still be useless.
• Twenty documents is a small sample. A change of two percentage points is noise. Treat only large moves as signal, or grow the set.
• Probes rot. When a real failure gets through, that is not an eval failure — it is a missing probe, and the fix is to add it and rerun the history.

That last habit is what turns this from a one-afternoon exercise into something that compounds. Every bug becomes a permanent check. After six months, the file is a map of every way this task has ever gone wrong, and no failure that has happened before can happen twice without you seeing it.

That is worth more than any leaderboard, and it started with five regexes.