Knowing whether the output is any good

Public benchmarks measure the average task. You have one task and a pile of awkward inputs. How to build the small, specific eval that actually catches your failures.

A leaderboard tells you how a model did on somebody else's exam. It cannot tell you whether it drops the second invoice number when a PDF has two, because nobody else's exam contains your PDFs.

The alternative is smaller than people expect and takes an afternoon: a handful of inputs you already know the answers to, a probe for each way the thing has failed before, and a number you can rerun after a prompt change. These posts build that, and are honest about what it does not cover.

Writing on this

You don't need a benchmark, you need five probes — Public evals measure the average task. Yours has awkward inputs and one definition of wrong. Building the small, specific eval that catches your failures in an afternoon. (2026-07-28, 4 min)

Courses that take it further

The eval you actually need — Stop reading leaderboards. Build the harness that measures your task, on your inputs. (4 lessons, 8 min, 4 free)
Coffee break: the summariser you can prove is wrong — Twenty minutes to build it, twenty more to find out where it fails. The second half is the point. (2 lessons, 40 min, 2 free)