The eval you actually need
Stop reading leaderboards. Build the harness that measures your task, on your inputs.
What this course is
Every model decision you make — hosted or local, big or small, this quarter's or last — is a claim you cannot check without a way to measure your own task. This course builds that instrument, and nothing else.
It is free, all of it. It is the prerequisite for the other courses rather than a product.
What you'll build
- A probe set drawn from your awkward inputs, not somebody's test split.
- A scorer you trust, including for the cases where the right answer is "refuse".
- A baseline number you can hold every later claim against.
Module 0 — Why leaderboards lie
Not because they're dishonest, but because they answer a question that isn't yours.
- The average task is not your task — A number that moved three points on a benchmark may move zero on your problem. (2 min)
- Three failures a benchmark will never show you — Refusals, near-misses and confident nonsense, none of which move an accuracy number. (2 min)
Module 1 — Tasks as data
Adding a probe should be appending to a list, never editing a runner.
- One record per probe — The shape that lets you add the hundredth test case as cheaply as the second. (2 min)
- Scoring without fooling yourself — Exact match is too strict, a model judge is too kind, and both are measurable. (2 min)
Related writing
- You don't need a benchmark, you need five probes — Public evals measure the average task. Yours has awkward inputs and one definition of wrong. Building the small, specific eval that catches your failures in an afternoon.