The eval you actually need syllabus
A benchmark measures the average task. You have one task, and awkward inputs. This is how to measure that instead.
Module 0 — Why leaderboards lie
Not because they're dishonest, but because they answer a question that isn't yours.
- The average task is not your task — A number that moved three points on a benchmark may move zero on your problem. (2 min)
- Three failures a benchmark will never show you — Refusals, near-misses and confident nonsense, none of which move an accuracy number. (2 min)
Module 1 — Tasks as data
Adding a probe should be appending to a list, never editing a runner.
- One record per probe — The shape that lets you add the hundredth test case as cheaply as the second. (2 min)
- Scoring without fooling yourself — Exact match is too strict, a model judge is too kind, and both are measurable. (2 min)