Scoring without fooling yourself

A scorer is another model of correctness. Exact match is brittle, fuzzy rules encode hidden policy, and model judges bring their own preferences. Validate the scorer before it validates anything else.

Choose the weakest scorer that works

Use exact checks for facts with one representation, deterministic rules for structural requirements, and a model judge only for genuinely semantic decisions. Every extra degree of freedom is another way for the scorer to disagree with you.

Build a double-labelled sample

Hand-label at least thirty stored responses without looking at the automated score. Then compare the two columns and inspect every disagreement before adjusting the scorer.

What limits an eval result?

Your model score is reported to one decimal place, but the scorer agrees with your labels only 84% of the time. Which number should dominate your confidence?

Record scorer agreement

Calculate agreements divided by double-labelled cases, multiplied by 100. Record the percentage and describe the largest disagreement category in the note.

Publish the eval card

Save a short eval card containing the probe source, scorer method, sample size, agreement rate, and known blind spots. This is the artifact a future model comparison must cite.

Review the instrument, not the model

Ask a peer to challenge whether another person could reproduce and interpret the result.