Write the ones you are afraid of

A probe set is useful in proportion to how much it can hurt you.

Think about the last time something you built looked fine in a demo and fell over in front of someone. What was the input? Not the category — the actual image, the actual lighting, the actual half-occluded object at the back of the shelf. That is probe one.

If you cannot remember a real failure, you have not run the thing at anybody yet. Go and do that first; an afternoon of real use is worth a week of imagined edge cases.

A skill, not a one-off

Every probe carries a skill, and the skill is what you actually report on. Ten probes across ten skills tells you nothing — one sample per cell. Ten probes across three skills tells you where you stand.

The probe it should refuse

At least one probe must have no correct answer. Ask about the strawberries in a picture with no strawberries.

A model that confidently invents an answer here will confidently invent one when it is looking at your workbench. Calibration is a skill like any other, and it is the one nobody measures because it feels like a trick question.

Fade a vague probe into a testable one

Watch this go from something you cannot score to something you can, then finish the last one yourself.

Which probe set would you rather own?

Two people build a probe set for the same shelf-picking task. Which one will still be useful in three months?

Count what came from memory

Of the probes you just wrote, how many came from something you watched fail, rather than from something you imagined might fail?

Record the honest count. This is the number that predicts whether the whole gap table is going to be worth anything, and it is the only chance you get to be honest about it before you start looking at scores.

What you avoided writing

The interesting part of a probe set is usually the probe somebody decided not to include.