Twenty minutes to break it

Five probes worth writing

Read your transcript properly and write down, before looking at any output:

1. A decision probe. One decision that was definitely made. Does the summary contain it?
2. An attribution probe. One commitment made by a specific person. Is it credited to them?
3. A negative probe. Something discussed and explicitly rejected. Does the summary present it as agreed?
4. An omission probe. The most consequential thing said. Is it there at all?
5. A fabrication probe. Something plausible that was never said. Does the summary contain it anyway?

Each one is a yes or no. That is the whole method, and it is the same method the eval course spends four lessons doing rigorously.

A probe, written properly

The difference between a probe and a vague feeling about quality.

Run all five

Write your five probes, then check the output you already have against each one. Yes or no, no partial credit — "sort of mentioned it" is a no, because in six months you will not remember what sort of meant.

Then run the same transcript twice more. Same input, same prompt. Note whether the five answers stay the same.

How many of five did it pass?

Your first real evaluation number. Record the count from the first run.

Did it give the same answers three times?

Record yes or no. If the same input produced different probe results across three runs, your summariser is not one system — it is a distribution, and a single demo tells you even less than you thought.

Three out of five

It passes three. It fails the attribution probe and the fabrication probe. You have twenty more minutes.

Forty minutes, one finding

Before the coffee goes cold.