The ceiling is a measurement, not an aspiration

The frontier model tells you the best achievable score on your probe set with today's technology and an unlimited network. Everything else in this course gets compared against it.

If the frontier model scores 6/9 on your probes, you have learned something far more valuable than any local-versus-hosted comparison: your task is harder than you thought, and no amount of local optimisation was ever going to save you. Better to find that out this afternoon than after the arm is on the bench.

Model ids, prices and access tiers move constantly. Check the date on anything in this course and follow the link to current docs before you budget against it.

Wire exactly one provider

Twenty lines. Image bytes and a prompt in, text out. No retries, no timing, no caching — those belong to the harness, and putting them in the provider is how two models stop being comparable.

Score it the boring way

Resist automating the grading on the first pass. Read the nine responses yourself, against your expected field, and mark them. It takes fifteen minutes and it will teach you that three of your probes were badly worded.

Automated grading is worth building after you have graded by hand once, because only then do you know what you are automating. Build it before, and you automate your own misunderstanding.

The ceiling score

Score out of your probe count, on the frontier model, graded by hand. Put the model id and the date in the note — this number expires.

What one full run cost

Total cents for the run, across every probe. Divide by the probe count if you want a per-call figure, but record the total — that is the number you will multiply later when you work out what a machine that looks at the world four times a second costs to run for a day.

The ceiling comes back high

The frontier model scores 8/9. Your one failure is the abstention probe — it invented an answer. What does that change about the rest of the course?

What the ceiling told you

Three numbers and one shape of failure. Turn them into a decision before you move on.