Why not a leaderboard

Benchmarks measure the average task. You have one task, and awkward inputs. The gap between those two things is the entire reason products fail after a good demo.

Tasks as data, not branches

Each probe is a record with a skill, so adding one is appending to a list rather than editing the runner:

One method per provider

A provider takes image bytes and a prompt and returns text. Everything else — timing, error capture, result shape — lives in the benchmark, so a provider stays tiny and adding the next model is twenty lines.

Failure isolation

A provider that times out records an error and the run continues.

One timeout must never cost you a whole evaluation run.

That is a five-line design decision and it will save you an afternoon at least once.

Offline tests

No model, no network, no API key — a stub provider instead. Your test suite should never depend on somebody else's uptime to tell you whether your own code works.

Time one full run

Run every probe you have against the stub provider, end to end, and time it on a wall clock rather than a profiler.

This is not a number about a model — that is what "no number at the end" meant up top. It is a number about your instrument. An eval you can run in twenty seconds is one you run after every change; an eval that takes ten minutes is one you run twice and then start guessing instead.

Record what yours costs today, before a single real provider is wired in, because from here it only gets slower.

Close the experiment loop

Turn the timing result into the next run rather than leaving it as a number.