AI that moves things

How to measure LLM latency: find which leg owns the two seconds

A voice command that takes two seconds to fire has four legs and only one of them is slow. Here is how to time each one separately before you optimise anything.

2026-08-03 · 4 min read

You say "lights on". About two seconds later, the lamp comes on. It feels bad, in the specific way that makes people stop using a thing they built, and the instinct is immediate: the model is too slow, swap it for a smaller one.

That instinct is wrong about half the time, and the half it is wrong about is expensive — you spend an evening swapping models and the two seconds barely move, because the two seconds were never in the model.

A voice-to-relay pipeline has four legs:

1. Capture — from the moment you stop speaking to the moment the audio buffer closes.
2. Transcription — audio in, text out.
3. Intent — text in, a decision out.
4. Actuation — the HTTP call to the plug, and the relay physically closing.

Any one of them can own the delay. Until you have four numbers, you do not know which.

Time the legs, not the pipeline

The measurement is unglamorous. Wrap each leg, record a monotonic timestamp on either side, print the deltas.

const marks = [];
const mark = (label) => marks.push([label, performance.now()]);

mark("capture-end");
const text = await transcribe(audio);
mark("transcribed");
const intent = await decide(text);
mark("decided");
await fetch(plugUrl, { method: "POST", body: '{"turn":"on"}' });
mark("actuated");

for (let i = 1; i < marks.length; i++) {
  console.log(`${marks[i][0].padEnd(12)} ${Math.round(marks[i][1] - marks[i - 1][1])}ms`);
}

Use performance.now() rather than Date.now() — it is monotonic, so an NTP correction mid-run cannot hand you a negative leg.

Run it ten times, not once. The first run of anything includes a cold model load, a DNS lookup, and a TLS handshake, and if you measure once you will draw a conclusion about a cache that will never be cold again.

What a real split tends to look like

Here is one run on my laptop, hosted transcription, hosted model, a Shelly plug on the same LAN:

The model — the thing everyone blames — is 310 ms of a 1,630 ms wait. Nineteen percent. If you replaced it with something twice as fast you would save 155 ms, which nobody in a room can feel.

The biggest leg is the one nobody thinks of as a leg at all: deciding you have finished speaking. That 800 ms is a voice activity detection timeout, and it is not compute. It is a constant somebody chose, sitting in a config file, waiting for a gap of silence long enough to be confident you are done.

Your split will differ — a worse network moves the weight to transcription, a busy API endpoint moves it to intent. That is the point. The number is not transferable, which is exactly why you have to take it yourself.

The cheap fix nobody tries first

Once the table exists, the first optimisation is obvious and free: drop the silence threshold from 800 ms to 400 ms. You lose nothing except tolerance for people who pause mid-sentence, and for a two-word command like "lights on" nobody pauses mid-sentence.

That is 400 ms off a 1,630 ms pipeline for a one-line change — more than swapping the model for something twice as fast, at none of the cost.

The second fix is nearly as boring: overlap the legs. Transcription can start streaming while you are still talking, so its 480 ms hides inside the capture window rather than following it. Now most of two legs cost you nothing.

None of this is clever. It is just what happens when you have four numbers instead of a feeling.

When the model genuinely is the problem

Sometimes the table comes back with intent at 1,400 ms of 1,900 ms, and then the instinct was right all along. Even then, "use a smaller model" is one of three options, and often the worst one:

• Move it local. A 3B model on the machine already in the room removes the round trip entirely, and for classifying a phrase into one of six intents, a 3B model is plenty. This is the point at which the local-or-nothing arithmetic is worth doing properly.
• Skip the model. "Lights on" does not need inference. A regex handles the ten phrases people actually say, and the model becomes the fallback for the eleventh. Median latency drops to nothing because the median case never calls it.
• Then make it smaller, and re-measure, because a smaller model that gets the intent wrong once in twenty costs you far more than 300 ms.

The habit worth keeping

Instrument before you optimise. It sounds like advice from a textbook, and everyone nods at it and then goes and swaps the model anyway, because swapping the model feels like progress and printing four numbers does not.

The four numbers take fifteen minutes. They will tell you that your latency lives in a silence timeout, or a TLS handshake you are redoing every request, or a smart plug that is answering via a server in another country. All three are real things I have found this way, and none of them would have been fixed by a faster model.

More on ai that moves things

• From a token to a servo: the last ten centimetres — Everything between a model's output and something physically moving — the boundary, the failsafe, and why the interesting engineering is all on the hardware side of the API call. (2026-06-23, 4 min)