Friday night: make one lamp answer to you syllabus

Cut the round trip

The transcription leg is a round trip to another continent for a two-word phrase. On a fixed command vocabulary, the smallest local model is usually enough — and "usually" is a claim you are about to check rather than accept.

What going local actually buys

It buys you the network, not the compute. A hosted transcription model is fast; what costs you is the trip there and back, plus whatever queueing happens at the other end on a busy evening.

So the win is largest on short phrases, where the round trip dominates, and smallest on long dictation, where real compute time takes over. "Lights on" is about as short as a phrase gets, which is why this build is a good place to see the effect clearly and a bad place to generalise from.

Swap in a local model

Pull the smallest English-only Whisper model — tiny.en, around 75 MB — and run the same audio through it. Two routes, pick whichever installs cleanly on your machine:

  • whisper.cpp: build it, fetch the ggml model, run the CLI (whisper-cli in current builds, main in older ones).
  • faster-whisper: a Python package, no compiler needed.

Run the first transcription before you time anything — model load is a one-off cost and including it in your measurement makes local inference look far worse than it is. Keep the model resident between commands.

The same measurement, again

Re-run last lesson's timing, same five points, same five repetitions, same phrase, same room. Record the new end-to-end number.

Same method or the comparison is worthless. If you also retuned the VAD, note that — you changed two things and the number cannot tell you which one moved it.

Reading your own result

Your local number is worse than the hosted one. What is the most likely explanation before you conclude local inference is a dead end?

When would you put it back on the network?

The build works and it is fast. Someone asks you to extend it to arbitrary phrasing — "it's a bit dark in here", "kill the lights", "can we get some light".

Publish the before and after

Post the two numbers with enough context that somebody else could get them too. A latency figure without the hardware, the model and the network it ran on is a number nobody can use — including you, in six months.

Before you go to bed

Two minutes, then close the laptop.