Local or nothing
Run open-weight models on hardware you own, and know what they cost you in watts and seconds.
What this course is
"Just run it locally" is advice that skips every hard part: what fits in the memory you have, what quantisation actually costs you on your task, and what the electricity bill looks like when the thing runs all day.
This course does the arithmetic. Module 0 is free, because you should be able to find out whether your hardware is a non-starter without paying anyone.
What you'll build
- A fit table for your own machine — what runs, at what context length, at what speed.
- A quantisation curve: quality against memory, measured on your task rather than on perplexity.
- A cost model in watts and wall-clock, next to the hosted API's price for the same work.
Module 0 — What actually fits
Before anything else: whether your hardware is in the game at all. Free, and it should be.
- Reading a model card for what it doesn't say — Parameter counts are marketing. Memory at your context length is the constraint. (1 min)
- VRAM arithmetic you can do in your head — Weights, KV cache, overhead — and which one surprises you. (2 min)
Module 1 — Quantisation without the folklore
Everyone repeats that 4-bit is fine. Measure it on your task instead.
- What quantisation costs you, measured — Not perplexity. Your harness, your probes, one run per format. (1 min, paid)
- Picking a format for your hardware — The right answer differs on Apple silicon, on a consumer GPU, and on a server card. (1 min, paid)
Module 2 — The cost model
Where local stops being cheaper, stated as a break-even you can defend.
- Tokens, watts and wall-clock — A machine that runs all day has an electricity bill and a depreciation schedule. (1 min, paid)
Related writing
- How to measure LLM latency: find which leg owns the two seconds — A voice command that takes two seconds to fire has four legs and only one of them is slow. Here is how to time each one separately before you optimise anything.
- What actually fits in 8GB of VRAM — The arithmetic behind model size, quantisation and context length — so you can work out what runs on the GPU you already own before downloading 40GB to find out.
- Local or hosted: where the crossover actually is — The break-even between a GPU you own and an API you rent is a division, not an argument. Here is the calculation, with the terms people leave out.