All topics

Running AI on hardware you own

What actually fits on a consumer GPU, what it costs to run, and where local inference stops being the cheaper answer — measured rather than argued.

"Just run it locally" is advice that skips every number that decides whether it works. The interesting questions are all quantitative: how much of a model fits in the memory you have, how many tokens a second you get once it does, what the electricity costs over a year, and at what request volume an API becomes the cheaper option again.

The writing here works those numbers on ordinary hardware — a laptop, a mid-range GPU, a Pi — and shows the measurement rather than the conclusion, because your machine is not the one in the benchmark.

Writing on this

  • What actually fits in 8GB of VRAM — The arithmetic behind model size, quantisation and context length — so you can work out what runs on the GPU you already own before downloading 40GB to find out. (2026-07-21, 4 min)
  • Local or hosted: where the crossover actually is — The break-even between a GPU you own and an API you rent is a division, not an argument. Here is the calculation, with the terms people leave out. (2026-06-09, 3 min)

Courses that take it further

  • Local or nothing — Run open-weight models on hardware you own, and know what they cost you in watts and seconds. (5 lessons, 6 min, 2 free)
  • Atoms, not pixels — Get a model out of the browser and onto an arm that costs less than a phone. (29 lessons, 1615 min, 5 free)
  • Where the energy actually goes — Joules, not vibes. Measure what your own work costs before joining an argument about data centres. (3 lessons, 65 min, 3 free)