What actually fits in 8GB of VRAM

The arithmetic behind model size, quantisation and context length — so you can work out what runs on the GPU you already own before downloading 40GB to find out.

2026-07-21 · 4 min read

"Will it run on my card?" has an arithmetic answer, and you can get it in thirty seconds without downloading anything.

The three things using your memory

Weights. Parameters × bytes per parameter. At 16-bit that is 2 bytes each, so a 7B model is about 14GB. Quantised to 4-bit it is roughly half a byte each, so about 3.5GB. This is the number people quote, and it is the only one they quote, which is where the trouble starts.

KV cache. Every token in the context gets a key and a value stored per layer, and this grows linearly with how much you feed it. For a 7B-class model it lands around 0.5MB per thousand tokens per layer-group — call it 0.5–1GB for an 8k context, more if the model has not been built with grouped-query attention. Fill a 32k context and this alone can exceed your weights.

Overhead. The CUDA context, the runtime, activations during the forward pass, and whatever your desktop compositor is already holding. Reserve 1–1.5GB and stop being surprised.

So the actual question is not "does a 7B model fit in 8GB", it is:

weights + kv_cache(context_length) + overhead ≤ vram

The table that answers it

For an 8GB card, with 1.2GB of overhead assumed:

Model	Quant	Weights	Left for KV	Practical context
3B	Q4	~1.8GB	~5.0GB	Comfortable at 32k
7–8B	Q4	~4.4GB	~2.4GB	Fine at 8k, tight at 16k
7–8B	Q5	~5.4GB	~1.4GB	8k, and not much headroom
7–8B	Q8	~8.0GB	none	Does not fit
13B	Q4	~7.4GB	none meaningful	Technically loads, unusable
13B	Q3	~5.6GB	~1.2GB	Fits, and noticeably degraded

Two things fall out of this that are worth internalising.

Quantisation buys more than model size does. Going from Q8 to Q4 on the same 7B model frees about 3.5GB — more than the entire footprint of a 3B model at Q4. The quality cost between Q8 and Q4 is small for most tasks; the cost between "fits" and "swaps to system RAM" is a factor of ten in speed.

Below Q4, stop. Q3 and lower degrade in a way that is easy to miss on casual chat and obvious on structured output — dropped JSON keys, wandering formats, instructions half-followed. If a 13B at Q3 is the only way to fit it, run the 8B at Q4 instead. It will be better and faster.

What "does not fit" actually feels like

It does not error. That is the trap. Most runtimes will happily offload the layers that do not fit into system RAM and keep going, and you get a model that works and generates at 2 tokens/second instead of 40.

If your local setup is inexplicably slow, this is the first thing to check — not the model, not the sampler, not the prompt. Check whether every layer is on the GPU. In llama.cpp-family tooling that is the offloaded-layers line in the startup log; if it says 24/33, eleven layers are running on your CPU and that is your answer.

Then measure, because the table is an estimate

Everything above is arithmetic, and arithmetic gets you to a shortlist, not to a decision. The numbers that decide it are ones you take yourself:

Tokens per second at your real context length, not at a two-token prompt. Long prompts are slower to process and that is where you actually live.
Time to first token, separately. On local hardware, prompt processing frequently dominates everything else, and it scales with input length while generation does not.
Watts at the wall, under load, with a plug meter. A 250W GPU pulling for 40 seconds per request is a real cost per run — work out what that comes to before you conclude local is free.

Local is not free; it is fixed. That is a different and often better property, but it only wins below a certain request volume, and the crossover point is a division you can do on paper. That calculation — cost per run local versus hosted, including the electricity and the hardware amortisation — is the one worth doing before you buy a card, and it is what Local or nothing spends three modules on.

The short version

Budget: weights + KV cache + ~1.2GB, not just weights.
Q4 is the sweet spot. Q8 is usually a waste of memory; Q3 is usually a waste of time.
On 8GB: an 8B at Q4 with an 8k context, or a 3B at Q4 with room to spare.
A slow local model is usually a partly-offloaded model. Check the layer count before blaming anything else.

Take it further

Local or nothing — Run open-weight models on hardware you own, and know what they cost you in watts and seconds. (5 lessons, 6 min, 2 free)
Where the energy actually goes — Joules, not vibes. Measure what your own work costs before joining an argument about data centres. (3 lessons, 65 min, 3 free)