Most “does this model fit my GPU” answers online are guesses. They are not. The memory an LLM uses breaks into two things you can calculate: the quantized weights, and the KV cache that grows with context. Do both, add overhead, and you get a number you can bet a GPU purchase on.

This guide walks the math with real numbers for 8B-class models. By the end: Llama 3.1 8B at Q4_K_M, 8k context, lands around 6.1–6.6 GB — comfortable on an 8 GB card, tight at 16k, gone at 32k.

What you need

A GPU (or CPU, if you are estimating RAM) with a known memory size.
A runner you will actually use: Ollama or llama.cpp — the math is runner-independent, but each reports memory differently.
Nothing else. No Python, no GPU-Z. A calculator works.

Step 1 — Pick a quantization and estimate the weight size

Start with the model’s parameter count, then divide by the quantization’s bits per weight:

weights (GB) ≈ parameters (B) × bits per weight / 8

The catch: quantizations are not exactly 4 or 8 bits. They are 4.8944 or 8.5008. The llama.cpp quantization README publishes exact bits-per-weight and file sizes per model. For Llama 3.1 8B:

Quantization	Bits/weight	File size
F16 (no quant)	16.0005	14.96 GiB
Q8_0	8.5008	7.95 GiB
Q6_K	6.5633	6.14 GiB
Q5_K_M	5.7036	5.33 GiB
Q4_K_M	4.8944	4.58 GiB
Q4_K_S	4.6672	4.36 GiB
Q2_K	3.1593	2.95 GiB

Q4_K_M is the default tag in the Ollama library for llama3.1:8b, listed as 4.9 GB — the same file in decimal GB (4.58 GiB × 1.074 ≈ 4.9 GB). When a guide says 4.6 GB and the runner says 4.9 GB, nobody is wrong: one is GiB, the other GB.

Recommendation: default to Q4_K_M. Q6_K buys noticeable quality at the cost of 1.6 GiB — on an 8 GB card that is the difference between fitting and not. Q2_K is where the accuracy curve gets ugly; skip it unless memory is your only option.

Step 2 — Calculate the KV cache

The KV cache stores what the model has read so far, and it grows with context. At FP16:

KV (bytes) = 2 × n_layers × n_head_kv × d_head × n_ctx × 2

The leading 2 covers key and value tensors; the trailing 2 is bytes per FP16 element. You need three numbers per model — layers, KV heads, and head dimension — all in the model’s config:

Model	Layers	KV heads	Head dim	KV per token
Llama 3.1 8B	32	8	128	128 KB
Qwen 2.5 7B	28	4	128	56 KB
Mistral 7B v0.3	32	8	128	128 KB
Gemma 2 9B	42	8	256	336 KB (naive)

Counts come from the Qwen 2.5 7B config, the Mistral 7B v0.3 config, and the Gemma 2 technical report (42 layers, 8 KV heads, 256-wide heads).

Two caveats. Gemma 2’s 336 KB/token is the naive figure: half its layers use a 4096-token sliding window, so the real footprint is lower. And the KV cache can be quantized: --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp, OLLAMA_KV_CACHE_TYPE=q8_0 in Ollama — halves the whole column.

Llama 3.1 8B works out to ~128 KB per token: 1 GB per 8k context, 4.2 GB per 32k.

Step 3 — Add overhead and compare with your GPU

Weights + KV + overhead. Overhead (CUDA runtime, embeddings, compute buffers) is typically 0.5–1 GB; Modal’s inference-cost guide uses a cleaner rule of thumb — raw weight size times 1.2.

Worked example — Llama 3.1 8B, Q4_K_M, on an 8 GB card:

weights      4.58 GiB
KV (8k ctx)  ~1.0 GB
overhead     ~0.5 GB
total        ~6.1 GB  → fits 8 GB

At 16k context: about 7.2–7.6 GB — tight. At 32k: about 9.3–9.8 GB, matching fast.io’s breakdown of ~9.4 GB total at 32k and ~13.6 GB at 64k.

What to expect on an 8 GB card (Q4_K_M)

Model	Weights	+ 8k ctx	+ 16k ctx	Fits 8 GB?
Llama 3.1 8B	4.58 GiB	~6.1 GB	~7.2 GB	yes / tight
Qwen 2.5 7B	~4.4 GiB	~5.5 GB	~6.0 GB	yes, comfortable
Mistral 7B v0.3	~4.5 GiB	~6.0 GB	~7.1 GB	yes / tight
Gemma 2 9B	~5.5 GiB	~7.2 GB	~8+ GB	yes at 8k only

Qwen and Gemma rows use the same bits/weight math; the rest is straight from the llama.cpp README. The localaimaster breakdown of Llama 3.1 8B Q4_K_M: 4.92 GB file + ~1.1 GB context + ~1 GB buffers ≈ 7 GB at 8k — matching the estimate.

Step 4 — Verify against the runner’s own report

Estimates are good. Confirmed is better. Both runners tell you the truth after loading.

Ollama picks a default context by your VRAM (Ollama docs): under 24 GiB gets 4k, 24–48 GiB gets 32k, 48 GiB and up gets 256k — and it recommends at least 64k tokens for agents and coding tools. Override with:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Then run ollama ps: SIZE is loaded memory, PROCESSOR is the GPU/CPU split (you want 100% GPU), CONTEXT is the window in use.

llama.cpp reads the native context from GGUF metadata with -c 0 — 8,192 for Llama 3 8B, 32,768 for Mistral 7B — and caps generation with -n. An explicit -c 32768 on an 8 GB card is where the estimate meets reality:

./llama-cli -m llama-3.1-8b-q4_k_m.gguf -c 32768 -ngl 99

Troubleshooting

CUDA error: out of memory in llama.cpp (exact log from the issue tracker: ggml/src/ggml-cuda/ggml-cuda.cu: CUDA error, current device: 0, in function alloc). Your estimate was right — the total exceeded the card. Fix in this order: lower -c, quantize the cache with --cache-type-k q8_0 --cache-type-v q8_0, reduce layers with -ngl if you are multi-GPU.
Ollama gets slow and the log says llm_load_tensors: offloading 17 repeating layers to CPU. Ollama does not fail when a model does not fit; it silently splits it — issue #8144 asks for a warning instead. Check ollama ps: a PROCESSOR of 48%/52% CPU/GPU means half the model sits in system RAM and speed dropped accordingly. Lower the context or pick a smaller quantization.
“My 8B model uses more memory than the 8 GB file size” — a common Hugging Face discussion thread. It is the KV cache: at a full 128k context, an 8B model needs roughly 16 GB of extra memory on top of the weights. Context dominates the moment you stop using the 4k default.
Your math says 4.6 GB, Ollama says 4.9 GB. GiB vs GB. 1 GiB = 1.074 GB. The model file is identical.

FAQ

Why does an 8B model need more than 8 GB of VRAM?

Usually no — the Q4_K_M weights are 4.6 GiB. What pushes past 8 GB is the KV cache at long contexts plus overhead; at 128k context an 8B model is a ~32 GB total. Quote the context size when comparing memory claims.

Does Q4 quantization mean the model is 4 GB?

No. Q4_K_M is 4.8944 bits per weight, so an 8B model is 4.58 GiB of weights before the KV cache. The “B × 0.5 GB” shortcut undersells it by about 10%.

Is a 16 GB card worth it over 8 GB?

For 8B-class models, yes: 16 GB buys you 32k+ context with room to spare (~9.4 GB total at 32k), Q6_K quants, and two smaller models loaded at once. On 8 GB you live inside the 4–8k window.

How much context can I actually run on 8 GB?

It depends on KV-per-token. Qwen 2.5 7B (56 KB/token) reaches 16k+ comfortably; Llama 3.1 8B (128 KB/token) hits its wall around 8–12k. Quantize the KV cache to q8_0 and you roughly double the room.

Sources

llama.cpp quantization README — bits/weight and GiB sizes per quantization.
Ollama context-length docs — default context by VRAM tier, 64k recommendation, OLLAMA_CONTEXT_LENGTH, ollama ps columns.
Gemma 2 technical report (arXiv:2408.00118) — layer counts, KV heads, head size, sliding-window attention.
Qwen 2.5 7B Instruct config.json — 28 layers, 4 KV heads, 3584 hidden size.
Mistral 7B Instruct v0.3 config.json — 32 layers, 8 KV heads, 32k position limit.
fast.io: llama.cpp context size — KV formula, per-8B total memory at 32k/64k/128k, -c 0 behavior.
Modal: how much VRAM do you need for inference — the ×1.2 overhead heuristic.
Ollama library — llama3.1 tags — default tag, 4.9 GB listing.
localaimaster: Llama 3.1 8B memory breakdown — 8 GB card reality check at 8k context.
localaimaster: Ollama not using GPU — partial offload symptom, OLLAMA_KV_CACHE_TYPE example.
Ollama issue #8144 — silent CPU fallback when a model does not fit.
llama.cpp issue #22075 — exact CUDA OOM log text.
Hugging Face discussion 140711 — 128k context memory reports from users.

Browse the Hardware pillar for GPU picks per VRAM tier, run the VRAM Calculator for any model, and see how to run a local LLM on Windows with Ollama for the hands-on side.

How to Estimate VRAM for Any LLM: The Math