Skip to content

Best 7B–8B LLMs for 8 GB VRAM: Qwen 3 8B, Gemma 3, Mistral

beginner local-llm By AI Hub Team · updated

8 GB is the most common VRAM size in consumer cards — RTX 4060, 4060 Ti, used RTX 3060s — and every 7B–8B model fits at Q4_K_M. The question is which one you can live with daily.

The short answer: Qwen 3 8B is the default, Mistral 7B the speed pick, Gemma 3 4B the headroom pick, and Gemma 3 12B the trap — CPU offload only, at a quarter of the speed.

All numbers below are from an RTX 4060 8 GB, as of October 2026; download sizes are Ollama’s.

At a glance — what to expect

ModelParamsQ4_K_M downloadTotal VRAM (4060 8 GB)SpeedSafe contextVision
Qwen 3 8B8.2B5.2 GB~8.6 GB (tight)~39–42 tok/s~12KNo
Llama 3.1 8B8.0B4.9 GB~8.2 GB (tight)~40 tok/s~14KNo
Mistral 7B v0.37.2B4.4 GB~7.9 GB~50 tok/s~8KNo
Gemma 3 12B12B8.2 GB~10.1 GB (offload)~15.6 tok/s~4KYes
Gemma 3 4B3.9B3.4 GB~4.7 GB~56 tok/s128KYes

Total VRAM = weights + KV cache + runtime overhead at the safe context. Speed is generation throughput.

Why 8 GB is a different game

At 8 GB you cannot afford q8_0 quantization or long contexts.

The KV cache is the part people forget: ~128 KB per token for a standard 8B, so an 8K context eats ~1 GB. The full formula is in How to Estimate VRAM for Any LLM.

Two consequences follow:

  • Context is your first knob. Halve it and you halve the cache.
  • Offload is the escape hatch — and it is slow. Anything that spills to RAM runs at CPU speed.

Qwen 3 8B — the default

Qwen 3 8B is the most capable model here: 8.2B parameters, 36 layers, Apache 2.0, 100+ languages, and a thinking mode toggled with /think and /no_think.

The Ollama tag is a 5.2 GB Q4_K_M file with a 40K context default — exactly what overflows. On the 4060 8 GB the total is ~8.6 GB: 108% of the card, ~0.6 GB over, ~39 tok/s, 12K safe context. The q8_0 variant is an 8.9 GB download and does not fit.

Cap the context and it becomes a daily driver — the best quality that fits on 8 GB.

Llama 3.1 8B — the safe baseline

Llama 3.1 8B is the established baseline: 8.03B parameters, 32 layers, 131K context.

The Q4_K_M download is 4.9 GB — the smallest of the three 8B models. On the 4060 8 GB: ~8.2 GB total, a tight fit at ~40 tok/s, 14K safe context. Fill a 16K context and speed drops to ~34 tok/s. The cap is real.

For the same model on a 16 GB card, see Run Llama 3.1 8B on a 16 GB GPU.

Mistral 7B — the speed pick

Mistral 7B v0.3 is the fastest of the group. The mistral:latest tag is v0.3: 4.4 GB Q4_K_M, 32K context, function calling, text only.

On the 4060 8 GB: ~7.9 GB total, ~50 tok/s, 350 ms first token, 8K safe context. In one estimate, an older RTX 2060 Super 8 GB ran it at ~61 tok/s. RAG and agentic prompts drop it to ~29 tok/s — prefill is the bottleneck there.

It wins when raw speed matters more than features.

Gemma 3 12B — the trap

Gemma 3 12B is the model that breaks the 8 GB rule. This one does not fit.

The Ollama tag is an 8.2 GB Q4_K_M file (128K context, text + image). On the 4060 8 GB: ~10.1 GB total, 126% of VRAM, ~2 GB over — partial CPU offload kicks in: ~15.6 tok/s (17.5 chat, 12.7 RAG), 4K safe context.

Two traps hide in the details:

  • The QAT file is larger, not smaller. gemma3:12b-it-qat downloads at 8.9 GB — 8.1 GB Q4_0 plus an 854 MB vision projector — versus 8.2 GB for plain Q4_K_M.
  • On weaker 8 GB cards it gets worse. On an RTX 3060 Ti 8 GB only Q1_0 fits fully on the GPU — a severe quality hit.

The model card reports MMLU 74.5, GSM8K 71.0, DocVQA 82.3, plus vision; on a 12 GB card the QAT version fits in ~9 GB. On 8 GB, skip it.

Gemma 3 4B — the headroom pick

Gemma 3 4B is the opposite end: a 3.4 GB bundle — 2.5 GB of 3.88B weights plus an 851 MB vision projector — leaving half the card free.

On the 4060 8 GB: ~4.7 GB total, 59% of the card, ~56 tok/s, 350 ms first token, a full 128K safe context. The only model here where long documents need no cap.

Choose it when vision, long context, or running two models at once matters. Weakest on raw reasoning, but the headroom is real.

KV cache per token

ModelLayersKV headsHead dimKV per token (FP16)
Llama 3.1 8B328128~128 KB
Mistral 7B v0.3328128~128 KB
Qwen 3 8B368128~144 KB
Gemma 3 4B344256~136 KB
Gemma 3 12B488256~384 KB

Gemma 3 uses a 1024-token sliding window on half its layers, so 384 KB/tok is an upper bound. OLLAMA_KV_CACHE_TYPE=q8_0 halves the cache for every model here (Ollama docs).

At 8K context, Gemma 3 12B needs ~3 GB of KV cache alone; Llama 3.1 8B and Mistral 7B need ~1 GB each.

Steps

1. Pick the baseline quant: Q4_K_M

Start with Q4_K_M, the quality-to-size balance Ollama ships by default. q6_K exists for Llama 3.1 8B at 6.6 GB and still fits, but leaves ~1.4 GB for the cache. q8_0 files run 8.5–8.9 GB and do not fit at all.

2. Pull the model and verify the download size

ollama pull qwen3:8b
ollama list

Check the GB column in ollama list. If you expected 5.2 GB and see 8.9 GB, you pulled a q8_0 tag. The table above lists Ollama’s reported sizes, as of October 2026.

3. Cap the context to stay inside 8 GB

Set the cap before the first request — the overflow happens at load time.

# Linux / macOS
OLLAMA_CONTEXT_LENGTH=8192 ollama serve

On Windows, set OLLAMA_CONTEXT_LENGTH as a system environment variable and restart Ollama: 4096 for Gemma 3 12B, 8192–12288 for the 8B models.

4. Measure speed on your own tasks

Run three prompts that match your real workload — a short chat, a document summary, a code fix — and note tokens/s plus first-token time. A chat can do 50 tok/s while a RAG prompt does 29, and both numbers are true.

5. Choose by workload, not by parameter count

Default: Qwen 3 8B. Speed: Mistral 7B. Vision or long context: Gemma 3 4B. Gemma 3 12B only with 16 GB of system RAM and patience. For any other card, the Model Finder and VRAM Calculator do the matching.

Troubleshooting

  • failed to allocate / CUDA out of memory on the first token — the context cap is too high. Lower OLLAMA_CONTEXT_LENGTH (start at 4096) or set OLLAMA_KV_CACHE_TYPE=q8_0.
  • The model loads but is far slower than the table — check ollama ps. If the GPU column is under 100%, part of the model is on the CPU — Gemma 3 12B’s normal state on 8 GB (~10% of layers offloaded).
  • ollama list shows 8.9 GB for “the 12B” — you have the QAT tag (gemma3:12b-it-qat). Use gemma3:12b.
  • Gemma 3 4B shows 3.4 GB, other sources ~2.2 GB — that figure is weights only; the Ollama bundle adds an 851 MB vision projector.
  • First token takes ~1 second on RAG prompts — prefill cost: the whole prompt is computed before the first token. Expected.

FAQ

Can I run Gemma 3 12B on 8 GB VRAM? Yes, with CPU offload: ~10.1 GB total, ~16 tok/s, ~4K safe context. Community runs it in the same regime on 8 GB VRAM plus 16–32 GB of RAM. A 12 GB card is the better fix — the QAT version fits in ~9 GB.

Is Q4_K_M enough, or should I use Q6_K? Q4_K_M is the baseline and the one that fits comfortably. The q6_K variants still fit but leave little room for the cache. On 8 GB, the quality gain is rarely worth the lost context.

Why is my 8B model slower than 40 tok/s? Three usual causes: partial CPU offload (check ollama ps), a context longer than the safe value (16K drops Llama 3.1 8B to ~34 tok/s), or RAG prompts (Mistral 7B drops from ~50 to ~29 tok/s).

Do I need much system RAM too? For the 7B–8B models: 16 GB is plenty. For Gemma 3 12B with offload, 32 GB is comfortable: in one community sweep, 8 GB VRAM + 32 GB RAM ran a 30B MoE model at ~15 tok/s with two thirds of the weights in RAM.

Recommendation

On an 8 GB card:

  • Default: Qwen 3 8B — the best quality that fits, thinking mode, Apache 2.0. Cap the context at 12K.
  • Speed: Mistral 7B — ~50 tok/s, but text-only with no thinking mode.
  • Headroom: Gemma 3 4B — 4.7 GB total, 128K context, vision; the only one that leaves room for a second model.
  • Skip: Gemma 3 12B — offload, ~16 tok/s, 4K context. Wait for a 12–16 GB card.

For any other card, start with the VRAM Calculator; the broader local-LLM landscape lives on the Local LLMs pillar page.

Sources