By the end of this guide, Llama 3.1 8B will be running locally with full GPU acceleration on a 16 GB card. The default llama3.1:8b model in Ollama is a 4.9 GB Q4_K_M quantization — a third of your VRAM. The rest goes to the context window, and that is where 16 GB setups lose speed: Ollama starts a card this size at only 4,096 tokens of context, and a 64k context costs about 8 GB of KV cache.

The model fits every 16 GB card — this local LLM guide covers the most common 2026 consumer setup. Speed is the only variable.

GPU	Bandwidth	Llama 3.1 8B Q4_K_M (estimated)
RTX 4060 Ti 16GB	288 GB/s	~35–46 tok/s
RTX 5060 Ti 16GB	448 GB/s (GDDR7)	~68 tok/s
RTX 5070 Ti 16GB	896 GB/s	40–55 tok/s (reported class)

Those figures come from third-party calculators, and they disagree: one benchmark puts the 4060 Ti at 34 tok/s, another at 45.9. Treat them as a range, not a measurement. The 4060 Ti 16GB is the cheapest 16 GB entry at ~~$575–900 new (as of October 2026); the 5070 Ti (~~$750 as of October 2026) doubles the bandwidth. The full card list is on our hardware page.

Requirements

A 16 GB GPU — NVIDIA, AMD, or Apple unified memory. A 10–12 GB card also runs it at 4k context.
NVIDIA: driver 551.61 or newer (570+ for Maxwell/Pascal cards like the GTX 1060).
AMD: ROCm v7 stack on Linux, or a Vulkan-capable driver on Windows.
Windows 10 22H2 or newer, macOS, or Linux.
~10 GB of free disk and 16 GB of system RAM.

1. Check your GPU and driver

Start with the driver.

nvidia-smi

If the command is not recognized, install the driver from NVIDIA first — Ollama will not see the GPU without it. The output shows your driver version; Ollama needs 551.61 or newer on Windows, and Maxwell/Pascal cards (GTX 1060 and older) need 570 or newer.

2. Install Ollama

Windows:

winget install Ollama.Ollama

or download the installer from ollama.com. No administrator rights needed; it installs into your home directory.

macOS:

brew install ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Verify with ollama --version. The server listens on 127.0.0.1:11434; models are stored in %HOMEPATH%\.ollama on Windows, and logs live in %LOCALAPPDATA%\Ollama.

3. Pull Llama 3.1 8B

ollama pull llama3.1:8b

The default tag is 4.9 GB. Other sizes, all on the model page:

Tag	Quant	Size
`llama3.1:8b` (default)	Q4_K_M	4.9 GB
`llama3.1:8b-q8_0`	Q8_0	8.5 GB
`llama3.1:8b-q6_K`	Q6_K	6.6 GB
`llama3.1:8b-q4_0`	Q4_0	4.7 GB
`llama3.1:8b-q3_K_M`	Q3_K_M	4.0 GB

For a 16 GB card, Q4_K_M is the sensible default: it leaves room for context. Q8_0 still fits with a modest context and is the quality pick — see the FAQ for the trade.

4. Verify full GPU offload

ollama run llama3.1:8b "Explain KV caching in two sentences"
ollama ps

You want 100% GPU in the PROCESSOR column and 4096 in CONTEXT. The server log should contain:

llm_load_tensors: offloaded 33/33 layers to GPU

Thirty-three, not thirty-two — llama.cpp counts the embeddings as a layer. If ollama ps shows CPU, or the log shows layers not offloaded, the model runs on the CPU. Check the driver floor first, then read Troubleshooting.

5. Set a context length that fits

Ollama gives a 16 GB card a 4k context by default — the setting for any GPU under 24 GiB. You can raise it. Llama 3.1 8B costs 128 KB of KV cache per token in FP16 (the derivation is in our VRAM estimation guide):

Context	KV cache	Q4_K_M total (weights ~4.9 GB)
4,096	~0.5 GB	~6.5 GB
16,384	~2 GB	~8 GB
32,768	~4 GB	~10 GB
65,536	~8 GB	~14 GB
131,072 (native)	~16 GB	~21 GB — does not fit

So 64k is the practical ceiling on 16 GB, not the 131k the model supports natively. Set it server-wide:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

or per request through the API with "options": { "num_ctx": 16384 } in the request body. If 64k with Q8_0 weights feels tight, set OLLAMA_KV_CACHE_TYPE=q8_0 — with flash attention (automatic when supported, or force it with OLLAMA_FLASH_ATTENTION=1) the KV cache halves: 64k drops to ~4 GB, freeing room for the 8.5 GB Q8_0 weights.

What to expect

Setup on RTX 4060 Ti 16GB	VRAM	Speed (est.)
Q4_K_M, 4k context	~6.5 GB	35–46 tok/s
Q4_K_M, 64k context	~14 GB	—
Q8_0, 4k context	~10 GB	~28 tok/s

Q8_0 runs near 28 tok/s. Check your own card in the VRAM calculator.

Troubleshooting

cudaMalloc failed: out of memory (also CUDA error: out of memory and the wrapper llama-server reported out-of-memory during startup) — an allocation failed during load. The usual cause is another model still loaded: Ollama keeps models warm for 5 minutes by default (OLLAMA_KEEP_ALIVE). Run ollama ps and ollama stop <model> for anything you do not need.

model requires more system memory (X GiB) than is available (Y GiB) — the scheduler refused the model before it tried to load. Shrink num_ctx or OLLAMA_NUM_PARALLEL — memory scales with their product — or stop other loaded models.

unable to load full model on GPU — a partial offload. Ollama offloads as many layers as it thinks will fit and runs the rest on CPU. Pick a smaller quant, or pin the split with num_gpu in the model options.

Silent CPU fallback — the nastiest one. A GitHub issue reports the scheduler can fall back to CPU with no user-visible warning, logging offloaded 0/66 layers to GPU while requests still “work.” If a response suddenly feels ten times slower, ollama ps is the only reliable answer.

signal: killed — that is the operating system’s OOM killer, not the GPU. You ran out of system RAM, not VRAM.

AMD: ROCm error: out of memory / hipMalloc failed — same VRAM math, ROCm allocator.

The 70B trap — point at llama3.1:70b (43 GB) on a 16 GB card and the client gets Error: error reading llm response: read tcp 127.0.0.1:49245->127.0.0.1:49210: wsarecv: An existing connection was forcibly closed by the remote host. That exact failure was reported on a 16 GB P100. Use the 8B.

When things do not fit, cheapest fix first: stop other loaded models, shrink context, set OLLAMA_KV_CACHE_TYPE=q8_0, constrain num_gpu, and only then lower the quant.

FAQ

Why is Ollama running on CPU even though I have a GPU? Check the driver floor first — 551.61 or newer on Windows, 570+ for Maxwell/Pascal. Then run ollama ps; if PROCESSOR says CPU, you likely hit the silent fallback above.

Is 16 GB enough for Llama 3.1 8B with a long context? Yes, up to about 64k at Q4_K_M. The KV cache costs 128 KB per token, so 64k is roughly 8 GB on top of the 4.9 GB of weights. The native 131k context does not fit.

Q4_K_M or Q8_0, which one? Use Q8_0 (8.5 GB) while your context stays under ~32k — the quality difference is real. Use Q4_K_M (4.9 GB) when you need 64k context. Both fit; it is a context-versus-quality trade.

Can I run Llama 3.1 70B on a 16 GB card? No. The 70B Q4_K_M is 43 GB. Partial offload will run, but at CPU speed it is unusable for chat.

Do I need to set OLLAMA_FLASH_ATTENTION? No — Ollama enables it automatically when the backend supports it. Set OLLAMA_FLASH_ATTENTION=1 only to force it, and only if you also use OLLAMA_KV_CACHE_TYPE=q8_0, which requires it.

Prefer a GUI over the terminal? The LM Studio vs Ollama comparison covers the two main runners and when each wins.

Sources

Ollama docs — Context length — default context by VRAM tier, OLLAMA_CONTEXT_LENGTH, the ollama ps columns.
Ollama docs — Windows — install without admin, the 551.61 driver floor, model directory, log locations.
Ollama docs — FAQ — flash attention behavior, OLLAMA_KV_CACHE_TYPE, per-request num_ctx.
Ollama — Llama 3.1 tags — exact tag sizes: 4.9 GB default, 8.5 GB q8_0, 43 GB 70B, 243 GB 405B.
ollama/ollama #6382 — 70B on a 16 GB P100: the client-side wsarecv error.
ollama/ollama #14258 — silent GPU-to-CPU fallback with no user-visible warning.
willitrunai — Llama 3.1 8B on RTX 4060 Ti 16GB — 9.3 GB VRAM breakdown, 45.9 tok/s estimate.
modelfit — RTX 4060 Ti — 34 tok/s estimate, 5060 Ti 16GB ~68 tok/s.
local-llm.net — RTX 5070 Ti benchmarks — 896 GB/s, 40–55 tok/s class, ~$750 (2026), 4060 Ti 16GB ~$450 (launch price; street ~$575–900, Oct 2026).
localaimaster — Ollama out-of-memory errors — the OOM error strings and the six fix levers.
Colab — bartowski Llama 3.1 8B Q8_0 — full-offload log: offloaded 33/33 layers to GPU.

Run Llama 3.1 8B on a 16 GB GPU