How to Choose a GPU for Local AI: VRAM First
Most people buy the biggest GPU their budget allows, then complain the model is slow or does not fit. The order should be reversed: pick the model, compute its memory, and only then choose the cheapest card that fits with headroom.
The number that decides everything: the 70B class at Q4 is 43 GB in Ollama’s llama3.3:70b (128K context). It does not fit on any single consumer GPU — not the 24 GB RTX 4090, not the 32 GB RTX 5090.
Decide what you will run
Match the workload to a model class:
| Workload | Model class | Reference model (Ollama, as of October 2026) | Download size |
|---|---|---|---|
| Chat, summarization, light code | 7–8B | llama3.1:8b | 4.9 GB |
| Coding with long context | 13–14B | qwen3:14b | 9.3 GB |
| Best local quality per dollar | 30–32B | qwen3:32b | 20 GB |
| Near-frontier quality, 128K context | 70B | llama3.3:70b | 43 GB |
Recommendation: buy the model first, not the GPU. A 32B at Q4 answers better than any 8B, and used 24 GB cards now cost less than new 16 GB ones.
Convert model sizes to VRAM
Weights in memory scale with quantization:
| Quantization | GB per 1B params |
|---|---|
| FP16 | 2.0 |
| Q8 | 1.0 |
| Q5 | 0.7 |
| Q4 | 0.6 |
| Q3 | 0.5 |
Ollama’s measured Q4_K_M sizes land at 0.61–0.66 GB/B (8B → 4.9 GB, 70B → 43 GB), so the 0.6 row is a fair planning number.
Add roughly 20–30% on top for the KV cache and runtime; the per-model formula is in How to Estimate VRAM for Any LLM.
Two worked examples:
- 32B at Q4: 20 GB of weights, about 4 GB left for cache on a 24 GB card — comfortable at moderate context, tight at 32K.
- 70B at Q4: 43 GB — no single card fits it. Two 24 GB cards (48 GB, NVLink) or a 96 GB Mac do.
Compare GPU options in your budget
Market context first: as of October 2026, new GPU prices are inflated by AI demand — the RTX 5090 launched at $1,999 and now trades at $3,000–5,000 street. VRAM per dollar points at used cards.
| Card | VRAM | Street price (October 2026) | Runs comfortably |
|---|---|---|---|
| RTX 3060 12 GB | 12 GB | ~$300 used, ~$466 new | 7–8B Q4/Q5 |
| Tesla P40 24 GB | 24 GB | $150–330 used | 7–14B Q4, 32B Q4 (slow) |
| RTX 4060 Ti 16 GB | 16 GB | ~$575–900 new | 8B Q8, 14B Q4 |
| RTX 5060 Ti 16 GB | 16 GB | ~$780 (MSRP $429) | 8B Q8, 14B Q4, faster |
| RTX 3090 24 GB | 24 GB | ~$600–1,400 used | 14B Q8, 32B Q4 |
| RTX 4090 24 GB | 24 GB | ~$2,000+ used | 32B Q4, faster |
| RTX 5090 32 GB | 32 GB | $1,999 MSRP, $3,000–5,000 street | 32B Q4 fast; 70B Q4 does not fit |
| Mac Studio M5 Max / M5 Ultra | 36–512 GB unified | from $2,499 / from $5,499 | 70B Q4 and up |
Per tier:
- 12 GB (RTX 3060): the default first card.
llama3.1:8b(4.9 GB) runs with headroom; 14B Q4 (9.3 GB) fits only with a short context. - 16 GB (4060 Ti / 5060 Ti): the 4060 Ti trades above its $449 launch price; the 5060 Ti buys 448 GB/s of GDDR7 for a 180 W draw. Both top out at 14B-class comfort.
- 24 GB (3090 / 4090 / P40): the 2026 sweet spot. Used 3090 prices are volatile: ~$600 in early 2026, $700–1,000 in June, ~$1,400 later as the 5090 spike pushed buyers into used 24 GB. Two 3090s with NVLink pool 48 GB and run 70B Q4. The P40 is the budget floor: $150–330, 346 GB/s, no Tensor Cores.
- 32 GB (RTX 5090): the fastest single card — ~60–70 tok/s on dense 32B Q4, ~150–200 on 8B in one hardware test (2026). It still does not fit 70B Q4.
- Macs: the M5 Max starts at $2,499 (36 GB), the M5 Ultra at $5,499 (96 GB; Apple, October 2026), with a 512 GB M5 Ultra expected in late October. 96 GB is where 70B Q4 (43 GB) fits with context to spare. Reference bandwidth: 546 GB/s (M4 Max), 819 GB/s (M3 Ultra).
Budget for the rest of the system
The GPU is half the invoice. Minimum power supplies:
| Card | Minimum PSU |
|---|---|
| RTX 5090 | 1000 W |
| RTX 4090 | 850 W |
| RTX 3090 | 750 W |
| RTX 5060 Ti | 600 W |
- RAM: 32 GB minimum; 64 GB if you expect CPU offload of layers or KV cache (offload section).
- P40 extras: a cooling shroud ($10–30) and an EPS 8-pin adapter ($8–15); no video out, so plan for headless.
- Dual 3090: add an NVLink bridge and a case that fits two 3-slot cards.
The AI Stack Builder does this from your VRAM, budget, and OS.
Buy and verify
Verify before you trust the purchase:
ollama pull qwen3:14b(9.3 GB) or your target model.ollama ps— the right column should read100% GPU;100% CPUmeans the model is on the CPU.- If the log shows
offloaded 0/66 layers to GPU, the model did not fit and Ollama fell back silently.
Then run one model at the edge of your budget — 14B Q4 on 16 GB, 32B Q4 on 24 GB — and expect a stable token rate with no out-of-memory errors. Install details: Run a Local LLM with Ollama on Windows.
What to expect: speed
Single-stream tok/s, Q4 quantizations:
| Card | 7–8B | 13–14B | 32B |
|---|---|---|---|
| RTX 3090 24 GB | 45–75 | 25–40 | ~12–20 |
| RTX 4090 24 GB | 55–95 | — | — |
| Tesla P40 24 GB | 15–25 | 8–12 | 4–7 |
| RTX 5090 32 GB | ~150–200 | ~124 | ~60–70 |
3090/4090/P40 figures: one hardware guide (June 2026). 5090 figures cluster across one hardware test, one estimate, and one community thread (2026). Reference points from one community benchmark (August 2026): 2.8 tok/s on a 4-vCPU/8 GB CPU, 63 tok/s on an RTX 4000 Ada 20 GB. Check your own card in the VRAM Calculator.
Troubleshooting
No compatible GPUs were discovered— Ollama found no usable GPU. Update the driver to 551.61+ and restart Ollama (issue #8674).CUDA error: out of memorywithggml-cuda.cu: CUDA error, current device: 0, in function alloc— model plus cache exceeded the card. Lower the context, drop a quant step, or add a card (llama.cpp #22075).offloaded 0/66 layers to GPU— the run is CPU-only; expect around 2.8 tok/s on a small machine.- Silent CPU fallback with no error — Ollama can fall back without complaining when a model does not fit (issue #8144). Always check
ollama ps. - P40 refuses FP16/INT8 fast paths — expected: no Tensor Cores. Use Q4_K_M and expect the 15–25 tok/s class on 7–8B.
FAQ
Is VRAM more important than GPU speed? Yes, until the model fits. A 24 GB 3090 beats a faster 16 GB card that cannot run the model at all.
Can I use two cards? Two 24 GB cards pool 48 GB and run 70B Q4 (43 GB). The 3090 is the cheap path — NVLink bridges are available.
Is the P40 really worth $200? As a first 24 GB card, yes: $150–330 all-in, and it runs 7–14B Q4. As a 32B daily driver, no — 4–7 tok/s is not comfortable.
RTX 5090 or Mac for 70B? The 5090 does not fit 43 GB. A 96 GB M5 Ultra Studio from $5,499 does, with context to spare; a dual-3090 rig costs less and is faster.
Browse the Hardware pillar for per-card pages, and see How to Run Llama 3.1 8B on a 16 GB GPU for the 16 GB scenario in detail.
Sources
- Ollama llama3.3 — 70B = 43 GB, 128K context.
- Ollama qwen3 — 8B/14B/32B sizes.
- Ollama llama3.1 — 8B = 4.9 GB.
- NVIDIA RTX 5090 — 32 GB, 1792 GB/s, 575 W, 1000 W, $1,999.
- PNY RTX 5060 Ti — 16 GB GDDR7, 180 W, 600 W.
- Apple Mac Studio — M5 Max $2,499 / 36 GB, M5 Ultra $5,499 / 96 GB (October 2026).
- TPU: 5090 specs — 512-bit, 21760 cores.
- TPU: 4060 Ti specs — 288 GB/s, 165 W.
- TPU: 3060 specs — 360 GB/s, 170 W.
- localaimaster: 3090 — 936 GB/s, 750 W, $700–1,000, tok/s.
- localaimaster: P40 — 346 GB/s, $150–330, tok/s.
- Spheron GPU guide — INT4 sizes, 20–30% overhead.
- localaimaster: M4 — M4 Max 546 GB/s, M3 Ultra 819 GB/s.
- presenc.ai: 5090 benchmarks — 8B/14B/32B tok/s.
- modelfit.io: 5090 — 69 tok/s 32B estimate.
- willitrunai: 5090 — 66.4 tok/s 32B Q4_K_M.
- shattered.io: price surge — ~$5,000 street (October 2026).
- TPU: 5090 price leaks — 2026 outlook.
- mindstudio: prices — used 3090 $1,400, 3060 $300.
- Newegg: 4060 Ti — $575–900.
- Amazon: 5060 Ti — $780.
- BestBuy: 3060 — $466.
- reddit: 5090 thread — community numbers.