Quantization
Quantization stores model weights with fewer bits — FP16 (2 bytes per weight), Q8 (1 byte), Q5 (~0.7), Q4 (~0.6), Q3 (~0.5). The goal: fit a bigger model in less VRAM, and move data faster.
The trade-off: quality drops slightly with each step down. In practice:
- Q8 — near-lossless, the “quality” choice when VRAM allows.
- Q4 (e.g. Q4_K_M) — the default; a good quality/size balance for most models.
- Q5/Q6 — the middle ground.
- Q3 and below — noticeable degradation; only for very tight hardware.
Where it is used: GGUF files (llama.cpp, Ollama, LM Studio), AWQ/GPTQ for CUDA runners (vLLM, TGI), and QLoRA for fine-tuning.