llama.cpp

llama.cpp is a lightweight C/C++ inference runtime that became the de facto standard for running quantized LLMs on consumer hardware. Ollama and LM Studio both build on top of it, so learning llama.cpp pays off even if you keep using their UIs.

It shines when you want to squeeze a model into limited VRAM: the GGUF format and its Q4/Q5/Q6/Q8 quantizations are the most efficient in the ecosystem, and the llama-cli, llama-server and llama-quantize tools cover chatting, serving and quantizing.

# Chat with a model
./llama-cli -m qwen2.5-7b-instruct-q4_k_m.gguf

# Start an OpenAI-compatible server
./llama-server -m qwen2.5-7b-instruct-q4_k_m.gguf -c 4096