By the end of this guide, any GGUF model will be served from your machine as an OpenAI-compatible API — on CPU, CUDA, Metal, or Vulkan — and you will know how to measure what your hardware actually does. llama.cpp is the C/C++ inference engine behind most local LLM tools, and running it directly hands you the full flag set instead of a wrapper.

The numbers up front: in one community benchmark harness, a single RTX 3090 (24 GB) pushed Llama 3.1 8B Q4_K_M at 96.8 tok/s — and a Mac mini M4 with 16 GB managed 20.95 ± 0.02 tok/s on the same model through Metal. Both figures are from named, reproducible runs, and both are what to expect from llama.cpp when the model fits.

Requirements

Any OS with a supported backend: CUDA, HIP, Metal, Vulkan, SYCL, CANN, MUSA, or OpenCL — or no GPU at all, it runs on CPU.
A GGUF model file. The standard sources on Hugging Face are ggml-org (official example repos), bartowski, and unsloth (collections including “Unsloth Dynamic 2.0 GGUF + Quants”).
Room for the model: Llama 3.1 8B Q4_K_M is about 4.9 GB (the bartowski GGUF file measured 4,920,739,232 bytes, about 4.58 GiB, in one run).
MIT license — no usage restrictions.

1. Install the unified llama CLI

Latest release as of October 2026: v0.7.0 (2026-10-15).

Linux / macOS:

curl -LsSf https://llama.app/install.sh | sh

Windows:

irm https://llama.app/install.ps1 | iex

Docker:

docker run -it -v ./models:/models ghcr.io/ggml-org/llama.cpp:server

The image serves on port 9931 with models mounted at /models. Since the v0.7.0-era CLI, everything is one binary with three subcommands: llama cli (interactive chat), llama serve (OpenAI-compatible server), and llama bench (benchmarking).

2. Serve a GGUF model on port 9931

Point it at a local file:

llama serve -m model.gguf

Or pull straight from Hugging Face — no manual download:

llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0

The default port as of v0.7.0 is 9931. Older docs and screenshots still show 8080 — always pass --port explicitly so your tooling matches:

llama serve -m model.gguf --port 8080 --no-webui

--host sets the bind address, and --no-webui drops the bundled web interface if you only want the API. One quirk: llama serve started without a model argument runs as a router rather than a server — if nothing is listening the way you expect, check that a model was actually passed.

3. Control memory with -ngl and -c

The two flags that decide whether the model runs fast, slow, or not at all:

-c / --ctx-size — context window in tokens.
-ngl / --gpu-layers — how many layers offload to the GPU. 0 = CPU only.
-np / --parallel — parallel slots.

llama serve -m model.gguf -c 8192 -ngl 0

Start at -ngl 0 to confirm the model loads at all, then raise the layer count until you hit the memory limit. For MoE models there is a dedicated dial, --moe-cache-mib, which the release notes claim can give roughly 2.2× decode speedup when the expert cache fits — treat that as the release-note claim, not a universal measurement. Speculative decoding is a separate switch: -md <draft> --spec-type draft-simple, or --hf-repo-draft <repo> --spec-type draft-mtp to pull the draft model from Hugging Face.

4. Benchmark with llama bench

llama bench -m model.gguf -r 10 -o csv
llama bench --list-devices

-r / --repetitions sets the sweep length (default 5), -o picks the output format — csv, json, jsonl, md, or sql — and --list-devices prints what your backend actually sees, which is the first thing to run when a GPU is missing. Two fit helpers are the underrated ones: -fit / --fit-target <MiB> sizes the run to a memory target, and -fitc / --fit-ctx fits the context to what remains.

5. Call the OpenAI-compatible API

The server exposes the standard endpoint:

http://localhost:9931/v1/chat/completions

curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

Anything that speaks OpenAI — chat clients, SDKs, RAG stacks — can point at it.

What to expect

Llama 3.1, llama.cpp, named hardware. The 3090 figures come from one community benchmark harness (vLLM 0.21.0 era, PyTorch 2.6 / CUDA 12.4, driver 550.127; batch 1, 1k prompt / 256 generated) — a fair reference, not a 2026 measurement:

Hardware	Model	Quant	Speed
1× RTX 3090 24 GB	Llama 3.1 8B	Q4_K_M	96.8 tok/s
1× RTX 3090 24 GB	Llama 3.1 8B	Q8_0	61.7 tok/s
4× RTX 3090 (TP=4)	Llama 3.1 70B	Q4_K_M	14.1 tok/s
RTX 4060 Ti 16 GB	Llama 3.1 8B	Q4_K_M	~35–46 tok/s (estimated)
Mac mini M4, 16 GB	Llama 3.1 8B	Q4_K_M	20.95 ± 0.02 tok/s

Two caveats on the table. The 4060 Ti row is a range of third-party estimates — willitrunai says 45.9, modelfit about 34; different estimation models, so read it as 35–46, not a measured number. The M4 row also measured 229.23 tok/s prompt processing and 6.1 GB resident memory at 8K context.

Troubleshooting

out of memory — the classic is -ngl 80 on a BF16 Phi-4-mini: too many layers offloaded for the VRAM available. Drop -ngl or switch to a quantized file. A full-precision file on a mid-size card is the usual cause.

ggml_backend_metal_init: error 36: Resource temporarily unavailable — Metal backend initialization failure, reported on macOS. It is an init race, not a model problem: retry the start, close other Metal-heavy apps, and confirm your OS meets the backend’s floor.

“The server is on 8080” / “the server is on 9931” — both have been true at different times. As of v0.7.0 the default is 9931; older tutorials show 8080. Pass --port explicitly and your tooling can never disagree.

llama bench --list-devices shows no GPU — the backend you want is not compiled or not visible: check the driver (CUDA), the ROCm/HIP setup, or on Mac that you are on a Metal-capable build.

FAQ

Which GGUF file should I actually download? The safe defaults: ggml-org repos for the official examples, bartowski for breadth, unsloth for dynamic quants. For a first 8B, Q4_K_M is the quant that appears in the most benchmarks — it is the one in the table above.

Can I run llama.cpp without a GPU? Yes — -ngl 0 is CPU-only mode, and the backend list includes plain CPU. Expect order-of-magnitude slowdowns versus the GPU rows above.

What is the difference between llama serve and llama cli? llama cli is an interactive terminal chat; llama serve is the OpenAI-compatible HTTP server. Same engine, two front doors.

Does it work in Docker? Yes: ghcr.io/ggml-org/llama.cpp:server serves on port 9931 with ./models mounted to /models.

That is the raw engine. If you want the same model with one fewer moving part, the Ollama on Windows guide walks the wrapper route, and Llama 3.1 8B on a 16 GB GPU sizes the hardware side. The llama.cpp review in the Software pillar covers features this guide skips, and the VRAM calculator tells you whether a given GGUF fits before you pull it.

Sources

ggml-org/llama.cpp README — backends, MIT license, install one-liners, Docker image, the unified llama CLI, llama serve/-hf/flags, the 9931 endpoint, llama bench flags.
ggml-org/llama.cpp v0.7.0 release — release date 2026-10-15; the --moe-cache-mib ~2.2× claim.
cap10bad/llm-inference-benchmarks — 1× 3090: 96.8 / 61.7 tok/s; 4× 3090 TP=4 70B: 14.1 tok/s (harness: vLLM 0.21.0 era, 2025–2026).
willitrunai — Llama 3.1 8B on RTX 4060 Ti 16 GB and modelfit — RTX 4060 Ti — the 35–46 tok/s estimate range (via docs/research/run-llama-31-8b-on-16gb-gpu.md).
picklog — Llama 3.1 8B on Mac mini M4 — 20.95 ± 0.02 tok/s Metal, 229.23 pp512, 6.1 GB resident, 86% of the 120 GB/s ceiling (September 2026).
ggml-org/llama.cpp discussion #4167 — 10-core M4, 7B Q4_0: 221.29 tok/s prompt / 24.11 tok/s decode.
ggml-org/llama.cpp issue #14740 — the “out of memory” after -ngl 80 and the ggml_backend_metal_init: error 36 string.
Hugging Face — bartowski / unsloth — GGUF quant sources.

How to Run a Local LLM with llama.cpp