By the end of this guide, a Hugging Face model will be serving an OpenAI-compatible API from your own GPU — one install command, one serve command, and the throughput engine behind a lot of production inference. vLLM is the runner to reach for when the goal is a server, not a chat window: PagedAttention, continuous batching, prefix caching, LoRA, speculative decoding, structured outputs, tensor parallelism, and 200+ supported architectures, with OpenAI-compatible, Anthropic, and gRPC APIs.

The numbers up front: in one community benchmark harness, a single RTX 3090 (24 GB) ran Llama 3.1 8B GPTQ-4 at 112.4 tok/s under vLLM — ahead of llama.cpp’s 96.8 tok/s on the same model and quant class at batch 1. That gap, and the batching that widens it under load, is why vLLM exists.

Requirements

An NVIDIA GPU with a driver covering CUDA 12.9 or 13.0 — the prebuilt GPU wheels target those two. If your driver is stuck on 12.8, the official 12.8 wheels now 404 and you build from source.
uv (or plain pip) and a Python environment.
A model: any Hugging Face model ID, or a local path.
Optional: Docker, which is a documented install path, and Apple Silicon support via the new vLLM-Metal backend and CPU wheels.

1. Install vLLM with uv

Latest release as of October 2026: v0.32.0 (published 2026-10-18).

uv pip install vllm
vllm download-kernels

download-kernels fetches the precompiled kernels after the install — skip it and the first serve will stall. Two notes for non-standard hardware:

Stuck on CUDA 12.8? The official 12.8 wheels 404 as of October 2026 — a staff reply on the vLLM forum confirms the current prebuilt set is 12.9/13.0, and the fix for 12.8 is a source build.
No NVIDIA GPU? CPU wheels exist — vllm-0.32.0+cpu-cp38-abi3-macosx_11_0_arm64.whl (about 30 MB) for Apple Silicon and manylinux aarch64 builds — and the installation docs now list a dedicated vLLM-Metal backend as the Metal path for Apple Silicon. Treat it as new: expect rough edges, and do not trust performance numbers for it yet.

2. Serve a model with vllm serve

vllm serve <model>

That is the whole server: it loads the model and starts an OpenAI-compatible endpoint at http://localhost:8000 by default. The flags that matter:

--host — defaults to "0.0.0.0", which binds all interfaces. Fine for a single-box dev run, but do not expose it to a network without auth.
--port — defaults to 8000.
VLLM_HOST, VLLM_PORT, VLLM_API_KEY — the environment-variable equivalents; VLLM_API_KEY is the built-in way to gate the API.

The conservative local form:

vllm serve <model> --host 127.0.0.1

3. Read the startup memory log

The first serve prints a wall of memory accounting. The line that scares people:

Free memory ...

That is an INFO-level log, not an error — it is printed when the KV cache size is not pinned, as vLLM reports what the GPU has free before deciding how much to give the cache. The actual tuning dials live in the conserving-memory reference: --gpu-memory-utilization sets how much of the GPU’s memory the server may use, and the KV cache sizing is where the rest goes. If the server OOMs at startup, lower the utilization target before reaching for a bigger card.

4. Call the OpenAI-compatible API

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<model>", "messages": [{"role": "user", "content": "Hello"}]}'

The README lists OpenAI-compatible, Anthropic, and gRPC APIs, so existing clients usually need only a base-URL change.

5. Fix CUDA driver errors

The two errors that cover most vLLM startup failures:

“CUDA driver version is insufficient for CUDA runtime version” — the wheel’s CUDA runtime is newer than your driver’s CUDA. Update the driver, pick a wheel matching your driver, or (stuck on 12.8) build from source.

CUDA error: no kernel image is available for execution on the device — a compute-capability or driver mismatch: the GPU is not in the wheel’s build matrix, or the driver is too old for the kernels it ships. Check the wheel’s supported-architecture list against your card before debugging deeper.

What to expect

Llama 3.1, one community benchmark harness (vLLM 0.21.0 era, PyTorch 2.6 / CUDA 12.4, driver 550.127; batch 1, 1k prompt / 256 generated) — a reference harness, not a v0.32.0 measurement:

Hardware	Model	Quant	Speed
1× RTX 3090 24 GB	Llama 3.1 8B	FP16	47.3 tok/s
1× RTX 3090 24 GB	Llama 3.1 8B	GPTQ-4	112.4 tok/s
1× RTX 3090 24 GB	Llama 3.1 8B	AWQ-4	108.1 tok/s
(same card)	Llama 3.1 8B	GGUF Q4_K_M (llama.cpp)	96.8 tok/s
4× RTX 3090 (TP=4)	Llama 3.1 70B	GPTQ-4	18.2 tok/s (peak 21.6 GB VRAM per GPU)
4× RTX 3090 (TP=4)	Llama 3.1 70B	AWQ-4	17.4 tok/s

Read the table twice before drawing conclusions. FP16 is the slowest row at batch 1 — vLLM’s advantage is quantized weights plus continuous batching, not raw precision. And the quality cost of the 4-bit columns is small: MMLU scores land at 95.2% (AWQ) and 94.8% (GPTQ group-64) of the FP16 baseline, versus 99.3% for Q8_0 and 91.5% for GGUF Q4_K_M.

Troubleshooting

“CUDA driver version is insufficient for CUDA runtime version” — driver/runtime mismatch. Update the driver first; if the driver cannot move, build from source against the CUDA you have.

CUDA error: no kernel image is available for execution on the device — the wheel has no kernels for your GPU’s compute capability, or the driver is too old. This is a build-matrix problem, not a config one: match wheel to card, or build.

A scary “Free memory” line at startup — INFO level, not an error. It is the memory report printed when the KV cache size is unset. The server continues; read on.

The API is reachable from the whole network — --host defaults to 0.0.0.0. Bind 127.0.0.1 for local use, or set VLLM_API_KEY and put a reverse proxy in front when you expose it.

FAQ

Is vLLM faster than llama.cpp? On the 3090 harness, at batch 1: quantized vLLM (112.4 GPTQ-4) beats llama.cpp GGUF Q4_K_M (96.8), while FP16 vLLM (47.3) loses. vLLM is built for sustained throughput — continuous batching — so the gap grows with concurrency, not just at batch 1.

Can I run vLLM on a Mac? Two paths: the new vLLM-Metal backend and CPU wheels (Apple Silicon ARM64, ~30 MB). Both are young, and there are no trustworthy performance numbers for them yet — expect the Metal path to keep moving.

Why does it report “Free memory” at startup? Because the KV cache size was not pinned — the line is an INFO log describing what the GPU had free before the cache was sized, not an error.

Do I need a Hugging Face account? For most models, no — public model IDs work directly. Gated models need a token.

That is the serving path. If you want the same model with a lighter footprint, the llama.cpp guide is the CPU-friendly sibling, and the Ollama on Linux guide covers the zero-config route. The vLLM review in the Software pillar covers architecture this guide skips, and the AI Stack Builder maps your VRAM and budget to a GPU, model, and runner.

Sources

vllm-project/vllm README — feature list (PagedAttention, continuous batching, 200+ architectures, OpenAI/Anthropic/gRPC APIs), uv pip install vllm, vllm download-kernels (raw main branch, verified working).
vLLM v0.32.0 release — published 2026-10-18; CPU wheels vllm-0.32.0+cpu-cp38-abi3-macosx_11_0_arm64.whl (~30 MB) and manylinux aarch64.
vLLM docs — CLI: serve — vllm serve <model>, default http://localhost:8000, --host default "0.0.0.0", --port default 8000, VLLM_HOST/VLLM_PORT/VLLM_API_KEY (HTTP 200, verified).
vLLM docs — Installation — Prebuilt Wheels / Requirements / Python setup / vLLM-Metal / Build from source / Docker paths.
vLLM docs — Conserving memory — the “Free memory” line is INFO-level; KV cache sizing and --gpu-memory-utilization.
discuss.vllm.ai — CUDA 12.8 not working — staff reply (June 2026): current prebuilt wheels are CUDA 12.9/13.0; 12.8 builds from source.
vllm-project/vllm issue #31170 — the CUDA error: no kernel image is available for execution on the device string.
cap10bad/llm-inference-benchmarks — 1× 3090: 47.3 / 112.4 / 108.1 tok/s and the llama.cpp comparison; 4× 3090 TP=4 70B: 18.2 / 17.4 tok/s (harness: vLLM 0.21.0 era, 2025–2026); MMLU quality figures.

How to Run a Local LLM Server with vLLM