vLLM

vLLM is a serving engine built for throughput: it keeps many concurrent conversations on one GPU using PagedAttention and continuous batching, which makes it the default choice when you self-host an API for a team or an app.

The built-in server speaks the OpenAI API dialect, so existing SDKs and clients work without changes:

vllm serve meta-llama/Llama-3.1-8B-Instruct

For a personal machine that runs one chat at a time, Ollama or LM Studio are simpler; reach for vLLM when you are serving many users or need maximum tokens per second.