Ollama

Ollama is the default answer to “how do I run a local LLM?” Install it, run ollama run llama3.1:8b, and you are chatting with a private model in under five minutes.

Strengths: the smoothest onboarding in the space, a curated model library, and an OpenAI-compatible API that every tool speaks. GPU detection is automatic.

Limitations: it is a wrapper around llama.cpp — when you need exotic quantizations, speculative decoding or custom schedulers, go to llama.cpp directly.

Best for: beginners, developers prototyping, and RAG backends. See the guide: Run a local LLM on Windows with Ollama.