By the end of this guide, Llama 3.1 8B will be answering questions from a Linux box — served by a systemd unit under a dedicated ollama user, with the API on port 11434. One curl line installs everything, and the GPU support matrix is wider on Linux than on any other OS.

The numbers up front: the default llama3.1:8b is a 4.9 GB Q4_K_M download, and in one community benchmark the same model hit 63 tok/s on an RTX 4000 Ada (20 GB). The real question on Linux is rarely speed — it is whether your GPU gets detected at all, and that has a specific log signature you will learn in step 4.

Requirements

A Linux distro with systemd (Ubuntu, Debian, Fedora, Arch — the docs’ service wiring assumes it).
NVIDIA: compute capability 5.0+ and driver 550+. Cards with CC 5.0–6.2 — the P40 is the classic example (CC 6.1) — need driver 570+.
AMD: a ROCm v7 driver, or a Vulkan-capable driver for the extended GPU list.
4 GB of disk for Ollama, plus room for models.

1. Install with the one-liner (or manual tarball)

curl -fsSL https://ollama.com/install.sh | sh

The script installs the binary, the ollama system user, and the systemd unit in one pass. As of October 2026 the latest release is v0.40.11 (published 2026-10-24).

Manual install, if your environment blocks remote scripts:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr
ollama serve
ollama -v

AMD variants exist as separate tarballs: ollama-linux-amd64-rocm.tar.zst, and ARM64 as ollama-linux-arm64.tar.zst. To upgrade a manual install, remove the old library first: sudo rm -rf /usr/lib/ollama, then re-extract.

The systemd wiring for a manual install is:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)
systemctl daemon-reload
systemctl enable ollama

with the service file in /etc/systemd/system/ollama.service pointing ExecStart at /usr/bin/ollama.

2. Verify the systemd service and logs

systemctl status ollama
journalctl -u ollama --no-pager --follow --pager-end

The status line should say active (running). The journal is where GPU detection is decided — keep it open in a second terminal before you pull a model, because the interesting lines appear at startup.

3. Pull and run llama3.1:8b

ollama pull llama3.1:8b
ollama run llama3.1:8b

Models land in /usr/share/ollama/.ollama under the ollama user. The default context is 4,096 tokens; raise it in-session with /set parameter num_ctx 8192, or server-wide with OLLAMA_CONTEXT_LENGTH=8192 in the service’s Environment section.

4. Check GPU detection in the journal

Start the server and watch:

journalctl -u ollama --no-pager --follow --pager-end

The GPU scan prints at INFO level, which is why grep -i error misses it:

msg="looking for compatible GPUs"
msg="no compatible GPUs were discovered"

That pair means Ollama ran but found nothing to offload to — the model will still run, on the CPU. Common causes and fixes:

Driver too old. The floor is 550+ (570+ for CC 5.0–6.2 cards like the P40). Check with nvidia-smi.
User not in the GPU groups. sudo usermod -aG render,video $(whoami), then log out and back in.
AMD gfx target mismatch. The log may show amdgpu detected, but no compatible rocm library found... or amdgpu is not supported (supported types:[gfx1030 gfx1100 ...]). Set HSA_OVERRIDE_GFX_VERSION to the nearest supported target, or use a smaller quant.
Docker. Run the container with --gpus=all and the NVIDIA container runtime.

To force a specific card, set CUDA_VISIBLE_DEVICES — list cards with nvidia-smi -L, or set -1 for CPU-only. After a suspend/resume cycle, if the GPU vanishes from the scan, reload the kernel module: sudo rmmod nvidia_uvm (and reload it per your distro’s procedure).

5. Expose the API if needed

By default the API binds to 127.0.0.1:11434. For other devices on the LAN, set OLLAMA_HOST=0.0.0.0:11434 in the service’s Environment section, then systemctl daemon-reload && systemctl restart ollama. Pin a specific release with the OLLAMA_VERSION environment variable before re-running the install script — the docs’ example (0.5.7) predates the current 0.40.11, so pin the version you actually want.

What to expect

Ollama on Linux supports the widest GPU matrix of any runner. From the official compute-capability table:

Compute capability	Example cards	Driver floor
12.0	RTX 5060–5090, RTX PRO 4000–6000 (Blackwell)	550+
9.0	H100, H200	550+
8.9	RTX 40xx	550+
8.6	RTX 30xx	550+
8.0	A100, A30	550+
7.5	GTX 1650 Ti, Titan RTX, RTX 20xx	550+
7.0	T4, Quadro	550+
6.1	P40, V100	570+
6.0	V100	570+
5.2	GTX 10xx	570+
5.0	GTX 980/970/960, Tesla M60/M40, K80, P6000	570+

AMD runs through ROCm v7 on Linux: the list covers RX 9070/9060/7900/7800/7700/7600 series, 6950 XT, 6900 XTX, 6800 XT, the AI PRO R9700/R9600D, the PRO W7900–V620 line, and Ryzen AI Max+ 395/390/385. “Additional AMD GPU support is provided by the Vulkan Library.”

Troubleshooting

msg="no compatible GPUs were discovered" — the INFO-level line that means CPU-only. Work through the step 4 checklist: driver version first, then group membership, then the AMD gfx target.

source=amd_linux.go:378 msg="amdgpu is not supported (supported types:[gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942])" — your AMD card’s gfx target is not in the compiled list. HSA_OVERRIDE_GFX_VERSION maps it to the nearest target; a smaller quant is the fallback.

Error: listen tcp 127.0.0.1:11434: bind: address already in use — two Ollamas. A running service plus a manual ollama serve in a terminal is the classic double-start: systemctl status ollama, ss -tlnp | grep 11434, kill the stray process — or run the extra instance on OLLAMA_HOST=127.0.0.1:11435.

GPU disappears after suspend/resume — sudo rmmod nvidia_uvm to clear the stale UVM state, then restart the service.

FAQ

Does Ollama on Linux need a GUI? No. The systemd install is headless-first — it is the natural choice for a server or a box you only reach over SSH. The API is the interface.

Can I run a P40 with Ollama on Linux? Yes — CC 6.1, with driver 570+. It is one of the cheapest cards that stays on the official support list.

Where do models live on Linux? Under the ollama system user, in /usr/share/ollama/.ollama. Uninstall cleanly: stop and disable the service, remove the files, userdel ollama, rm -r /usr/share/ollama.

AMD card, ROCm, still CPU — what next? Check the exact amdgpu lines in the journal, map your card’s gfx target against the supported list, and set HSA_OVERRIDE_GFX_VERSION if the mapping is off. If the card is outside the list entirely, Vulkan is the extended path.

That is the complete Linux path. The Ollama on macOS guide covers the other major OS, and the Ollama software review covers features this guide skips. To pick a runner for your exact box, use the runner picker, and for the model lineup see the Local LLMs pillar.

Sources

Ollama docs — Linux — the install one-liner, manual tarballs, systemd wiring, journalctl logs, updates and pinning, uninstall.
Ollama docs — GPU — the compute-capability table, driver floors (550+, 570+ for CC 5.0–6.2), CUDA_VISIBLE_DEVICES, the rmmod nvidia_uvm note, the AMD/ROCm v7 list.
ollama/ollama GitHub releases — v0.40.11, published 2026-10-24.
localaimaster — Ollama not using the GPU — the INFO-level looking for compatible GPUs / no compatible GPUs were discovered strings and the AMD amd_linux.go warnings.
ollama/ollama #11774 — a “no compatible GPUs” report with the journal excerpts.
ollama/ollama #14511 — the bind: address already in use error and its fix.

How to Run a Local LLM on Linux with Ollama