Skip to content

Build a Local RAG Pipeline with Ollama and LlamaIndex

intermediate general By AI Hub Team · updated

Retrieval-Augmented Generation (RAG) answers questions from your own documents instead of what a model memorized. The loop is short: an embedding model turns chunks into vectors, a local store keeps them, and a small LLM answers from the chunks it finds.

You can build the whole stack locally in about 50 lines of Python. The two models total 5.2 GB — a 274 MB embedding model and a 4.9 GB answerer — and it runs on the same machine as your chat. No cloud, no API keys. For a definition-level breakdown, see the RAG glossary entry.

Requirements

  • Ollama running on port 11434 (v0.40.10 as of October 2026) — install guide
  • Python 3 and pip
  • pip install ollama chromadb
  • Two pulls: nomic-embed-text (274 MB) and llama3.1:8b (4.9 GB)

That is the whole stack: one model server, one embedding package, one file-based vector store.

Pull an embedding model

Embedding models are a separate family from chat models: they turn text into a vector — an array of numbers — and semantically close texts end up close. Ollama ships its own library, and the small ones run comfortably on CPU:

ModelDimensionsDownloadMax contextNotes
all-minilm38446 MB256 tokenssmallest; good for a smoke test
nomic-embed-text768274 MB2K default windowmost-downloaded embedding model in the Ollama library
mxbai-embed-large1024670 MB512 tokensbest MTEB among the small models (64.68, English v1)
qwen3-embedding:4b25602.5 GB40Kquality step up, still fully local

Vector length depends on the model — Ollama’s docs say “typically 384–1024 dimensions” — and /api/embed returns L2-normalized vectors, so cosine similarity is the natural metric.

Recommendation: start with nomic-embed-text. It is 274 MB, 768-dimensional, and the most-downloaded embedding model in the Ollama library (92.8M downloads as of this writing). Two gotchas: embedding models can only generate embeddings — ollama run nomic-embed-text "hi" fails — and you must use the same model for indexing and querying.

ollama pull nomic-embed-text
ollama pull llama3.1:8b

Chunk and index your documents

A chunk is a piece of document small enough to embed in one call. The numbers: LlamaIndex defaults to 1024 tokens with 20 overlap; one community thread used 2000-token chunks with 200 overlap. Your ceiling is the embedding model’s context: mxbai-embed-large tops out at 512 tokens, and nomic-embed-text’s default window in Ollama is 2K — raise num_ctx for longer chunks.

Store chunks with metadata — source file, page, section — so answers can say where they came from:

import ollama
import chromadb

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.create_collection(name="docs")

chunks = ["..."]  # your text, split into chunks

for i, chunk in enumerate(chunks):
    response = ollama.embed(model="nomic-embed-text", input=chunk)
    collection.add(
        ids=[str(i)],
        embeddings=response["embeddings"],
        documents=[chunk],
        metadatas=[{"source": "file.md"}],
    )

Use PersistentClient, not Client(): the plain client keeps everything in memory, and the index vanishes on restart.

Retrieve and generate

The query is the same call in reverse: embed the question, take the nearest neighbors, feed the top chunk to the generator.

question = "What animals are llamas related to?"

response = ollama.embed(model="nomic-embed-text", input=question)
results = collection.query(
    query_embeddings=response["embeddings"],
    n_results=4,
)
data = results["documents"][0][0]

output = ollama.generate(
    model="llama3.1:8b",
    prompt=f"Using this data: {data}. Respond to this prompt: {question}",
)
print(output["response"])

Top-k matters. LlamaIndex’s default is 2, its docs example uses 4, and one community thread landed on 4. Start at 4 and check whether answers cite chunks they should not.

Scale up with LlamaIndex

For a real corpus, hand-rolled chunking gives way to a framework. Ollama officially integrates with LlamaIndex (and LangChain), and LlamaIndex’s own cookbook is a fully local pipeline with Chroma and Ollama. Packages:

pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama llama-index-vector-stores-chroma

(One AMD setup guide pins llama-index 0.12.10 with the Ollama integrations at 0.5.0 — a version snapshot, not a requirement.)

import chromadb
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.readers import SimpleDirectoryReader
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import OllamaLLM
from llama_index.vector_stores.chroma import ChromaVectorStore

vector_store = ChromaVectorStore(chroma_collection=chromadb.PersistentClient(path="./chroma_db").get_or_create_collection("docs"))
documents = SimpleDirectoryReader("./data").load_data()

index = VectorStoreIndex.from_documents(
    documents,
    storage_context=StorageContext.from_defaults(vector_store=vector_store),
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=20)],
    embed_model=OllamaEmbedding(model_name="nomic-embed-text", base_url="http://localhost:11434"),
)
engine = index.as_query_engine(
    similarity_top_k=4,
    llm=OllamaLLM(model="llama3.1:8b", base_url="http://localhost:11434"),
)
print(engine.query("What animals are llamas related to?"))

LlamaIndex handles reading the directory, splitting, embedding, storing and querying — the 512/20 split from the AMD guide, top-k 4.

What to expect

ComponentSizeWhere it runs
nomic-embed-text274 MBCPU is fine
llama3.1:8b (Q4_K_M)4.9 GBGPU if you have one; CPU works, slowly
Chroma indexa local directorydisk

Answer speed is what you get from a plain 8B chat: in one community benchmark (August 2026), 2.8 tok/s on a 4-vCPU/8 GB CPU and 63 tok/s on an RTX 4000 Ada 20 GB. RAG prompts add retrieved chunks to the input, so first-token time creeps up — per-model numbers are in Best 7B–8B LLMs for 8 GB VRAM. For sizing your machine: VRAM Calculator and How to Choose a GPU for Local AI.

Troubleshooting

  • curl: (7) Failed to connect to localhost port 11434 — Ollama is not running. Start it; every call in the pipeline hits that port.
  • 404 page not found from the Ollama API — endpoint mismatch. The current endpoint is /api/embed with an input field; the legacy /api/embeddings expects prompt. Swapping one for the other fails.
  • can only be used to generate embeddings — you asked an embedding model to chat. It cannot; use llama3.1:8b for answers.
  • Garbage retrieval, no error — the index was built with a different embedding model than the query. Re-index with the same model; there is no warning for this.
  • Chroma returns nothing after a restart — you used in-memory chromadb.Client(). Switch to PersistentClient and re-index.

FAQ

Do I need a GPU for RAG? No for embeddings — nomic-embed-text is 274 MB and runs on CPU. The answerer is where a GPU earns its keep: 63 tok/s versus 2.8 tok/s in one community benchmark.

Can I switch embedding models after indexing? Yes, but you must re-embed the whole corpus — dimensions differ (384, 768, 1024, 2560) and mixing models silently produces garbage retrieval.

What do dimensions mean? Vector length. More dimensions usually mean better retrieval at the cost of storage and speed. 384 (all-minilm) is fine for a first pass; 768 (nomic) is the default; 1024 and up when retrieval quality is the bottleneck.

Does this work on a Mac? Yes. Ollama, Chroma and LlamaIndex are all cross-platform, and model sizes are identical. If you are picking the runner itself, the Runner Picker starts from your OS and experience level.

Where do I go from here? The Task Finder has a RAG category for tools that do this for you, and the Software pillar covers the runners and frameworks in depth.

Sources