Tutorial 12: Complete Local RAG Pipeline

By Jakub RusinowskiLast updated 12 min readAdvanced

Build "chat with your documents" yourself, in one short Python script: read your PDFs, split them into passages, embed them with a local model in Ollama, store them in a local vector database, and answer questions with citations. Nothing leaves your computer.

Which page is which. Want it working with no code at all? Use the recipe Chat with your PDFs privately (AnythingLLM, Open WebUI or LM Studio). Want to know what RAG is first? Read Intro to RAG. This page is the build-it-yourself version, for when you want to see and control every step.

What you'll build

  1. Extract the text of every page with pypdf.
  2. Chunk each page into overlapping passages.
  3. Embed each passage with an Ollama embedding model (embeddinggemma).
  4. Store passages, embeddings and their file and page number in Chroma, a vector database that lives in a folder on your disk.
  5. Retrieve the passages closest to a question, and answer with a local chat model that must cite the page.

1. Install

You need a recent Python 3 and Ollama.

bash
pip install ollama chromadb pypdf
ollama pull embeddinggemma

Then pull a chat model for the answers. The Model Finder picks one that fits your computer; any chat model you already have in Ollama works too.

2. The script

Save this as rag.py:

python
"""Local RAG: index a folder of PDFs with Ollama embeddings into Chroma, then ask questions."""
import argparse
import pathlib

import chromadb
import ollama
from pypdf import PdfReader

EMBED_MODEL = "embeddinggemma"   # an Ollama embedding model; change it and you must re-index
DB_PATH = "rag-db"               # where Chroma keeps the index on disk
CHUNK_CHARS = 1200               # passage length, in characters
OVERLAP_CHARS = 200              # how much consecutive passages share


def chunks(text: str):
    step = CHUNK_CHARS - OVERLAP_CHARS
    for start in range(0, len(text), step):
        yield text[start:start + CHUNK_CHARS]


def embed(texts: list[str]) -> list[list[float]]:
    return ollama.embed(model=EMBED_MODEL, input=texts)["embeddings"]


def collection():
    return chromadb.PersistentClient(path=DB_PATH).get_or_create_collection("docs")


def index(folder: str) -> None:
    col = collection()
    for pdf in sorted(pathlib.Path(folder).glob("*.pdf")):
        for page_no, page in enumerate(PdfReader(pdf).pages, start=1):
            text = (page.extract_text() or "").strip()
            if not text:
                print(f"  skipped {pdf.name} page {page_no}: no text layer (a scan needs OCR first)")
                continue
            parts = list(chunks(text))
            col.upsert(
                ids=[f"{pdf.name}:{page_no}:{i}" for i in range(len(parts))],
                documents=parts,
                embeddings=embed(parts),
                metadatas=[{"file": pdf.name, "page": page_no}] * len(parts),
            )
        print(f"indexed {pdf.name}")


def ask(question: str, model: str, k: int) -> None:
    hits = collection().query(query_embeddings=embed([question]), n_results=k)
    context = "\n\n".join(
        f"[{meta['file']} p.{meta['page']}]\n{doc}"
        for doc, meta in zip(hits["documents"][0], hits["metadatas"][0])
    )
    reply = ollama.chat(model=model, messages=[
        {"role": "system", "content": "Answer only from the context. After each claim, cite the [file p.N] tag it came from. "
                                      "If the context does not contain the answer, say you cannot find it."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ])
    print(reply["message"]["content"])


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    sub = p.add_subparsers(dest="cmd", required=True)
    sub.add_parser("index").add_argument("folder")
    a = sub.add_parser("ask")
    a.add_argument("question")
    a.add_argument("--model", required=True, help="a chat model you have pulled in Ollama")
    a.add_argument("-k", type=int, default=4, help="how many passages to retrieve")
    args = p.parse_args()
    if args.cmd == "index":
        index(args.folder)
    else:
        ask(args.question, args.model, args.k)

What each part does:

  • chunks() cuts a page into passages of CHUNK_CHARS characters that share OVERLAP_CHARS with the next one, so a sentence split at a boundary still appears whole in one passage.
  • embed() calls Ollama's embedding API. The same model must embed your documents and your questions.
  • index() stores each passage with its file name and page number, and uses upsert, so running it again on the same folder updates the index instead of duplicating it.
  • ask() retrieves the k closest passages and tells the chat model to answer only from them and to cite [file p.N].

3. Run it

Put your PDFs in a folder, then:

bash
python rag.py index ./my-pdfs
python rag.py ask "What is the notice period for ending the lease?" --model <your-chat-model>

A good answer quotes the right passage and ends with a tag such as [lease.pdf p.1]. Open that page and check it.

4. Make it better

  • Scanned PDFs have no text layer. pypdf cannot read text in images; the script skips those pages and says so. Run OCR first (for example Tesseract), then index the result.
  • Passage size. Short passages find precise facts; longer ones keep more context together. Change CHUNK_CHARS and OVERLAP_CHARS, delete the rag-db folder, and index again.
  • How many passages. Raise -k when answers miss things that are spread across pages. More passages need a longer context window in the chat model.
  • Changing the embedding model means re-indexing. Embeddings from different models do not mix: delete rag-db and run index again.
  • Other languages. embeddinggemma was trained on text in many languages, so non-English documents work without changes.
  • Summaries are not retrieval. "Summarise the whole document" needs the whole document, not four passages. Summarise page by page instead.

Deciding whether a retrieved chunk is actually relevant is a yes/no question, and a decision model gives you a probability you can threshold; how to choose that threshold is covered in its own guide.

Going further

← PreviousIntro to RAG: Chat with DocsNext →Building Autonomous Agents