Recipe

Chat with your PDFs privately

Ask questions about your own PDFs and get answers that cite the page, without uploading the files anywhere.

BeginnerDesktop appBrowserInstructions checked October 15, 2026

Written by Jakub Rusinowski · Updated

You install free desktop software (AnythingLLM with Ollama, or LM Studio on its own), download one small model, and drag your PDFs in. The app splits each file into passages, finds the ones that match your question and hands them to the model, which answers with a citation you can click. A computer with 16 GB of memory and no graphics card can do it; a graphics card or a Mac makes it faster.

Will this work on my computer?

The Model Finder's picks for “Ask my documents” on one modest machine of each kind. Estimated, not measured.
Your computerWhat the Model Finder sets upExpect
PC with 16 GB of RAM and no graphics cardQwen 3.5 4BUsable
PC with an 8 GB graphics cardQwen 3.5 4BSmooth
PC with a 24 GB graphics cardGemma 4 26B-A4BSmooth
M1 Mac with 16 GB of memoryQwen 3.5 9BUsable

A long document uses more memory than a short chat. If answers start missing things from long files, see Answers ignore half the document under Troubleshooting.

Check my exact hardware

What you'll build

  1. Your PDFs
  2. Text extraction
  3. Split into passages
  4. Embeddings (small local model)
  5. Find the passages closest to your question
  6. Chat model (local) writes the answer with a citation

Two things happen on your computer: an embedding model turns every passage into numbers so the app can find the right ones fast, and the chat model reads only those passages and writes the answer. That is why it can answer questions about a long manual without reading all of it at once, and also why it is weaker at "summarise this whole document" (see Troubleshooting).

Before you start

  • Windows, macOS or Linux.
  • Your PDFs on the same computer. If they are scans (photos of pages), you need text recognition (OCR); see Troubleshooting.
  • No account is needed for AnythingLLM Desktop or LM Studio. Open WebUI asks you to create a local admin account the first time; it stays on your machine.

Recommended stack

Pick: AnythingLLM Desktop with Ollama. AnythingLLM ships with its own embedding model and vector database, so there is nothing to wire together; Ollama runs the chat model.

If you want a browser interface: Ollama + Open WebUI. More settings, more control over how documents are split and searched, and folders that stay in sync.

Simplest of all for one file: LM Studio. Attach a PDF to a chat. If the document fits in the model's context, LM Studio gives the model the whole thing; if it is too long, it switches to searching for the relevant passages.

The chat model is the one the table above names for your computer.

Setup steps

Path A: AnythingLLM Desktop with Ollama (easiest)

  1. Download AnythingLLM Desktop for your system from anythingllm.com and install it.
  2. Install Ollama from ollama.com/download and pull the model the table names for your computer:
bash
ollama pull qwen3.5:4b       # for No GPU, 8 GB GPU
ollama pull gemma4:26b-a4b   # for 24 GB GPU
ollama pull qwen3.5:9b       # for Mac
  1. On AnythingLLM's first launch, choose Ollama as the model provider and select that model.
  2. Create a workspace (one per topic or project).
  3. Drag your PDFs into the chat, or click + in the prompt box.
  4. If AnythingLLM offers Embed, accept it: embedded documents are available to every thread in the workspace, attached ones only to the current chat. Labels can differ slightly between versions.
  5. Ask a question and click the citation under the answer to see which passage it used.

Path B: Ollama + Open WebUI

  1. Install Ollama and pull the model, as in Path A.
  2. Install Open WebUI. With Docker, and Ollama on the same computer:
bash
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  -e WEBUI_SECRET_KEY=your-secret-key \
  --name open-webui --restart always ghcr.io/open-webui/open-webui:main
  1. Replace your-secret-key with the output of openssl rand -hex 32. Without Docker: pip install open-webui, then open-webui serve.
  2. Open http://localhost:3000 (Docker) or http://localhost:8080 (pip) and create the admin account.
  3. Go to Workspace > Knowledge > Create, name it, upload your PDFs and wait for processing to finish.
  4. In a chat, type # and pick the knowledge base.
  5. For a folder you keep adding to, Add Content > Sync directory mirrors it incrementally.

Our full install walkthrough: Open WebUI + Ollama setup.

Path C: LM Studio (one document)

  1. Install LM Studio from lmstudio.ai and download the model the table names.
  2. Attach the PDF to the chat. LM Studio accepts .pdf, .docx and .txt.
  3. Ask your question using words you expect to appear in the document; that is LM Studio's own tip for better retrieval.

Test it

Pick a PDF you know well and ask something with one exact answer on one page:

"In the lease, what is the notice period for ending the contract, and which clause says so?"

A good answer quotes or closely paraphrases the clause, names it, and its citation opens the right passage. A bad answer is fluent but has no citation, or cites a passage that does not say it.

Verify it worked

  • The answer cites a passage, and that passage really contains the answer.
  • Ask about something that is not in the document. A good setup says it cannot find it; a weak one invents an answer.
  • Turn off Wi-Fi and ask again. It still answers.

Did you get this working?

Troubleshooting

  • It cannot find text in a scanned PDF. Scans are pictures of text. In Open WebUI, switch the content extraction engine (Tika or Docling) and turn on PDF Extract Images (OCR) in Settings > Admin > Documents. Check the PDF opens normally in a PDF viewer first.
  • "Summarise this document" gives a thin answer. Retrieval hands the model a few passages, not the whole file. In Open WebUI, switch the attached knowledge to Full Context; in LM Studio, use a document short enough to fit. For long documents, summarise section by section.
  • Answers ignore half the document. The model's context window is too small, so the app trims what it sends. In Open WebUI, raise the context length under Settings > Admin > Models, Advanced Params. What a longer context costs in memory: context length and out-of-memory errors.
  • Tables come out scrambled. Text extraction reads tables as loose lines. Try another extraction engine, or paste the table into the chat as text.
  • Your documents are not in English. The default embedders in AnythingLLM and Open WebUI (all-MiniLM-L6-v2) are trained mainly on English. In Open WebUI, switch the embedding engine to Ollama and a multilingual embedder such as bge-m3 (ollama pull bge-m3), then re-index every document in Settings > Admin > Documents: embeddings from different models do not mix.
  • Open WebUI cannot see Ollama. Connection refused fixes.

What stays on your computer

ComponentRuns whereNote
Chat model (Ollama, LM Studio)On your computerUnless you pick an Ollama cloud model (a :cloud tag) or connect a cloud provider's API key
Embedding modelOn your computer, after a one-time downloadAnythingLLM downloads its built-in embedder on the first embed; Open WebUI's default embedder is local too
Text extraction and OCROn your computer, or online if you choose a cloud optionOpen WebUI can also use cloud extractors such as Azure or Mistral OCR; choose a local one for private files
Your PDFs and the search indexOn your computerStored by the app on your disk
App and model downloadsOn your computer, after a one-time downloadNeeds the internet once; after that it works offline

Ollama cloud models run on Ollama's servers, not on your computer. To make sure nothing is sent, set OLLAMA_NO_CLOUD=1 (or "disable_ollama_cloud": true in ~/.ollama/server.json) and restart Ollama.

Go deeper

Sources

The steps on this page were checked against these official pages on the date shown.