Written by Jakub Rusinowski · Last updated 2026-07-08 · Prices verified 2026-03-01
RAG is the workload where local has the strongest claim: retrieval quality matters more than raw model size (a well-tuned local pipeline with a 30B model routinely beats a frontier model over a worse index), the documents themselves are usually the sensitive asset, and RAG's context-stuffed prompts burn input tokens at a rate that makes metered APIs expensive. At ~2M tokens/day a used RTX 3090 build breaks even against GPT-4o in about 5 months — and your knowledge base never leaves your network.
RAG-specific dimensions — this workload is not generic chat.
| Dimension | Local | Cloud |
|---|---|---|
| Where documents & embeddings live | Your disk, your vector store | Uploaded to provider (or a second vendor for embeddings) |
| Token economics | Context-stuffing is free — electricity only | RAG is input-heavy; every retrieved chunk is billed per query |
| Answer quality driver | Retrieval tuning (chunking, embeddings, reranking) | Same — model tier helps less than people expect |
| Embedding model | Open embedders (bge, nomic) run on CPU/GPU free | Per-token embedding fees at indexing and query time |
| Re-indexing freedom | Re-embed the corpus nightly if you like | Re-indexing a large corpus is a billable event |
| Long-context alternative | Limited — VRAM caps context | 1M-token contexts can sometimes skip RAG entirely |
| Cost at 2M tok/day | ~$24/mo electricity (+$52/mo amortization) | ~$285/mo (GPT-4o) / ~$17 (Mini-class) |
Internal knowledge-base assistant: ~2M tokens/day, heavily input-weighted — 2M tokens/day, Used RTX 3090 24GB Value King vs GPT-4o (OpenAI), electricity $0.15/kWh. Adjust every input in the interactive calculator on this page.
| Cloud cost / month | $285 (GPT-4o, $2.5/M input + $10/M output) |
| Local cost / month (24-mo TCO) | $76.11 — $24.44 electricity + hardware amortization |
| Hardware up-front | $1,240 (Used RTX 3090 24GB Value King) |
| Break-even | Month 5 — cumulative cloud spend passes local |
Estimates: 70/30 input/output mix, 24-month amortization, no resale value, load-time electricity only. Cloud prices last verified: 2026-03-01. Hardware street price checked: 2026-07-06.
For generic chat, the frontier model wins on quality and the argument for local is cost/privacy. RAG scrambles that: answer quality in retrieval-augmented systems is dominated by whether the right chunks reach the prompt — chunking strategy, embedding quality, reranking, query rewriting — and only secondarily by the generator model's tier. A 30B local model reading the right three paragraphs beats GPT-4o reading the wrong ones, every time. Which means the part of the stack worth obsessing over runs identically well on your own hardware, and the part where the frontier is better (generation) is the part RAG makes matter less. Our local RAG guide covers the tuning that actually moves quality.
RAG prompts are stuffed: system prompt + query + five or ten retrieved chunks, often 3,000–8,000 input tokens per question answered in 200. That 90%+ input weighting makes RAG the most expensive-per-interaction chat workload on metered APIs — and the cheapest locally, where context is just VRAM you already own. At ~2M tokens/day (a moderately busy internal knowledge assistant), GPT-4o runs ~$285/month; the same pipeline on a used RTX 3090 build costs ~$24/month of electricity, breaking even on the $1,240 hardware in about 5 months. Embeddings compound this: cloud embedding APIs bill re-indexing your corpus every time you improve chunking (and you will, repeatedly, while tuning), whereas open embedders like bge or nomic-embed re-index nightly for free.
The honest counterweight: mini-class models are cheap even at RAG volumes (~$17/month at our preset), and if your generator can be Mini-class *and* your documents aren't sensitive, cloud stays the budget option. RAG's local case is strongest when either quality needs exceed Mini-class or the corpus itself is the constraint — which brings us to the real issue.
Ask what's actually *in* the knowledge base: contracts, internal wikis, customer tickets, research notes, the accumulated institutional memory of your organization. A cloud RAG deployment doesn't send the occasional prompt to a third party — it uploads the entire corpus (to the embedding provider, often a vector-DB SaaS, and chunks of it to the LLM on every query). That's three vendors holding your documents to run one Q&A system. A fully local pipeline — open embedder, local vector store (Chroma/Qdrant on your disk), local generator — holds the corpus at exactly one location: yours. For anything under NDA, GDPR-sensitive, or simply strategic, this is the argument that ends meetings; the GDPR comparison makes the compliance version of it.
Three real cases. Million-token contexts: Gemini-class models can sometimes skip RAG entirely — paste the whole corpus per query. Elegant for small corpora (and billed accordingly; it makes economic sense only at low query volume). Managed pipelines: the assistants-with-file-search products get a prototype running in an afternoon with zero retrieval engineering — the right first step to prove value before building anything. Very sparse usage: a knowledge base queried ten times a day doesn't justify hardware; a $5/month cloud bill does.
One used RTX 3090 (24 GB) carries the whole thing: Qwen 3 32B Q4 as generator (~19 GB), nomic-embed on the side, Qdrant or Chroma for vectors, orchestrated by LlamaIndex or a few hundred lines of your own Python. That's the $1,240 build; a 16 GB card with a 14B generator is the budget version and loses less quality than you'd guess (retrieval dominates, again). Prototype in the cloud if you like — then bring the corpus home.
RAG is the strongest local use case on this site: the corpus is usually the most sensitive thing an organization owns, the token pattern (huge inputs, endless re-indexing) is exactly what metered pricing punishes, and answer quality hinges on retrieval work that runs the same everywhere. Prototype on a managed cloud pipeline to prove the value, then move the corpus onto your own hardware — a used RTX 3090 carries a full production pipeline and pays for itself within half a year at moderate volume.
Not sure which tier fits? The build recommender maps budgets to complete part lists — or check what your existing GPU already runs for free.
Methodology & assumptions. All cost figures are estimates from one shared model (lib/costCompare.ts): cloud costs = tokens/day × published per-1M-token prices at a 70/30 input/output split × 30 days; local costs = hardware amortized over 24 months with no resale value + electricity for load time only at your rate, with machine count scaled when volume exceeds one machine’s throughput. Cloud prices carry per-entry source URLs and verification dates; hardware prices come from the curated /build catalog (street prices with check dates). Real bills vary with usage mix, discounts, and idle power — treat break-even months as directional, not contractual.