Run a private RAG knowledge base on a single GPU

The second item in 5 things you can do with a local LLM was document Q&A on sensitive files. This is the deep-dive: how to build a Retrieval-Augmented Generation (RAG) stack that runs entirely on your hardware, what each piece does, which model to pick, and the honest limits of doing retrieval locally.

The use case is sharp: you have documents — contracts, medical records, a confidential research dump, internal wikis, legal correspondence — that absolutely cannot go to a third-party embedding or chat API. You want to ask questions of that corpus and get sourced answers. RAG is how. Local RAG is how you do it without the privacy asterisk.

What RAG actually is (one paragraph)

A language model alone can't "read your documents" — it has a fixed context window, and your corpus doesn't fit. RAG solves this in two steps: (1) at index time, every document is chunked and each chunk is turned into a vector embedding by a separate (smaller) model, then stored in a vector database; (2) at query time, the user's question is embedded the same way, the database returns the closest-matching chunks, and those chunks are inserted into the language model's prompt as context. The model then answers with citations to the retrieved passages. The language model never "learned" your documents — it's reading the relevant excerpts at query time.

The privacy implication: if both the embedding model and the language model run locally, no document text and no query ever leaves your machine. That's the whole point.

The four pieces, all local

1. The embedding model — turns text into vectors. Small, fast, runs on CPU or a sliver of GPU. Common open choices include BGE, Nomic Embed, and the gte family. These are 1B-parameter-class models — trivial to run locally.
2. The vector store — holds the embeddings and does similarity search. Options range from a local file-backed store (Chroma, Qdrant, LanceDB) for personal corpora to a Postgres+pgvector setup for larger ones. For most personal/private use, a local embedded store is plenty.
3. The language model — reads the retrieved chunks and answers. Same local model server as everything else (Ollama for ease).
4. The orchestration layer — the glue: chunk the docs, embed them, query the store, build the prompt, call the LLM. This is what a framework (LangChain, LlamaIndex) or a packaged RAG UI (Open WebUI's document feature, AnythingLLM, etc.) gives you.

The packaged-UI path is the right starting point: you get a web interface where you upload documents, pick a model, and chat with citations — no orchestration code to write. Move to a code-first framework when the packaged UI's chunking or retrieval doesn't fit your corpus.

Model choices

Embeddings: any of the open BGE / Nomic / gte models is fine to start. Embedding quality matters, but for most personal corpora the differences are second-order compared to chunking strategy. Don't overthink the first pick.

Language model: the same recommendations as the self-host starter apply, with one twist — for RAG, long context is valuable, because retrieving more chunks per query (and giving the model room to reason over them) improves answer quality. Lean toward the larger-context options in your VRAM tier. A 30B-A3B MoE (why MoE) at Q4_K_M is the same workhorse default as elsewhere; on a 12GB card, an 8B with a healthy context window is the realistic fit. Verify with Model Fit.

What it's good at

• "What does this contract say about X" — RAG's sweet spot. Retrieve the clauses mentioning X, ask the model to summarize or extract the obligation. Sourced, fast, repeatable.
• Summarizing long PDFs — chunk, embed, retrieve the most relevant passages, summarize. Handles 200-page documents the model could never fit in raw context.
• Finding contradictions across documents — "do these three contracts say different things about termination notice?" Retrieve the relevant clauses from each, ask the model to compare.
• Internal-wiki Q&A — point the stack at your team's docs; new hires get a sourced answer bot that never leaks the docs to a vendor.
• Personal research assistant — a folder of papers and notes you can interrogate, with every answer pointing back to the source chunk.

Where it falls down

• Precise numerical reasoning across many documents. "Add up the consideration across these 47 contracts" — retrieval surfaces the clauses but the model can still fumble the arithmetic. Use RAG for finding the numbers; do the math yourself or with a script.
• Citation quality depends on retrieval quality. If the wrong chunks get retrieved, the model answers from the wrong context — confidently. Always click through to the source passage. RAG hallucination looks different from raw LLM hallucination (it's "right document, wrong reading" or "wrong document, confident summary"), but it still happens.
• Tables, scans, and images in PDFs. Plain-text extraction misses table structure and OCR errors propagate into embeddings. Pre-process carefully; a RAG stack is only as good as the text that goes into it.
• Throughput on big corpora. Indexing thousands of documents through a local embedding model takes real time. It's a one-time-ish cost (re-index when the corpus changes), but plan for it.

Why local matters here, specifically

Document Q&A is the use case where the privacy argument is least optional. The whole value of RAG is asking questions of your documents — and the moment those documents transit a third-party embedding API or chat endpoint, you've lost the property that made RAG worth building. For anything under NDA, HIPAA, attorney-client privilege, or just commercial sensitivity, local isn't a preference; it's the only compliant shape.

This is also a use case where the cost argument compounds. Embedding a large corpus once is expensive through an API; embedding it locally is the electricity. Querying it a thousand times is expensive through an API; locally, fixed cost. The break-even on a RAG stack comes fast.

A reasonable starting config

• Hardware: any 12GB+ card (B580 / RTX 3060 / RX 7600 XT class); embeddings are light, the LLM is the constraint. See Builds.
• Vector store: a local embedded store (Chroma or LanceDB) for personal corpora; Postgres+pgvector if you want it to scale and already run Postgres.
• Embedding model: an open BGE or Nomic Embed model, via Ollama or directly.
• Language model: Qwen3-Coder-30B-A3B Q4_K_M (general-capable despite the coder lean) or a same-tier general model, with as much context as your VRAM allows.
• Orchestration: a packaged RAG UI to start; a code-first framework (LlamaIndex / LangChain) when you outgrow it.

The takeaway

Private RAG is the workflow where "local" stops being a preference and becomes the only compliant architecture. The stack is four well-understood pieces, all of which run on a single consumer GPU, and the result is a sourced-answer bot over your own documents that no vendor ever sees. Start with a packaged UI, accept the honest limits (retrieval can be wrong; numbers need verification; PDF extraction needs care), and you have something genuinely useful — and genuinely yours.

The runtime setup is the same as the self-host starter; the model-fit math for the LLM is at Model Fit. For the broader privacy framing — why local is the only compliant shape for sensitive workloads — start with why local AI matters in 2026.