What Is Retrieval-Augmented Generation (RAG)? A Plain-English Guide

Retrieval-augmented generation (RAG) is an AI architecture that improves large language model (LLM) outputs by first retrieving relevant documents from a knowledge base, then passing those documents as context to the LLM before generating a response — grounding the model's output in specific, current, or private knowledge that the base model wasn't trained on.

RAG was introduced by Meta AI researchers in a 2020 paper (Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks) and has become the dominant approach for building AI systems that need to answer questions about specific, up-to-date, or proprietary content.

---

Why RAG Exists: The LLM Knowledge Problem

Large language models are trained on static snapshots of text data with a knowledge cutoff date. This creates three problems:

Staleness: The model doesn't know about events, publications, or changes after its training cutoff. Ask GPT-4 about a company's current pricing, and you'll get outdated information.

Hallucination: When asked about specific facts outside its training data, an LLM will sometimes generate plausible-sounding but incorrect answers. Without access to authoritative sources, the model can't distinguish what it knows from what it's confabulating.

Private knowledge: A model trained on public internet data doesn't know your company's internal documentation, your proprietary research, or your customer database. It can't answer questions about private information.

RAG addresses all three by retrieving relevant, authoritative documents at query time and providing them to the model as explicit context.

---

How Retrieval-Augmented Generation Works

A RAG system has two main phases:

Phase 1: Indexing (Offline)

Before any queries run, documents are processed and stored in a way that enables fast retrieval:

1. Document chunking: Source documents (PDFs, web pages, documentation, database records) are split into chunks of typically 512–1,024 tokens.
2. Embedding: Each chunk is converted to a vector embedding — a numerical representation of its semantic meaning — using an embedding model (OpenAI's text-embedding-3-small, Cohere's embed-v3, etc.).
3. Storage: Embeddings are stored in a vector database (Pinecone, Weaviate, Chroma, pgvector in PostgreSQL).

Phase 2: Retrieval + Generation (Online, per query)

When a user submits a query:

1. Query embedding: The query is converted to a vector embedding using the same model.
2. Semantic search: The vector database finds the chunks whose embeddings are most similar to the query embedding (nearest neighbor search).
3. Context assembly: The retrieved chunks are assembled into a context prompt.
4. Generation: The LLM receives the query + retrieved context and generates a response grounded in the specific retrieved documents.

---

A Worked Example

A software team builds a RAG system over their internal engineering documentation (Confluence, ADRs, runbooks):

Indexing: 3,000 Confluence pages → chunked → embedded → stored in pgvector.

Query: An engineer asks "What's our standard approach for database migrations in the payments service?"

Retrieval: Vector search finds the 5 most relevant chunks: the payments service ADR on migration strategy, a runbook for zero-downtime migrations, and a past incident review that added a constraint.

Context: These 5 chunks are assembled into the prompt: "Answer based on the following documents: [chunk 1]... [chunk 2]..."

Generation: The LLM reads the retrieved documents and generates: "Based on your ADR from March 2024, the payments service uses..." — with a response grounded in the actual internal documentation.

Without RAG: the LLM would either say "I don't have information about your internal systems" or, worse, hallucinate a plausible-sounding but wrong answer about standard migration practices.

---

RAG vs. Fine-Tuning: When to Use Which

	RAG	Fine-Tuning
Best for	Factual Q&A over specific docs	Adapting model style/format/behavior
Knowledge updates	Real-time (update the index)	Requires re-training
Private data	Yes (retrieved at query time)	Risky (data embedded in weights)
Cost	Per-query retrieval cost	High upfront training cost
Hallucination risk	Lower (grounded in retrieved docs)	Higher for specific facts
When to choose	Your problem is "the model doesn't know X"	Your problem is "the model doesn't behave like X"

The most common mistake: choosing fine-tuning when the real problem is a knowledge gap. Fine-tuning trains the model to behave differently; it does not reliably add new facts. If you want a model to know your documentation, RAG is the right tool.

---

Common Misconceptions About RAG

"RAG eliminates hallucinations." RAG reduces hallucinations by providing authoritative context, but doesn't eliminate them. The LLM can still misinterpret retrieved documents, cherry-pick details, or generate text that goes beyond what the retrieved context supports. RAG should always be combined with response validation for high-stakes use cases.

"Larger chunks always work better." Chunk size is a real hyperparameter that affects RAG quality. Too small: individual chunks lack enough context. Too large: chunks contain off-topic information that dilutes retrieval relevance. 512–1,024 tokens is a common starting range, but optimal chunk size depends on your document structure.

"RAG requires a dedicated vector database." Vector search is available in PostgreSQL (pgvector), SQLite (sqlite-vss), and many standard databases. You don't need Pinecone or Weaviate to build a RAG system — they provide scale and performance advantages, but for a small-to-medium document corpus, an existing database with vector extension works.

"Semantic search is always better than keyword search." Hybrid retrieval — combining vector (semantic) search with BM25 (keyword) search — consistently outperforms either alone. Semantic search misses exact-match queries; keyword search misses paraphrased concepts. Most production RAG systems use hybrid retrieval.

---

Where WebSnips Fits

WebSnips implements a lightweight version of the RAG principle for personal knowledge workers:

• The knowledge base: Your WebSnips library of saved web articles, curated by you
• The retrieval: Creator Studio lets you select which saves to draw from for a given writing task
• The generation: Creator Studio generates a cited first draft from your selected sources

This is "personal RAG" — instead of embedding a company's documentation, you're drawing from your personal research library. The principle is the same: ground the AI's output in specific, curated sources rather than letting it draw on general training data.

For developers who also do research writing, WebSnips provides the curated source layer that general-purpose AI tools lack.

---

Related Concepts

• Personal Knowledge Management: The human analog to RAG — building personal systems that retrieve the right knowledge at the right time
• Vector Embeddings: The numerical representations that enable semantic search in RAG
• Semantic Search: Finding documents by meaning rather than exact keyword match
• Fine-Tuning: The alternative to RAG — adapting model behavior rather than adding knowledge

---

FAQ

What's the best vector database for a new RAG project? For prototyping: Chroma (simple, local, Python-native). For production on existing PostgreSQL: pgvector. For scale with managed infrastructure: Pinecone or Weaviate. For teams already on Supabase: Supabase vector (pgvector). Don't over-engineer the vector DB choice early — the retrieval quality depends far more on chunking and embedding model choice than on the database itself.

How do I evaluate whether my RAG system is working? Use the RAGAS framework (Retrieval-Augmented Generation Assessment): measures context recall (did retrieval find the right chunks?), context precision (were the retrieved chunks relevant?), and answer correctness (did the generation accurately use what was retrieved?). Each dimension can fail independently — you need to measure all three to know where to improve.

Can I build RAG with open-source models? Yes — RAG works with any LLM capable of following instructions. LLaMA 3, Mistral, and Qwen are capable local models. Ollama (local model serving) + ChromaDB (local vector store) + LangChain or LlamaIndex (RAG orchestration) is a fully open-source stack that runs locally without API costs.

What's the context window limit for RAG? RAG retrieval should stay within the LLM's context window, which limits how many chunks can be provided per query. Modern models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) support 128K–1M token context windows, dramatically expanding what can be retrieved. For large document sets, re-ranking retrieved chunks before passing them to the LLM helps prioritize the most relevant content within the context budget.

---

Conclusion

Retrieval-augmented generation (RAG) is the standard approach for building AI systems that need to answer questions about specific, current, or private knowledge — indexing documents as vector embeddings, retrieving the most relevant chunks at query time, and providing them as context to the LLM.

For developers building on internal documentation, engineering knowledge bases, or any domain where factual accuracy matters: RAG is almost always the right choice over fine-tuning when the problem is a knowledge gap.

The fundamentals: quality document ingestion, good chunking, hybrid retrieval, and evaluation with RAGAS. The database and framework matter less than these.

Try WebSnips free if you do research-to-writing work — a curated personal knowledge base with AI generation from your selected sources.