Retrieval-Augmented Generation
On RAG — the pragmatic solution to LLM hallucination and knowledge staleness. The pipeline, vector databases, chunking strategies, reranking, and why giving a language model a library card turns out to be more useful than making it memorize the library.
Retrieval-Augmented Generation
Every large language model has a dirty secret: it makes things up. Not occasionally, not as an edge case, but as a fundamental feature of how it operates. A language model generates the most probable continuation of its input, and sometimes the most probable continuation is a confident, fluent, entirely fabricated claim. This is the hallucination problem, and it is, in my estimation, the single biggest obstacle to deploying LLMs in any domain where being wrong has consequences — medicine, law, finance, engineering, and basically everything that matters.1
Retrieval-Augmented Generation (RAG) is the most practically successful response to this problem. The idea, first formalized by Lewis et al. (2020) at Meta AI, is conceptually simple: before the model generates an answer, retrieve relevant documents from an external knowledge base and include them in the context. Instead of asking the model to recall facts from its training data (which may be outdated, incomplete, or hallucinated), you give it the facts and ask it to synthesize. The model becomes a reader rather than a memorizer.2
This sounds like it should have been obvious. In retrospect, it was. But the path from “obvious idea” to “reliable production system” required solving a chain of engineering problems — retrieval, chunking, embedding, reranking, context management — that are individually tractable but collectively fiddly. RAG is less a technique than a pipeline, and the pipeline has a lot of joints where things can go wrong.
The Basic Pipeline
A standard RAG system has three phases:
1. Indexing. You take your knowledge base — documents, web pages, database records, PDFs, whatever — and convert it into a searchable format. This means splitting documents into chunks, computing embedding vectors for each chunk using a model like text-embedding-ada-002 or the open-source E5 family, and storing those vectors in a vector database.
2. Retrieval. When a user asks a question, you embed the query using the same embedding model, search the vector database for the \(k\) most similar chunks (typically via approximate nearest neighbor search), and optionally rerank the results using a more expensive cross-encoder model.
3. Generation. You concatenate the retrieved chunks into the language model’s context window along with the user’s question and a system prompt like “Answer the question based on the following context. If the answer is not in the context, say so.” The model generates its answer grounded in the retrieved documents.
This is the textbook version. Every production RAG system deviates from it, because every step hides decisions that dramatically affect quality.
Chunking: The Unglamorous Foundation
Chunking — how you split documents into retrieval units — is the least studied and most consequential part of the RAG pipeline. Get it wrong and nothing downstream can save you.
The naive approach is fixed-size chunking: split every document into 512-token segments. This is fast and simple and frequently terrible. A 512-token boundary might land in the middle of a sentence, splitting a concept across two chunks. A table might be separated from its caption. A section header might end up in a different chunk than the content it introduces.3
Better approaches include:
Recursive character splitting, which tries to split on paragraph boundaries first, then sentences, then words. LangChain popularized this and it’s become the default for many applications.
Semantic chunking, which uses embedding similarity to identify natural breakpoints — split where the topic changes, not where the character count reaches a threshold. Greg Kamradt’s work on this was influential.
Document structure-aware chunking, which respects the document’s own hierarchy — sections, subsections, paragraphs — and chunks accordingly. This requires parsing the document format (markdown headers, HTML tags, PDF structure), but preserves semantic coherence.
Proposition-level chunking, introduced by Chen et al. (2023), which uses an LLM to decompose text into atomic propositions before indexing. Each proposition is a self-contained factual claim. This maximizes retrieval precision at the cost of indexing compute.
The right chunking strategy depends on your documents and queries. Short, factual questions about specific claims favor small, precise chunks. Complex questions requiring synthesis of multiple paragraphs favor larger chunks with overlap. There is no universal best practice, and anyone claiming otherwise is selling a framework.4
Vector Databases and Embedding
The retrieval step requires an embedding model and a vector database. The embedding model converts text into dense vectors such that semantically similar texts land near each other in embedding space. The vector database stores millions or billions of these vectors and supports fast similarity search.
The embedding model matters more than most people realize. The MTEB leaderboard tracks embedding model performance across retrieval benchmarks, and the gap between the best and median models is large — 10-15 percentage points on standard retrieval metrics. As of early 2026, the frontier includes models like Cohere Embed v3, Voyage AI, and open-weight options like BGE-M3 and GTE from Alibaba. Instruction-tuned embedding models — where you can specify whether the input is a query or a document — consistently outperform their non-instruction-tuned counterparts.5
The vector database landscape has exploded. Pinecone, Weaviate, Qdrant, Milvus, Chroma — these are purpose-built for vector similarity search. But you can also get surprisingly far with PostgreSQL + pgvector or even a brute-force search over a NumPy array if your corpus is small enough. The choice of vector database is rarely the bottleneck. The choice of embedding model almost always is.
Hybrid Search: The Best of Both Worlds
Pure vector search has a well-known failure mode: it struggles with exact keyword matching. If your user asks about “error code E-4012” and your documents contain that exact string, a dense embedding model might retrieve documents about error codes in general rather than the specific one. This is because dense embeddings capture semantic similarity, not lexical overlap.
The solution is hybrid search — combining dense vector search with traditional sparse retrieval methods like BM25. BM25 excels at exact and near-exact matches; dense retrieval excels at semantic similarity. Together, they cover each other’s blind spots.
Most modern vector databases support hybrid search natively. Weaviate has a hybrid query mode with a configurable alpha parameter that blends BM25 and vector scores. Qdrant supports sparse vectors alongside dense ones. Pinecone offers hybrid search as a first-class feature.
In practice, I’ve found that hybrid search with alpha ≈ 0.7 (70% dense, 30% sparse) is a reasonable default for most applications. For domains with lots of jargon, codes, or proper nouns — legal, medical, technical support — increasing the sparse weight helps substantially. For open-ended, conversational queries, pure dense search is often fine.
The less-discussed variant is multi-vector retrieval, exemplified by ColBERT (Khattab & Zaharia, 2020), which represents each query and document as a set of vectors (one per token) and computes fine-grained token-level similarity. ColBERT achieves retrieval quality close to cross-encoders at search speeds close to single-vector methods. ColBERTv2 and its RAGatouille wrapper have made this practical for production systems.
Reranking: The Quality Multiplier
Retrieval is fast but imprecise. The top-\(k\) chunks from a vector search are approximately relevant, but their ordering is noisy and some genuinely irrelevant results slip through. Reranking — applying a more expensive model to rescore and reorder the retrieved chunks — is the single highest-leverage improvement you can make to a RAG pipeline.6
The standard approach is a cross-encoder reranker: a model that takes a (query, document) pair as input and outputs a relevance score. Unlike embedding models (which process query and document independently), cross-encoders can attend across both inputs, capturing fine-grained interactions. This makes them much more accurate but much slower — you can’t pre-compute embeddings, so you must run the model on every query-document pair at query time.
In practice, you retrieve a broad set (say, top 50 or 100) with fast vector search, then rerank down to the top 5-10 with a cross-encoder. Cohere Rerank, Jina Reranker, and open-source models like BGE-reranker and cross-encoder/ms-marco-MiniLM-L-12-v2 are common choices. The improvement is typically 5-15% on end-to-end answer quality metrics, which in production terms is the difference between “sometimes useful” and “reliable.”
Advanced RAG Patterns
The basic retrieve-and-generate pipeline is a starting point. Production systems have evolved substantially:
Query transformation. The user’s raw query is often a poor retrieval query. “What did the company say about revenue last quarter?” contains the user’s intent but lacks the specific terminology that would match the relevant document. HyDE (Hypothetical Document Embeddings) generates a hypothetical answer first and uses that as the retrieval query — on the theory that the hypothetical answer will share vocabulary with the real document. Multi-query approaches generate several paraphrases of the query and retrieve against all of them.7
Agentic RAG. Instead of a single retrieve-then-generate step, an agent orchestrates multiple retrieval and reasoning steps. The model reads retrieved documents, identifies gaps, formulates follow-up queries, retrieves again, and iterates until it has enough information to answer. Frameworks like LlamaIndex and LangChain have built abstractions for this, though the complexity cost is real — more steps mean more latency, more token cost, and more opportunities for error propagation.
Graph RAG. Microsoft’s GraphRAG uses an LLM to extract entities and relationships from documents, builds a knowledge graph, and retrieves subgraphs rather than text chunks. This helps with questions that require synthesizing information across multiple documents — “What are the connections between company X’s partnerships and their patent portfolio?” — where standard chunk-level retrieval falls short.
Context compression. With long context windows (Gemini supports 1M+ tokens, Claude supports 200K), you can fit more retrieved documents into the prompt. But more isn’t always better — Liu et al. (2023) showed that models struggle with information in the middle of long contexts (the “lost in the middle” effect). Context compression techniques summarize or filter retrieved documents to keep only the most relevant passages.
RAG vs. Fine-Tuning vs. Long Context
The question I get most often: when should you use RAG versus fine-tuning versus just shoving everything into a long context window?
RAG is best when: your knowledge base changes frequently, you need citations and traceability, you need to handle more information than fits in any context window, or you can’t afford to retrain models. Most enterprise applications fall here.
Fine-tuning is best when: you need the model to learn a new behavior or style rather than new facts. Fine-tuning a model to speak in your company’s voice or to follow a specific output format is effective. Fine-tuning for factual knowledge is fragile — the model may still hallucinate, and updating the knowledge requires retraining.8
Long context is best when: the total relevant information fits in the context window and latency allows it. For tasks like “summarize this 50-page document” or “answer questions about this codebase,” just loading everything into context works surprisingly well and requires no infrastructure. The Gemma 4 family’s 128K native context (and some variants pushing to 256K) makes this viable for an increasingly wide range of tasks.
In practice, these approaches compose. RAG to retrieve the right documents + long context to include multiple retrieved chunks + fine-tuning to ensure the model follows your output format reliably. The best production systems I’ve seen use all three.
Evaluation: How Do You Know Your RAG System Works?
RAG evaluation is harder than it looks because there are multiple failure modes:
Retrieval failure: the right document exists but wasn’t retrieved.
Ranking failure: the right document was retrieved but ranked too low to make the context window cutoff.
Generation failure: the right document was in the context but the model ignored or misinterpreted it.
Integration failure: the model’s answer is correct per the documents but doesn’t actually answer the user’s question.
Each requires different metrics. Retrieval quality is measured by recall@k and MRR. Generation quality is measured by faithfulness (does the answer match the documents?) and relevance (does the answer address the question?). RAGAS and TruLens provide frameworks for automated RAG evaluation, using LLMs to judge retrieval and generation quality.
The most insidious failure mode is partial hallucination: the model uses some retrieved facts correctly and then fills in gaps with fabricated details. The answer looks grounded because it cites real information, but it includes claims that aren’t in the source documents. Detecting this requires comparing the generated answer against the retrieved documents at a claim level, which is computationally expensive and itself error-prone.
Where RAG Is Going
RAG is evolving in two directions simultaneously. First, toward greater sophistication: multi-hop retrieval, graph-based knowledge organization, adaptive chunking, learned retrieval functions. Second, toward greater simplicity: longer context windows are reducing the need for retrieval pipelines (why retrieve when you can load everything?), and better base models are reducing the need for external grounding.
I suspect both trends will continue, and RAG won’t disappear — it will become infrastructure. Just as databases didn’t disappear when RAM got cheaper, retrieval pipelines won’t disappear when context windows get longer. The information that matters will always exceed the context window, and the need for traceable, updatable, auditable knowledge sources will only grow as AI systems are deployed in higher-stakes settings.9
The original RAG paper described the approach as giving language models “access to external knowledge.” Five years later, that framing still captures the core value proposition: language models are powerful reasoners but unreliable knowledge stores. RAG separates the reasoning from the storage, plays to the model’s strengths while compensating for its weaknesses, and — most importantly — gives you something you can debug when it goes wrong.