Large language models (LLMs) write fluent answers, but they only know what was included in training and cannot reliably identify the source of every fact. Retrieval-augmented generation (RAG) addresses those limits by searching an external knowledge source before the model answers, then adding relevant passages to the prompt so the response can be grounded in that material.
RAG, defined: Retrieval-augmented generation is an AI architecture that combines an information retrieval system with a generative language model. At query time, it retrieves passages relevant to a question from an external knowledge source, adds them to the model’s input, and generates an answer grounded in that context.
This guide explains the RAG pipeline, chunking, embeddings, vector indexes, BM25 and hybrid retrieval, reranking, architectural variants, evaluation, failure modes, security, and the main tools used to build RAG systems. Research references are listed at the end.
Why Do Language Models Need Retrieval?
An LLM stores learned information in its weights, often called parametric memory. That memory is fixed at training time, difficult to inspect directly, and may not include private or newly updated information. Four limits matter in practice:
- Knowledge freshness: training data cannot keep pace with every new policy, product release, or event.
- Private knowledge: the model does not automatically know an organization’s policies, manuals, support tickets, or contracts.
- Hallucination: when information is missing, a model can still produce a confident but incorrect answer.
- Provenance: without source context, it can be difficult to verify where a statement came from.
RAG adds a second kind of memory: an external index that can be updated, restricted, and audited without retraining the model. Because the system knows which passages it retrieved, it can also cite them. RAG does not eliminate hallucinations; a model can misread, ignore, or go beyond the retrieved text.
Where Did RAG Come From?
The term was introduced in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and colleagues. Their system paired a sequence-to-sequence generator with a neural retriever over a dense index of Wikipedia. The paper described RAG-Sequence, which uses the same retrieved passages across an answer, and RAG-Token, which can condition individual generated tokens on different passages.
The retriever built on Dense Passage Retrieval (DPR), published by Vladimir Karpukhin and colleagues in 2020. DPR uses separate encoders for questions and passages to map them into vectors. Modern RAG systems usually use separate, off-the-shelf components: a parser, embedding model, search index, and general-purpose LLM connected by a pipeline.
How Does a RAG System Work?
A RAG system has two related workflows. The indexing pipeline runs when documents are added or changed. The query pipeline runs whenever a user asks a question.
The indexing pipeline
- Load: collect source documents from files, websites, databases, wikis, or APIs.
- Parse and clean: extract text and preserve useful structure such as headings and tables.
- Chunk: split content into passages that can be retrieved independently.
- Embed: convert each chunk into a numeric vector representing its semantic features.
- Index: store the vectors, original text, and metadata for later search.
The query pipeline
- Optionally rewrite or decompose the user’s question.
- Retrieve candidate passages with keyword search, vector search, or both.
- Rerank candidates with a more precise relevance model.
- Build a prompt containing instructions, the question, and selected context.
- Generate an answer, ideally with citations linked to retrieved passages.
Every stage affects answer quality. A strong LLM cannot use a passage that retrieval missed, and accurate retrieval can still fail if the prompt omits or buries the key evidence.
Stage 1: How Are Documents Ingested and Parsed?
RAG can use unstructured documents such as PDFs, Word files, slides, and web pages; semi-structured content such as support tickets, email, and chat logs; structured data such as spreadsheets and database rows; and live sources queried at answer time.
Parsing is often underestimated. PDFs may have columns, repeated headers, scanned pages, and tables whose structure is lost during extraction. OCR is needed for scanned images. Bad parsing creates chunks that mix unrelated ideas, making even a strong embedding model less useful.
Capture metadata such as source, title, heading, author, date, version, language, and access permissions. Metadata supports filtering, freshness rules, citations, and access control. Deduplication also matters because near-identical copies can crowd the top search results.
Stage 2: How Does Chunking Work?
Chunking splits a document into passages that become the unit of retrieval. Smaller chunks can represent a single idea precisely but may lose context. Larger chunks preserve surrounding explanation but can blend several topics and consume more of the prompt’s token budget.
| Strategy | How it works | Strength | Trade-off |
|---|---|---|---|
| Fixed-size with overlap | Splits every set number of tokens and repeats some tokens between chunks. | Simple and predictable. | Can split sentences or ideas. |
| Recursive or structural | Splits by headings, then paragraphs and sentences. | Respects document structure. | Chunk sizes vary. |
| Semantic | Starts a new chunk when meaning changes. | Keeps related sentences together. | Needs more computation and tuning. |
| Sentence-window | Indexes a sentence but returns neighboring sentences. | Precise matching with context. | Creates more index entries. |
| Parent-child | Searches small child chunks but returns a larger parent section. | Combines precise retrieval and richer context. | Requires more complex storage and logic. |
Overlap can reduce the chance that an answer is split at a chunk boundary. A few hundred tokens with modest overlap can be a starting point, but there is no universally best size. Legal contracts, code, chat transcripts, and FAQs behave differently, so tune chunking against an evaluation set. Adding the document title and section heading to each chunk can also improve retrieval.
Stage 3: What Are Embeddings and Vector Indexes?
An embedding model converts text into a vector, a list of numbers intended to capture meaning. Texts with similar meanings tend to land near each other, even if they use different words. For example, “How do I reset my password?” and “I forgot my login credentials” can be close in embedding space.
Similarity is commonly measured with cosine similarity, which compares the angle between vectors. Other systems use dot product or Euclidean distance. Use a metric that matches the embedding model’s training and recommended configuration.
How does vector search scale?
Exact search compares a query against every stored vector, which can become slow for large collections. Approximate nearest-neighbor (ANN) indexes trade a small amount of recall for faster search. Common families include:
- HNSW (Hierarchical Navigable Small World): builds layers of proximity graphs. Sparse upper layers enable long jumps, while dense lower layers refine the search.
- IVF (inverted file index): groups vectors into clusters and searches only selected clusters.
- Product quantization (PQ): compresses vectors into short codes to reduce memory use, usually with some accuracy trade-off.
ANN indexes expose parameters that balance recall, latency, and memory. Metadata filtering restricts results by properties such as document type, date, or user permissions. Applying permission filters during retrieval helps prevent unauthorized chunks from entering the prompt.
Stage 4: How Does Retrieval Work? Sparse, Dense, and Hybrid Search
Sparse retrieval with BM25
Sparse or lexical retrieval matches words in the query with words in each passage. BM25 scores documents using term frequency, inverse document frequency, and length normalization. It is strong for exact identifiers such as product codes, error messages, names, acronyms, and legal citations. It can miss a relevant passage when the query uses different words from the source.
Dense retrieval with embeddings
Dense retrieval embeds the query and returns nearby vectors. It can handle paraphrases, synonyms, and questions phrased differently from the source. It may miss exact identifiers with little semantic meaning and depends on how well the embedding model fits the domain.
Hybrid retrieval and Reciprocal Rank Fusion
Hybrid retrieval runs keyword and vector search together because the methods have complementary strengths. Their raw scores are not directly comparable, so systems often merge ranked lists rather than scores. Reciprocal Rank Fusion (RRF) assigns each result a contribution of 1 divided by a constant plus its rank across each list.
For example, with the commonly used constant k = 60, a passage ranked second in both lists scores 1/62 + 1/62 = 0.03226. A passage ranked first in one list and fourth in the other scores 1/61 + 1/64 = 0.03202. The second passage wins narrowly because both retrievers agree it is relevant.
Stage 5: How Are Queries Transformed?
Users rarely phrase questions exactly as source documents do. Query transformation can rewrite a vague question into a standalone search query, generate multiple query variants, split a complex question into sub-questions, or route it to the right index. Hypothetical Document Embeddings (HyDE) asks an LLM to draft a hypothetical answer and uses its embedding for search; the hypothetical text may contain errors, so retrieved source passages must still ground the final answer.
Transformations can improve recall, but each adds latency and cost. Apply them selectively, such as when initial retrieval returns weak results or a question needs several pieces of evidence.
Stage 6: What Does Reranking Do?
First-stage retrieval is optimized for speed across many chunks. Reranking is optimized for precision over a smaller candidate set. A reranker scores the candidates and moves the most relevant passages to the top.
- Bi-encoders: encode the query and each document separately. Their vectors can be precomputed, making retrieval fast.
- Cross-encoders: process the query and passage together to produce a relevance score. They are often more accurate but must run for each candidate.
- Late interaction: models such as ColBERT retain fine-grained representations and compare them at query time, balancing detail with speed.
A common design retrieves dozens of candidates with hybrid search, reranks them, and sends only the strongest few passages to the LLM. The correct number depends on latency, accuracy, and context limits.
Stage 7: How Is the Prompt Augmented?
Augmentation is more than pasting text into a prompt. A useful prompt includes instructions to answer from the supplied context, labeled passages with source identifiers, the user’s question, and output requirements such as format and citation style. Tell the model to say when the context does not contain an answer rather than guess.
Context order matters. Research on long-context models has found that relevant information can be used less reliably when it sits in the middle of a long prompt. Sending fewer, better passages and placing key evidence near the beginning or end can help. Context compression can reduce irrelevant text when token budgets are tight.
Stage 8: How Does Grounded Generation Work?
Grounding means the answer’s claims are supported by the provided sources. Systems encourage it with inline citations, abstention rules, and post-generation checks that compare claims with cited passages. Lower sampling temperature can reduce creative drift in factual tasks, but does not guarantee correctness.
A model can still combine sources incorrectly, misinterpret a passage, or fill gaps from its own learned knowledge. Evaluation should therefore measure faithfulness separately from answer correctness.
What Are Naive, Advanced, and Modular RAG?
- Naive RAG: a basic index, retrieve, and generate loop. It is simple, but sensitive to poor chunking and irrelevant context.
- Advanced RAG: adds improvements such as metadata, query rewriting, hybrid search, reranking, and context compression.
- Modular RAG: treats retrieval, memory, routing, and fusion as interchangeable modules that can be rearranged or repeated.
These categories describe increasing control over the pipeline, not separate products. Most production systems use some combination of advanced and modular techniques.
What Are the Main Types of RAG?
| Variant | Core idea | Best suited to |
|---|---|---|
| Standard RAG | Retrieve once, then generate. | Direct factual questions. |
| Self-RAG | The model decides when to retrieve and critiques passages and its output. | Questions where retrieval is not always needed. |
| Corrective RAG (CRAG) | Scores retrieval quality and triggers filtering or fallback search. | Incomplete or uneven knowledge bases. |
| GraphRAG | Builds a knowledge graph and community summaries. | Questions about themes across a corpus. |
| Agentic RAG | An LLM agent plans multi-step retrieval and tool use. | Complex, multi-hop research. |
| Multimodal RAG | Retrieves images, tables, audio, or video as well as text. | Manuals, diagrams, and media archives. |
Self-RAG can retrieve on demand and critique its own output. Corrective RAG evaluates whether retrieved documents fit the query and may refine them or add web search. GraphRAG uses entity relationships and summaries to answer broad questions that individual chunks may not cover. Agentic RAG gives an LLM control over multiple searches and tool calls, which can help with multi-hop questions but raises cost and latency. Multimodal RAG extends retrieval beyond text.
How Do You Evaluate a RAG System?
Because RAG combines retrieval and generation, evaluate each stage separately and then evaluate the complete answer.
Retrieval metrics
- Recall@k: the share of relevant passages appearing in the top k results.
- Precision@k: the share of the top k results that are relevant.
- Mean Reciprocal Rank (MRR): the average reciprocal rank of the first relevant result.
- nDCG: a ranking metric that gives more credit to highly relevant passages near the top.
Generation metrics
- Faithfulness: are answer claims supported by the retrieved context?
- Answer relevance: does the answer address the question?
- Context relevance: is the retrieved text focused on the question?
- Correctness: does the answer match a trusted reference answer, where one exists?
Frameworks such as Ragas automate some checks, often using LLMs as judges. Because automated judges can also be wrong, calibrate them against human ratings. A robust process uses a fixed test set, automated metrics for each stage, and regular review of real questions and answers.
Why Do RAG Systems Fail?
RAG failures can happen before the LLM generates a word or during generation itself. Seven common failure points are shown below.
| Failure point | What happens | Typical response |
|---|---|---|
| Missing content | The knowledge base lacks the answer, but the model guesses. | Abstention rules and coverage checks. |
| Missed top-ranked documents | The answer exists but ranks too low. | Hybrid search, reranking, and query rewriting. |
| Not in context | Retrieved information is dropped before prompting. | Better context selection and ordering. |
| Not extracted | The model misses information in noisy context. | Fewer, cleaner chunks and compression. |
| Wrong format | The model ignores output instructions. | Structured output constraints and validation. |
| Incorrect specificity | The answer is too broad or too narrow. | Clarifying questions and query rewriting. |
| Incomplete | The answer leaves out available details. | Decomposition and multi-step generation. |
Production systems also face stale indexes, conflicting source versions, parsing errors, and duplicate content. Monitor the complete pipeline, not just the final answer.
What Are the Security and Privacy Risks of RAG?
RAG connects a model to data and introduces new attack surfaces. Risks include unauthorized retrieval, cross-tenant data leakage, embedding inversion, data poisoning, and indirect prompt injection from malicious instructions hidden in retrieved content.
- Permission-aware retrieval: filter results using the current user’s source-system permissions.
- Tenant and sensitivity separation: partition indexes or apply strict filters for different teams and data classes.
- Trusted ingestion: validate sources and monitor changes to indexed content.
- Untrusted context: instruct the model to treat retrieved text as data, not as commands to follow.
- Auditing: log queries, retrieved chunks, and answers in line with privacy and retention requirements.
Embeddings should receive protection appropriate to the source data. They are not automatically anonymous or harmless simply because they are vectors.
RAG vs. Fine-Tuning vs. Long-Context Prompting
| Dimension | RAG | Fine-tuning | Long-context prompting |
|---|---|---|---|
| How knowledge is added | Retrieved at query time. | Learned into model weights. | Documents are placed in the prompt. |
| Updating knowledge | Re-index changed documents. | Retrain or fine-tune again. | Change the supplied documents. |
| Citations | Natural fit for retrieved passages. | Not inherent. | Possible but harder to trace. |
| Best for | Large, changing knowledge bases. | Style, format, and behavior. | Small document sets or one-off analysis. |
| Main trade-off | Retrieval infrastructure and misses. | Training data and compute. | Token cost, latency, and context-position effects. |
These approaches can complement one another. Fine-tuning changes how a model behaves; RAG supplies knowledge that needs to stay current or cite sources; long context can be useful when a small set of documents needs to be read together.
What Are the Best RAG Tools?
No single tool builds a complete RAG system. A working system combines components from several layers. The tools below are grouped by function, not ranked, and their features, licenses, and pricing should be checked against current official documentation before implementation.
RAG frameworks
| Framework | What it focuses on | When to consider it |
|---|---|---|
| LangChain | LLM application integrations, retrievers, and vector stores; LangGraph supports agent workflows. | Teams building broad LLM and agent applications. |
| LlamaIndex | Data connectors, indexing, and query engines for retrieval over private data. | Applications centered on document ingestion and querying. |
| Haystack | Modular pipelines and components for production RAG. | Teams that want explicit pipeline composition. |
| DSPy | LLM pipelines as code and optimization against metrics. | Teams evaluating and tuning prompt programs systematically. |
| Microsoft GraphRAG | Graph-based RAG implementation. | Questions requiring relationships and whole-corpus summaries. |
Vector databases and search engines
- Milvus: distributed vector database with multiple ANN indexes and hybrid search options.
- Qdrant: vector search with payload filtering, dense and sparse search, and quantization.
- Weaviate: vector database with hybrid search and vectorizer integrations.
- Chroma: vector store suited to local development, prototyping, and hosted use.
- pgvector: PostgreSQL extension for vector search using HNSW or IVFFlat indexes.
- Pinecone: hosted vector database with dense, sparse, full-text, and hybrid query features.
- OpenSearch: search engine that combines keyword search with vector and neural queries.
- Faiss: similarity-search library for embedding into applications; it is not a full database service.
Teams already using PostgreSQL may start with pgvector to avoid a separate system. Large self-hosted deployments may evaluate Milvus or Qdrant, while teams that prefer managed infrastructure may consider hosted services. The right fit depends on scale, latency, filtering, operations, and cost.
Document parsing and ingestion
- Docling: converts document formats into AI-ready output with layout analysis, table recognition, and OCR.
- Unstructured: partitions PDFs, Word files, HTML, email, and images into structured elements.
- LlamaParse: a commercial document-parsing service from the LlamaIndex ecosystem.
Embedding and reranking
Sentence Transformers provides embedding models, cross-encoder rerankers, sparse encoders, and multi-vector models. Hosted embedding and reranking APIs are also available from model providers and cloud platforms. Choose using a representative set of your own documents and questions; changing the embedding model later generally means re-embedding the index.
Evaluation and observability
- Ragas: RAG evaluation metrics and synthetic test-set generation.
- DeepEval: test-style evaluation for faithfulness, contextual precision and recall, and answer relevance.
- TruLens: evaluation and tracking for LLM experiments, including context relevance and groundedness.
- Arize Phoenix: tracing, retrieval and response evaluations, datasets, and experiments.
Tracing can reveal why an answer failed, such as the correct chunk being retrieved but ranked too low to reach the prompt.
Managed RAG platforms
- Amazon Bedrock Knowledge Bases: managed ingestion, vector storage, and retrieval connected to supported data sources.
- Azure AI Search: hybrid queries, vectorization, semantic ranking, and agentic retrieval capabilities.
- Google RAG Engine: managed ingestion, chunking, embedding, indexing, retrieval, and generation capabilities in Google Cloud.
Managed platforms can shorten the path to a working system and reduce infrastructure work. Trade-offs include provider-specific APIs, less control over some stages, and usage-based costs.
How to choose RAG tools
- Decide whether you want managed infrastructure or a custom pipeline.
- Start with systems you already operate, such as PostgreSQL, OpenSearch, or a particular cloud platform.
- Estimate corpus size, query volume, latency targets, and access-control needs.
- Use a dedicated parser if your source files contain complex tables, scans, or layouts.
- Choose evaluation and tracing tools before launch so changes can be measured consistently.
- Verify current features, licenses, security terms, and pricing in official documentation.
Where Is RAG Used?
RAG is useful wherever an answer must be grounded in a specific body of changing or private information. Examples include customer-support assistants that search help articles, internal knowledge assistants for policies and wikis, IT assistants for runbooks, legal research over contracts, healthcare literature tools with expert review, and search experiences that retrieve pages and cite sources.
RAG Glossary
- Chunk: a passage used as the unit of indexing and retrieval.
- Embedding: a vector representation intended to capture meaning.
- Vector index: a store optimized for similarity search over embeddings.
- ANN: approximate nearest-neighbor search that finds close vectors without comparing every vector.
- BM25: a keyword relevance scoring function based on term frequency, rarity, and document length.
- Hybrid search: a combination of keyword and vector retrieval.
- RRF: a rank-fusion method that merges ranked lists.
- Reranker: a model that reorders retrieved candidates by relevance.
- Grounding: supporting claims with supplied evidence.
- Faithfulness: the degree to which an answer is supported by its retrieved context.
- Context window: the amount of text a model can process in one request.
- RAG framework: a library that connects loaders, chunkers, embeddings, indexes, retrievers, and LLMs.
Frequently Asked Questions
What is RAG in AI?
RAG, or retrieval-augmented generation, retrieves relevant information from an external knowledge source and adds it to a language model’s prompt so the answer can use that information instead of relying on training data alone.
What is the difference between an LLM and RAG?
An LLM generates text from patterns learned during training. RAG is an architecture around a model that adds a retrieval step so it can use external, updatable information and cite retrieved sources.
Is ChatGPT a RAG system?
ChatGPT is an application built on large language models. When it uses search or uploaded files, it follows a retrieval-augmented pattern by gathering relevant information and using it to generate a response. That description does not imply that every response uses retrieval.
Does RAG eliminate hallucinations?
No. RAG can reduce unsupported answers by supplying evidence, but a model can still misread sources, combine them incorrectly, or add claims that the sources do not support.
What is a RAG pipeline?
A RAG pipeline prepares documents by parsing, chunking, embedding, and indexing them, then retrieves and reranks relevant passages for each question before generating a response.
What is the difference between dense retrieval and BM25?
BM25 matches query words and works well for exact terms such as codes and names. Dense retrieval compares embeddings to match meaning and paraphrases. Hybrid systems use both.
What is agentic RAG?
Agentic RAG lets an LLM agent plan multiple retrieval steps, inspect results, run follow-up searches, and use tools until it has enough evidence to answer.
What is GraphRAG?
GraphRAG builds a knowledge graph of entities and relationships and summarizes related communities to answer broader questions about a dataset.
How does RAG handle long documents?
It splits documents into chunks, retrieves the most relevant passages, and adds them to the prompt. Parent-child chunking, sentence windows, and context compression can balance precision and context.
What data sources can RAG use?
RAG can use documents, web pages, wikis, tickets, email, chat logs, code, database records, and live APIs. Each source needs suitable parsing, metadata, and access controls.
What is the best vector database for RAG?
It depends on infrastructure and scale. pgvector can suit PostgreSQL users, Chroma can suit prototypes, Milvus and Qdrant can suit self-managed deployments, and hosted services can suit teams that want less operational work.
When should you use RAG instead of fine-tuning?
Use RAG when knowledge changes often, needs citations, or must respect access permissions. Fine-tuning is more suited to changing model behavior, style, or output format. The approaches can be combined.
Summary
Retrieval-augmented generation gives a language model access to knowledge outside its weights at the moment it answers. An indexing pipeline parses, chunks, embeds, and stores documents. A query pipeline transforms the question, retrieves candidates using keyword and vector search, reranks them, and builds a prompt for grounded generation.
Failures can come from parsing, chunking, retrieval, context selection, or generation. A reliable RAG system therefore needs evaluation across the full pipeline, security controls that respect source permissions, and monitoring after launch. Choose tools by layer and by the requirements of your own data and workload.
