Retrieval-Augmented Generation (RAG) extends an LLM's effective knowledge by dynamically fetching relevant information from an external corpus at inference time. Where a base LLM is limited to what it learned during training, a RAG-enhanced system can draw on any indexed document — current documentation, proprietary knowledge, recent events — without retraining.
What RAG is and why it matters
RAG solves a fundamental limitation of static LLMs: they can't know what they weren't trained on. For anything requiring current, proprietary, or project-specific knowledge — your codebase, your internal documentation, your customers' data — RAG is the primary solution.
The pattern was popularized by a 2020 Meta AI paper and has become the dominant architecture for LLM applications that need to work with real-world knowledge bases. Rather than prompting the model to recall facts it may have learned imperfectly, RAG gives it the facts directly at query time — significantly reducing hallucination rates on knowledge-intensive tasks.
RAG architecture overview
A basic RAG pipeline has three stages. Indexing: documents are chunked into segments, passed through an embedding model to produce vector representations, and stored in a vector database alongside their source text. Retrieval: when a query arrives, it's embedded with the same model, and the N most similar document chunks are fetched from the vector database. Augmented generation: the retrieved chunks are inserted into the LLM's context alongside the query, and the model generates a response grounded in the retrieved content.
Extensions to this basic pattern include: hybrid search (combining vector similarity with keyword search for better recall), reranking (using a second model to score retrieved chunks for relevance before including them), multi-hop retrieval (iterating retrieval to answer complex multi-part questions), and agentic RAG (letting the model decide when and what to retrieve).
Chunking and indexing strategies
How you split documents for embedding significantly affects retrieval quality. Common strategies:
- Fixed-size chunking: split on token or character count with overlap. Simple, works reasonably well, but can cut semantic units in half.
- Sentence or paragraph chunking: split on natural language boundaries. Better semantic coherence; works well for prose documentation.
- Structural chunking: split on document structure (headings, sections, function definitions). Best for structured technical content — code files split at function boundaries, docs split at H2 sections.
- Hierarchical chunking: index at multiple granularities (document, section, paragraph) and retrieve at the level appropriate for the query. Most powerful, most complex.
For project planning documents and technical documentation, structural chunking at section granularity usually produces the best retrieval results. For code, function-level chunking with file-level context attached is a common high-performing pattern.
Embedding and retrieval quality
Retrieval quality is the most important variable in a RAG system. Poor retrieval — returning chunks that don't contain the answer — is worse than no retrieval, because it confidently grounds the model in wrong information. Evaluating retrieval quality separately from generation quality is essential.
Build a ground-truth retrieval eval set: a list of representative queries, each with its expected source chunks. Measure recall@K (what fraction of expected chunks appear in the top K results) and precision@K (what fraction of retrieved chunks are relevant). Tune chunking size, embedding model, and similarity threshold until retrieval quality is reliable before worrying about the generation step.
Reranking can significantly improve precision at the cost of additional latency. After initial retrieval returns the top 20 candidates, a reranker (often a cross-encoder model) scores each candidate against the query and reorders them. The top 5 after reranking are much higher quality than the top 5 from initial retrieval.
RAG in AI coding workflows
RAG is increasingly built into AI coding tools. Cursor, Cline, and GitHub Copilot all maintain some form of codebase index that enables retrieval over your project. When you ask about a function, the tool fetches the relevant file; when you ask about a pattern, it retrieves examples from your codebase.
Planning documents can be added to this retrieval corpus. A project that indexes its PRD, architecture spec, and user stories alongside the codebase gives the coding agent a complete knowledge base — it can retrieve not just code context but requirement context when generating or debugging. The coordinate system in InitRepo's generated documents is specifically designed to support this pattern, mapping requirements to implementation locations across documents.
Getting started
For a first RAG system, the quickest path is: choose a managed vector database (Pinecone, Weaviate, or a Postgres extension like pgvector), use an off-the-shelf embedding model (text-embedding-3-small from OpenAI, or a capable open-source alternative), split your documents at section boundaries, and build a minimal retrieval function that returns the top 5 chunks for a query.
Validate retrieval quality on a small eval set before wiring up generation. Once retrieval is reliable, plug it into your LLM prompt as a context section. Iterate on chunking and retrieval rather than on the prompt — retrieval quality is almost always the binding constraint.
For enterprise-scale RAG, see the Enterprise RAG Implementation Guide.