AI context management is the practice of controlling what information is present when an LLM generates a response. Every decision about what to include, exclude, compress, or retrieve from a model's context window affects the accuracy, relevance, and cost of its outputs. Getting this right is one of the most practical skills in applied AI development.
The context problem
LLMs are stateless. Each call receives a fresh context window — the model doesn't remember prior conversations unless you explicitly include them. In simple chat applications this is easy to manage: just include the conversation history. But for applications that work with large codebases, long-running projects, or complex domain knowledge, managing what goes into that window becomes a genuine engineering problem.
The challenge has two faces. Too little context: the model lacks the information it needs and produces generic, off-target, or hallucinated output. Too much context: the model's attention is diluted across irrelevant information, latency and cost increase, and output quality can paradoxically decline. Effective context management navigates between these failure modes.
Types of context
Context falls into four categories, each with different characteristics and management strategies:
- Instruction context — system prompts, task descriptions, output format specifications. Usually small, always present, rarely changes within a session.
- Conversational context — the history of a dialogue. Grows over time; usually included in full for short conversations, compressed or summarized for long ones.
- Reference context — documentation, specifications, code files, planning documents. The largest category; selective inclusion is essential.
- Retrieved context — content fetched dynamically based on the current query. Enables much larger knowledge bases than any context window could hold.
Most context management complexity lives in the reference and retrieved categories, because that's where project-specific knowledge lives.
Context window limits in practice
Modern LLMs support context windows ranging from 32K to over 1M tokens. Despite this, context window size alone doesn't solve the management problem. Filling a large window with low-relevance content still dilutes model attention, and cost scales with tokens processed. A 200K token context isn't an invitation to throw everything in; it's permission to be more selective rather than less.
The "lost in the middle" problem is also real: research consistently shows that LLMs attend better to content at the beginning and end of their context than to content in the middle. For long contexts, this means the order of your context sections matters — place the highest-signal content first or last, not buried in the middle.
Memory and persistence patterns
Across a long project, you need ways to persist knowledge beyond a single context window. The main patterns are: document-based memory (write everything down; include the relevant documents in each new session); summarization (at the end of a session, generate a summary that captures the key decisions and artifacts; include the summary in the next session); and external storage (write facts to a database and retrieve them on demand).
For most development workflows, document-based memory is the most practical. Planning documents, architecture decision records, and user stories are already the right artifacts — they just need to be kept current. A project that starts with good planning documents has its memory system built in.
Retrieval-augmented approaches
When your knowledge base is too large to fit in context, retrieval-augmented generation (RAG) dynamically fetches the most relevant sections for each query. An embedding model converts your documents and the incoming query to vector representations; the system retrieves the documents with the highest similarity to the query.
The quality of a RAG system depends on chunking strategy (how documents are split for embedding), embedding model choice, retrieval algorithm, and post-retrieval reranking. For project documentation, retrieval at section granularity tends to produce better results than either full-document or sentence-level chunking.
Choosing the right strategy
For small projects with a handful of planning documents: static inclusion in context. Include the relevant documents at the start of each session; the overhead is minimal and the results are reliable.
For medium projects with substantial documentation: selective inclusion guided by the task at hand. Develop a convention for which documents to include for which types of tasks.
For large projects or products with user-generated content: RAG is essential. Build an index, implement retrieval, and evaluate retrieval quality before relying on it for production use.
See AI Context Management Best Practices for a detailed implementation guide, or the RAG guide for a deep dive on retrieval-based approaches.