Context compression reduces the token count of information before it enters an LLM's context window, allowing more content to fit within budget constraints and reducing cost and latency without proportionally degrading quality. A well-compressed context often outperforms a raw, uncompressed one — because compression forces prioritization of signal over noise.
Why compression matters
Even as context windows expand, compression remains valuable. Every token in the context window costs money to process and takes time to attend over. A 200K token window filled with low-relevance content produces slower, more expensive responses than a 20K token window containing only the relevant material. Compression is not a workaround for small windows; it's a quality discipline that pays dividends at any window size.
Compression also counteracts attention dilution. LLMs don't attend equally to all content — they attend more to the beginning and end of context, and more to content that appears frequently or stands out structurally. Compressing irrelevant content removes the distraction, giving the model cleaner signal on what matters.
Summarization approaches
Summarization is the most general compression technique: use an LLM (or the same model in a pre-processing step) to produce a condensed version of a document before including it in the main prompt. Effective for prose documentation, long user stories, and conversation history.
Extractive summarization selects and concatenates the most important sentences from the original — fast and preserves exact wording, but can be incoherent across extracted sentences. Abstractive summarization generates a new text that captures the meaning — more coherent, preserves nuance better, but introduces potential generation errors. For factual content like specifications, extractive is safer; for prose like meeting notes, abstractive often produces cleaner results.
Rolling summarization handles growing conversation histories: every N turns, compress the oldest N turns into a summary paragraph that carries forward the key decisions and context. The summary replaces the raw turns, freeing space for new content while preserving the thread.
Selective retention
Rather than summarizing everything, selective retention includes only the portions of a document that are relevant to the current task. For a 50-page architecture spec, you might extract only the sections relevant to the database layer for a database-related coding task, discarding the API and frontend sections entirely.
Selective retention requires a relevance model — either keyword-based filtering, embedding similarity against the task description, or an LLM call that identifies which sections to include. The overhead of the selection step is usually well worth the savings in the main context, especially for tasks repeated many times.
Hierarchical chunking
Hierarchical chunking indexes content at multiple granularities and includes only the level needed for each task. A document might be indexed as: full document, per-section summaries, and per-paragraph text. A query that needs high-level context gets the document summary; one that needs specific details gets the relevant paragraphs.
For coding tasks, a practical hierarchy: include the architecture overview for system-level reasoning, the module design for area-level reasoning, and the specific function signatures and implementations for task-level reasoning. Don't include the full codebase when the architecture overview is sufficient; don't use the architecture overview when function signatures are needed.
Prompt-level compression
Prompt compression techniques reduce token count at the instruction level. Key approaches: remove redundant qualifications and hedges from instructions; replace verbose descriptions with structured lists; use abbreviations and shorthand that the model understands from context; and eliminate examples when the model can infer the pattern from a single one.
LLMLingua and similar prompt compression research shows that many prompts can be compressed by 2–4× with minimal quality loss by removing tokens the model would have predicted anyway from context. The gains are most significant for very long prompts; for short prompts, the compression overhead may not be worth it.
Choosing a strategy
Choose your compression approach based on your content type and constraints:
- Short prose documents (<5K tokens): no compression necessary; include verbatim.
- Long prose documents (5K–50K tokens): extractive or abstractive summarization, or selective retention based on task relevance.
- Structured technical content (API docs, specs): schema-based compression — extract signatures and definitions, discard explanatory prose.
- Growing conversation history: rolling summarization every 10–20 turns.
- Large codebases: retrieval over function-level chunks rather than full-file inclusion.
The common thread: compress deliberately, not as a last resort when you run out of tokens. Build compression into your context pipeline from the start and tune it with a representative eval set.
See LLM Context Window Optimization for broader context management strategies, or the RAG guide for retrieval-based approaches to context scaling.