LLMs are stateless: each call begins with no knowledge of previous calls. But the applications built on them often need persistent memory — user preferences that carry across sessions, facts established in earlier conversations, progress on long-running tasks. Memory management is the discipline of giving LLM applications the continuity that the base model lacks.
Types of LLM memory
Memory in LLM systems falls into four categories, borrowed from cognitive science:
- In-context memory — the content currently in the context window. Immediately available, but ephemeral (lost at the end of the session) and limited in size.
- Episodic memory — records of specific past events or interactions. "What did the user say in the last session?" Stored externally and retrieved selectively.
- Semantic memory — general facts and knowledge about the world or domain. "What is the database schema for this project?" Stored externally in a knowledge base or document store.
- Procedural memory — how to perform tasks. Usually encoded in the system prompt as instructions or few-shot examples.
Most context engineering work targets episodic and semantic memory — the persistent facts and histories that need to be available across sessions.
In-context memory management
The context window is the only memory the model has direct access to in a given call. Managing it well means deciding what to include, what to compress, and what to evict as the window fills. Rolling summarization handles growing conversation histories: summarize the oldest N turns into a compact summary and replace them. The summary fits in less space; the key facts carry forward.
For long-running tasks, a "working memory" document — a scratchpad the model updates at each step — provides durable within-session memory that can be explicitly managed. The agent writes decisions, observations, and open questions to the scratchpad; the scratchpad is always included at the start of the context, ensuring the most important state is always present even if earlier turns roll off.
External memory stores
External memory persists beyond any single session. The most common implementations: a vector database for semantic memory (embed facts and retrieve them by similarity); a relational database for structured episodic memory (user preferences, session summaries, task progress); and a document store for reference memory (planning documents, architecture specs, user stories).
The retrieval interface is what matters most. External memory is only useful if it can be queried at the right granularity for the task. A project architecture document is useful retrieved as a whole; a specific API signature is useful retrieved as a single fact. Design the storage structure with retrieval in mind, not just write time.
Episodic memory patterns
For applications that need to remember specific interactions — what a user preferred, what was discussed in the last session, what decisions were made yesterday — episodic memory is essential. Common patterns: end-of-session summarization (the model writes a structured summary of the session; the summary is stored and retrieved at the start of the next); fact extraction (the model identifies discrete facts from the conversation and stores them in a structured form); and event logging (every significant action is logged with metadata for later retrieval).
The end-of-session summary pattern is particularly useful for coding workflows. At the end of each session, the agent writes a handoff document: what it did, what decisions it made, what files it changed, and what it planned to do next. This becomes the starting context for the next session, providing full operational continuity without requiring the full conversation history.
Semantic memory for projects
Project planning documents are semantic memory. They encode the facts about the system — what it's supposed to do, how it's supposed to work, what constraints it must satisfy — in a form the agent can read. Maintaining them as a first-class engineering artifact makes the project's semantic memory explicit, auditable, and version-controlled.
The alternative — implicit semantic memory, where the agent's knowledge of the project lives only in conversation history or the developer's head — produces brittle systems. Knowledge that lives only in expired context windows is effectively lost.
Implementation guidance
For most projects, start with document-based semantic memory (planning documents in version control) and end-of-session summary for episodic memory. These require no external infrastructure and handle the majority of memory use cases in software development.
Add vector-based retrieval when the semantic memory grows large enough to require selection. Add structured episodic storage when you need to query past interactions by attribute (user, date, topic) rather than reading them sequentially. Build agentic memory management only when the task genuinely requires the model to decide what to remember.