Pattern
How AI agents remember
AI agents remember in layers: short-term context is what the model can see in the current conversation, and long-term memory is stored outside the model and fetched back in when relevant. Vector stores find memories by meaning, graph stores by how people, records and events connect.
Explained from public sources. Mai has not built this yet.
On this page
In plain English
A language model has no memory of its own. Each time it is called, it only knows what is in front of it: the instructions, the conversation so far and whatever data the system chose to include. Everything an agent “remembers” beyond that is stored somewhere else and looked up when needed, much like a good assistant keeping a notebook and checking it before a meeting. The craft is deciding what goes in the notebook, how to find it again, and when to throw things away.
How it works
It helps to think of three layers.
Short-term context. This is the context window: everything the model reads on this call. It is fast and precise, but limited in size and wiped after each call. A long conversation fills it up, so the system has to choose what stays. Common tactics are keeping the most recent turns word for word and replacing older ones with a running summary.
Working state. For a task that takes several steps, the agent keeps structured notes outside the prompt: the goal, what has been done, what is waiting. This is ordinary application state in a database, and it is what lets a task survive a restart.
Long-term memory. Facts, preferences, past decisions and documents that should outlast a single conversation. Three stores do most of the work:
- Records. Plain database rows for things with a known shape: a customer’s preferred contact time, an approved supplier list. Exact, cheap and easy to audit.
- Vector store. Text is split into passages and each one is turned into an embedding, a list of numbers that represents its meaning. A question is embedded the same way, and the store returns the passages that sit closest to it. This finds “the email about the boiler” when you asked about “heating repairs”.
- Graph store. Entities (people, properties, contracts) are nodes and their relationships are edges. Graphs answer questions about connections and can summarise across a whole collection. The GraphRAG paper shows the idea: build an entity graph from documents, summarise clusters of related entities, and use those summaries for broad questions where plain vector search struggles.
Now: Request. Then Context window. Then Recall: Records, Vector store, Graph. Then Write back.
Two research ideas shaped how people build this. MemGPT treats the context window like a computer’s main memory and long-term storage like its disk, with the agent itself moving information between the two tiers. The Generative Agents paper keeps a full log of experiences, retrieves the relevant ones when deciding what to do, and periodically writes higher-level reflections that summarise many small memories into one useful conclusion.
Writing is as important as reading. After a conversation, a memory step decides what is worth keeping, rewrites it as a short, self-contained fact with a date and a source, and checks it against what is already stored so it updates rather than duplicates.
Where it goes wrong
- Wrong recall. Vector search returns what is similar, not what is true or current. An old price list can outrank the new one. Memories need dates, sources and a way to mark them as superseded.
- Stuffed context. Pulling in too much makes answers worse and costs more. Recall should be selective, with a budget for how much memory enters each prompt.
- Remembering the wrong things. An agent that saves everything stores gossip, mistakes and one-off remarks as permanent facts. Rules for what is worth keeping matter more than the database.
- Poisoned memory. Text from an email or web page can carry instructions. If it is saved as a memory, it comes back later looking trustworthy. Content from outside should be stored as data, labelled with its origin, and never treated as instructions (see agent safety).
- Personal data. Long-term memory about people is personal data under UK GDPR. It needs a purpose, a retention period and a way to find and delete it on request.
- Leaking across boundaries. A memory written while helping finance should not surface in a conversation with a contractor. Memory needs the same access rules as the systems it came from.
How Mai AI uses it
In a Mai AI system, I design memory around the client’s existing records first. The business already has a system of record for customers, properties or orders, so agents read and write there instead of keeping a private copy that drifts. In the AI chief of staff, the business’s records live in Dataverse and email, calendars and files come through Microsoft Graph, so the agents look facts up at the source.
On top of that I add the layers that a system of record does not cover: a running summary per conversation, working state for multi-step tasks, a vector index over documents and correspondence for search by meaning, and a graph only when the questions really are about relationships. Every stored memory carries where it came from and when, and each specialist agent sees only the memory its area is allowed to see, which fits naturally with routing and specialist agents.
Memory is also where a system gets better over time, through the procedures described in self-improving skills. If you want to see how memory would fit your own data, it is one of the layers in a custom AI operations system.
Questions and answers
How do AI agents remember past conversations?
The model itself forgets everything between calls. The system around it saves notes, facts and summaries to a database, then searches that store and puts the relevant pieces back into the prompt the next time they are needed.
What is the difference between short-term and long-term memory in AI agents?
Short-term memory is the context window: the instructions, conversation and data the model can see right now. Long-term memory is stored outside the model, survives between sessions, and has to be retrieved before the model can use it.
What is a vector database used for in AI agents?
A vector database stores text as embeddings, lists of numbers that capture meaning, so the agent can find memories that are similar in meaning to the current question even when the words differ.
When should an AI agent use a knowledge graph instead of a vector store?
Use a graph when the questions are about relationships, such as which tenants are linked to which properties and contractors. Vector search finds similar passages; a graph follows connections and can answer questions that span many records.
Harnesses known for this
- Harness profileHermes Agent
Hermes Agent is an open-source AI agent from Nous Research that runs on a server or your own machine, talks to you through the terminal or chat apps, and writes reusable skills from the work it does. It suits people who want a long-lived agent that remembers, runs scheduled jobs and improves with use, on any model provider.
- Harness profileClaude Code and Agent SDK
Claude Code is Anthropic's agentic coding tool that reads a codebase, edits files and runs commands from the terminal, IDE, desktop app or browser. The Claude Agent SDK packages the same agent loop and tools as a Python and TypeScript library, for building your own agents on Claude.
- Harness profileLangGraph
LangGraph is an open-source framework from LangChain for building stateful agents as graphs of steps, with saved checkpoints so a run can pause, wait for a person and resume. It is best for long-running, multi-step work where you want explicit control over every path.
- Harness profileCrewAI
CrewAI is an open-source Python framework for building teams of agents, each with a role, a goal and tools, that work through tasks together. It is best for work that splits naturally into roles, wrapped in event-driven Flows that keep the overall process under control.
Related entries
- PatternSelf-improving skills
Self-improving agents get better by writing down what worked as reusable skills: short procedures, checklists or scripts they load next time a similar task comes up. The model does not change; the library of procedures around it does, ideally with a person reviewing each change.
- PatternRouting and specialist agents
Routing means one agent reads each request and hands it to the specialist best placed to handle it, such as finance, operations or sales. Each specialist has its own instructions, tools and permissions, which keeps answers sharper and access tighter than one agent trying to do everything.
- PatternAgent safety
You make an AI agent safe by limiting what it can do, not by trusting what it says. Give it the narrowest tools and permissions, run risky work in a sandbox, require a person to approve consequential actions, and treat every email, document and web page it reads as possibly hostile.
- ExperimentAura
Aura is an open-source AI companion with a state of mind that carries from one moment to the next: emotions that obey physics, memory that remembers how things felt, and five language models thinking in layers.