Skip to content

Pattern

How AI agents remember

AI agents remember in layers: short-term context is what the model can see in the current conversation, and long-term memory is stored outside the model and fetched back in when relevant. Vector stores find memories by meaning, graph stores by how people, records and events connect.

StudiedLast reviewed 4 min read

Explained from public sources. Mai has not built this yet.

On this page

In plain English

A language model has no memory of its own. Each time it is called, it only knows what is in front of it: the instructions, the conversation so far and whatever data the system chose to include. Everything an agent “remembers” beyond that is stored somewhere else and looked up when needed, much like a good assistant keeping a notebook and checking it before a meeting. The craft is deciding what goes in the notebook, how to find it again, and when to throw things away.

How it works

It helps to think of three layers.

Short-term context. This is the context window: everything the model reads on this call. It is fast and precise, but limited in size and wiped after each call. A long conversation fills it up, so the system has to choose what stays. Common tactics are keeping the most recent turns word for word and replacing older ones with a running summary.

Working state. For a task that takes several steps, the agent keeps structured notes outside the prompt: the goal, what has been done, what is waiting. This is ordinary application state in a database, and it is what lets a task survive a restart.

Long-term memory. Facts, preferences, past decisions and documents that should outlast a single conversation. Three stores do most of the work:

  • Records. Plain database rows for things with a known shape: a customer’s preferred contact time, an approved supplier list. Exact, cheap and easy to audit.
  • Vector store. Text is split into passages and each one is turned into an embedding, a list of numbers that represents its meaning. A question is embedded the same way, and the store returns the passages that sit closest to it. This finds “the email about the boiler” when you asked about “heating repairs”.
  • Graph store. Entities (people, properties, contracts) are nodes and their relationships are edges. Graphs answer questions about connections and can summarise across a whole collection. The GraphRAG paper shows the idea: build an entity graph from documents, summarise clusters of related entities, and use those summaries for broad questions where plain vector search struggles.

Now: Request. Then Context window. Then Recall: Records, Vector store, Graph. Then Write back.

The model only sees its context; memory is written out and fetched back

Two research ideas shaped how people build this. MemGPT treats the context window like a computer’s main memory and long-term storage like its disk, with the agent itself moving information between the two tiers. The Generative Agents paper keeps a full log of experiences, retrieves the relevant ones when deciding what to do, and periodically writes higher-level reflections that summarise many small memories into one useful conclusion.

Writing is as important as reading. After a conversation, a memory step decides what is worth keeping, rewrites it as a short, self-contained fact with a date and a source, and checks it against what is already stored so it updates rather than duplicates.

Where it goes wrong

  • Wrong recall. Vector search returns what is similar, not what is true or current. An old price list can outrank the new one. Memories need dates, sources and a way to mark them as superseded.
  • Stuffed context. Pulling in too much makes answers worse and costs more. Recall should be selective, with a budget for how much memory enters each prompt.
  • Remembering the wrong things. An agent that saves everything stores gossip, mistakes and one-off remarks as permanent facts. Rules for what is worth keeping matter more than the database.
  • Poisoned memory. Text from an email or web page can carry instructions. If it is saved as a memory, it comes back later looking trustworthy. Content from outside should be stored as data, labelled with its origin, and never treated as instructions (see agent safety).
  • Personal data. Long-term memory about people is personal data under UK GDPR. It needs a purpose, a retention period and a way to find and delete it on request.
  • Leaking across boundaries. A memory written while helping finance should not surface in a conversation with a contractor. Memory needs the same access rules as the systems it came from.

How Mai AI uses it

In a Mai AI system, I design memory around the client’s existing records first. The business already has a system of record for customers, properties or orders, so agents read and write there instead of keeping a private copy that drifts. In the AI chief of staff, the business’s records live in Dataverse and email, calendars and files come through Microsoft Graph, so the agents look facts up at the source.

On top of that I add the layers that a system of record does not cover: a running summary per conversation, working state for multi-step tasks, a vector index over documents and correspondence for search by meaning, and a graph only when the questions really are about relationships. Every stored memory carries where it came from and when, and each specialist agent sees only the memory its area is allowed to see, which fits naturally with routing and specialist agents.

Memory is also where a system gets better over time, through the procedures described in self-improving skills. If you want to see how memory would fit your own data, it is one of the layers in a custom AI operations system.

Questions and answers

How do AI agents remember past conversations?

The model itself forgets everything between calls. The system around it saves notes, facts and summaries to a database, then searches that store and puts the relevant pieces back into the prompt the next time they are needed.

What is the difference between short-term and long-term memory in AI agents?

Short-term memory is the context window: the instructions, conversation and data the model can see right now. Long-term memory is stored outside the model, survives between sessions, and has to be retrieved before the model can use it.

What is a vector database used for in AI agents?

A vector database stores text as embeddings, lists of numbers that capture meaning, so the agent can find memories that are similar in meaning to the current question even when the words differ.

When should an AI agent use a knowledge graph instead of a vector store?

Use a graph when the questions are about relationships, such as which tenants are linked to which properties and contractors. Vector search finds similar passages; a graph follows connections and can answer questions that span many records.

Sources

  1. MemGPT: Towards LLMs as Operating Systems (Packer et al., arXiv)arxiv.org
  2. Generative Agents: Interactive Simulacra of Human Behavior (Park et al., arXiv)arxiv.org
  3. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Edge et al., arXiv)arxiv.org