Most of us here have shipped something with an LLM behind it and then watched a user hit the same wall: the thing has no idea what happened yesterday. This is a walk through what a memory layer actually does, in the order the code runs, because the framework docs tend to jump straight to the API call.
The Model Is Not Forgetting, It Never Knew
Every call to a language model is independent. You send a prompt, you get a response, and nothing about that exchange persists anywhere the model can reach. The context window is working memory for one session, and when the session ends it is gone.
That means "memory" is not a model feature you enable. It is an application layer you build or adopt, sitting between your code and the API call.
The Four Steps That Make Up A Memory Layer
Every implementation, whether you roll your own or pick a framework, comes down to the same four operations.
Extract. Something has to decide what in this exchange is worth keeping. Most naive implementations store the whole transcript, which is the cheapest thing to build and the most expensive thing to live with.
Store. The extracted facts go somewhere with structure: a vector index for similarity, a relational table for exact lookups, a graph if the relationships between entities matter more than the entities.
Retrieve. On the next request, something has to pull the relevant subset. This is where the quality of your extract step gets its bill. A store full of noise makes retrieval look broken when the retriever is fine.
Inject. The retrieved context goes into the prompt. The model has not changed at all. Only what it sees has changed.
The Two Failure Modes You Will Actually Hit
Pull too much and you pay in tokens and latency on every single request, and the signal gets buried in the middle of a long context where models attend to it least.
Pull too little and the agent confidently repeats a decision the user already reversed. That one is worse, because it reads as the model hallucinating when the store simply never surfaced the correction.
There is a third that creeps in over months: nothing ever expires. Preferences change, architectures change, and a memory layer with no invalidation path will keep handing the model facts that were true in March. Old memory treated as current truth is more damaging than no memory at all.
What To Decide Before You Write Any Of It
Two questions decide most of the design. What has to be true before a fact earns permanent storage, and what causes that fact to be invalidated when it stops being true.
Answer those and the rest is plumbing. Skip them and you will ship something that works beautifully in a demo and degrades quietly over the first few thousand real conversations.
If you want the longer version, including how the framework options compare and what the production cost and latency picture looks like, there is a full guide to AI memory for applications that covers it.
