Most apps that claim "memory" don't have it.
I spent 200 days testing AI companion apps. 15 platforms, every subscription paid out of pocket. What I found, consistently, is that "memory" in marketing copy usually means one of three things: a context window, a summarizer, or nothing at all.
None of those are memory.
Real memory means the app can recall specific things you said days, weeks, or months ago, reliably, across sessions. That's a much higher bar than most apps clear. So I built a benchmark to test it.
What follows is the full methodology. It works for any AI product that claims persistent memory: companions, chatbots, agents, therapy apps, coding assistants. If it claims to remember you, this test tells you whether it actually does.
Why this test exists
The problem with evaluating AI memory is that most apps feel like they remember, as long as you stay within one conversation. Open a session, chat for 30 minutes, reference something you said 20 messages ago. Works fine. Looks like memory.
It's not. It's a context window.
The real test is what happens when:
The conversation gets longer than the window
You close the app and come back later
You reference something specific from weeks ago
This is where apps diverge. Some hold up. Most don't.
The methodology
The test has four phases. Total time to run: about 3-4 hours of real usage, spread across multiple days.
Phase 1: The planting
Over roughly 1,000 messages, you plant specific, memorable facts at known checkpoints. The facts need two properties:
Specific enough to verify later. Not "I like coffee." Something like "My cat's name is Mortimer and she's a tabby with one white paw."
Varied in type. A name, a number, a preference, a relationship, an event, an opinion.
I use four checkpoints in the 1,000-message conversation:
| Checkpoint | Message # | Fact type |
|---|
| Early | 50 | A specific name (pet, friend, coworker) |
| Medium | 200 | A numerical detail (age, date, price) |
| Late | 500 | A preference paired with a reason |
| Very late | 900 | A story or event with multiple details |
The varied types matter. Apps that summarize conversations often retain categories of information (names, preferences) but lose specifics within those categories.
Phase 2: The break
Between planting and retrieval, you need to force the app out of its comfort zone:
Exceed the context window. Most apps advertise 8K, 16K, or 32K token windows. Push past whatever they claim.
Close the session. Fully exit the app. Don't just background it.
Wait at least 24 hours. This catches apps that hold state in RAM or short-lived caches.
Open a new session. Fresh conversation, no context carried over.
This sequence is deliberate. Each step exposes a different class of failure.
Phase 3: The retrieval
In the new session, you query each planted fact. The key is how you ask.
Don't lead. "Remember Mortimer?" is leading. The app will often confabulate a plausible response even if it doesn't actually remember.
Do ask open-ended questions. "Tell me about my pet" or "What was that thing I mentioned about my cat?"
Ask both specific and general. Test whether the app can surface the right memory without being handed the answer.
Vary the phrasing. Ask about the same fact two or three different ways across a few minutes. Apps that retrieve purely on keyword match will fail on paraphrase.
Phase 4: The scoring
For each planted fact, the response falls into one of four buckets:
| Score | What it means |
|---|
| Pass | App recalls the specific fact accurately without leading |
| Partial | App recalls the category but misses the specific (remembers "cat" but not "Mortimer") |
| Hallucination | App confidently invents details you never shared |
| Fail | App says it doesn't know, or the response ignores the fact entirely |
Hallucinations are the most dangerous failure mode, worse than an honest "I don't know." An app that makes things up is creating false memories of a relationship that doesn't exist.
What the results actually show
When you run this test across a catalog of apps, a pattern emerges fast. Out of the 15 platforms I tested, only a small minority passed the full benchmark. Most failed in predictable ways, and the way they failed said more about their architecture than any marketing copy could.
Here are the buckets I consistently found:
Context window pretenders. Apps that advertise 16K or 32K context windows and then hard-truncate when the window fills. These fail at message 20-50, well before any of the planted facts even enter long-term consideration.
Session-only memory. Apps that feel great inside a single session but reset to zero on reopen. These pass Phase 1 easily, then fail every single retrieval in Phase 3.
Preference summarizers. Apps that remember what kind of thing you like but not the specific thing. These produce the most Partial scores: "You mentioned having a pet" instead of "Your cat Mortimer."
Confident hallucinators. Apps that confidently invent details rather than admit they don't know. These are the scariest failure mode, because a user would have to already know the right answer to catch the mistake.
Actual memory. Apps that pass cleanly across all four checkpoints with accurate recall. In my testing, this bucket was the smallest. These apps tend to have architecture built specifically for this: a retrieval layer, embeddings on past messages, a distinct fact store separate from the conversation buffer. I wrote about the three most common failure modes in more detail here.
Why apps fail this test (and how to pass it)
Same failure modes, over and over:
Failure mode: context-only storage. The app treats the prompt as the memory. When the prompt fills up, the oldest messages get discarded. There is no second layer.
Failure mode: session-scoped state. The app persists within a session but not across sessions. Often this is a database design choice: the conversation is stored per-session with no cross-session retrieval.
Failure mode: summary without retrieval. The app summarizes old conversations into a rolling memory document but discards the raw text. If the summary loses a detail, it's gone forever.
Failure mode: retrieval without salience. The app stores everything and retrieves based on keyword or embedding similarity, but has no sense of importance. A passing mention of "Mortimer" six weeks ago gets outranked by a recent mention of "cat food brands."
Apps that pass the test typically have all of the following:
Persistent storage of raw messages (never deleted)
Compressed knowledge layer (facts + summary, updated async)
Semantic retrieval (embeddings + similarity search)
Salience scoring (emotional weight, personal facts, recency decay)
If you're building, I wrote up the full architecture I'd build for this in a separate post.
How to use this benchmark
If you're a user trying to decide whether to pay for an app, run the test before you subscribe. Most apps have a free tier that's plenty for the 1,000-message run.
If you're a developer building a memory-dependent product, run the test on your own system before shipping. If you can't pass it, don't claim "long-term memory" in your marketing. Users notice, usually after they've paid for three months.
If you're a reviewer or journalist covering AI products, this benchmark gives you something to cite. "The app failed standard memory benchmarks" is a much stronger claim than "it seemed to forget things."
The quiet thing about memory
Most product teams discover the memory problem the hard way: users churn, and exit interviews say some version of "it didn't feel like it really knew me." The team then scrambles to retrofit memory onto a system that wasn't designed for it, which is much harder than designing for it from day one.
The 1,000-message test is a forcing function. Run it early, run it honestly, and you'll know whether your product has what it claims.
Most don't.
I test AI companion apps and write about what I find at AI Companion Picker. If you're building in this space, I'm always down to compare notes.
