Persistent AI memory is the layer that lets an assistant remember facts about you across sessions, so you do not restate your job, your stack, or your preferences at the start of every conversation.
Ours was clever and it was wrong. In March 2026 we rebuilt it by removing almost every retrieval mechanism we had built, including the vector similarity search, and the results got better.
This is the story of what we deleted and why.
What does persistent AI memory actually store?
It stores durable facts about you, not conversation transcripts. Things like the language you work in, the name of your company, how you like answers formatted, or a constraint you mentioned once and should never have to mention again.
Memories are extracted from conversations and written to a Postgres table, and you can see, edit, and delete every one of them in your profile preferences. Explicit commands work too: telling ChatFuse to remember, forget, or update something is detected directly rather than being left to inference.
How does ChatFuse create a memory in the first place?
ChatFuse creates them two ways. Most are extracted automatically from the conversation by a model that reads the exchange and returns structured candidates, which are then checked against what is already stored before anything is written.
The second way is you saying so. Telling ChatFuse to remember, forget, or update something is caught by direct command detection rather than left to the extraction model to notice. That distinction matters because implicit extraction is a judgement call and an explicit instruction is not, and the two should never compete.
Each memory carries an importance level that decides how hard it fights for a place in the retrieved set: always, context, or rare. Pinning one puts it at the front regardless.
Why did the original design use vector search?
Because it is the obvious answer, and on paper it is the right one. You embed each memory, embed the incoming query, and retrieve the memories closest in vector space. Only the relevant ones reach the model, which keeps the prompt small and focused.
We went further than that. On top of vector similarity there was an LLM relevance filter to second guess the results, a text search fallback, graph expansion to pull in related memories one hop away, a score based selection fallback, and a proactive surfacing mechanism that injected a memory at random 15% of the time. Every layer was added to fix a problem the layer beneath it caused.
What went wrong with it?
The retrieval kept returning memories that had nothing to do with the question, and each safety net made the problem harder to see.
The similarity threshold sat at 0.3, which is low enough that almost any query matches almost any memory. The LLM relevance filter was told to consider indirect relevance, so it kept nearly everything, and when it errored it fell back to returning all candidates. Two separate injection pipelines had drifted apart and behaved differently. One of them labelled the injected block "Relevant user memory:", and the model read that label as an instruction, so it dutifully worked personal details into answers that never called for them.
Ask for a poem and get your job title in it. That was the symptom. None of the individual pieces were broken, which is exactly why it took so long to find.
What did you replace it with?
One ChatFuse database query. All active memories, ordered by pinned status, then importance, then recency, capped at 20 rows.
That is the whole retrieval path now. No embedding comparison at request time, no relevance filter, no fallback chain. We still compute and store embeddings on write, because they cost little and may earn their place later, but nothing in the read path touches them.
The model decides what is relevant at generation time, which is the approach ChatGPT and Claude both take. It turns out the frontier model reading the prompt is better at judging relevance than a cosine similarity score computed before the model ever sees the question.
Does injecting every memory not waste context?
Less than the alternative did. A 20 row cap on short factual statements is a small block of text, and it is smaller than what the old pipeline was returning once the relevance filter gave up and passed everything through.
There is a real limit here and we are not pretending otherwise. This works because personal memory sets are small. If a user accumulated thousands of memories the cap would start dropping things that matter, and selection would have to come back in some form. We would rather add that when the data demands it than keep a complicated retrieval path that was never earning its complexity.
How do you stop memory leaking into creative work?
With an explicit instruction in the system prompt rather than a retrieval rule, and the same instruction in both pipelines.
Both paths now tell the model to use memory only when it is needed for a complete and accurate answer. For creative and generative work, images, poems, stories, jokes, brainstorming, the plain version comes first and a personalized alternative can be offered after. The injected block is wrapped in <user-memory> XML tags instead of a sentence that reads like a command, which stopped the model treating the label itself as an instruction.
Changing "Relevant user memory:" to a tag fixed more misbehaviour than any retrieval change we made.
How does memory work when the model changes mid conversation?
The memory belongs to your account rather than to any provider, so it is handed to whichever model answers next.
ChatFuse routes consecutive turns to different models depending on the task, which we covered in AI model orchestration. A model picking up turn 3 still gets the same memory block that turn 1 got. That continuity is most of the argument for keeping memory at the platform layer rather than inside a single provider's account.
Who can see my ChatFuse memories?
You, and the model answering your message. Access runs through a guard that enforces incognito mode and a kill switch, so a conversation started in incognito never reads or writes memory at all.
Every memory is visible and editable in your profile preferences, and deleting one removes it from retrieval immediately. Our wider position on access is in zero trust AI data security.
What happens when two memories contradict each other?
A conflict service checks for contradictions when a new memory is written and resolves them rather than storing both. Without that step the model receives two incompatible facts with no way to know which is current, and the answer becomes a coin flip.
Is persistent AI memory the same as a long context window?
No. A long context window holds one conversation. Memory persists across conversations, across sessions, and across models. A model with a million token window still starts every new chat knowing nothing about you.
What did ChatFuse learn from the memory rebuild?
That most of our retrieval sophistication was compensating for a threshold set too low, and that each additional layer made the root cause harder to see. We removed 8 mechanisms and replaced them with an ordered query.
The rule we took from it: when a pipeline needs a fallback for its fallback, the problem is usually the first step, not the missing third one.
How do I try persistent AI memory in ChatFuse?
Memory is on by default and everything it holds is visible in your profile. Start free, tell it something worth remembering, and check what it kept.
Comments
Loading comments…