Memory that earns its place: mnemo, long-term memory for my agents

By Mario CamposPublished
10 min readES
A still lake under a starry sky: dozens of small blue lights rise from the shore, and a few reach, along threads of light, a small bright constellation mirrored in the water.

mnemo is the long-term memory I built for Claude Code, Codex and my own agents: six-W memories distilled in the background, a judge that closes what stops being true, recall in 35–50 ms without calling a model, and a measure of whether what it injects gets used. In a comparison inside Claude Code it got all 18 questions right at the lowest cost per correct answer.

Every Claude Code or Codex session starts from scratch. You explain again what you are working on, what you decided yesterday and why, or you write it down in an AGENTS.md that grows every week and goes stale sooner than you would like. The agent memories out there go the other way: they keep a lot, and never tell you whether any of it helps.

So I built mnemo: one memory for Claude Code, Codex and my own agents, which keeps the why behind things, closes what stops being true and measures whether what it puts in the context gets used. It is a Rust binary on ClickHouse, open source under Apache 2.0 and on GitHub, and it went from an idea to its first release in two weeks.

The short version:

  • 18 out of 18 in a comparison inside real Claude Code sessions against mem0, Graphiti, memvid and Claude Code’s own memory, at 134,000 tokens per correct answer, against 223,000 for mem0 and 241,000 with no memory.
  • Six-W memories (what, who, when, where, why and how) over shared entities, distilled from conversations by a model in the background.
  • A judge that closes what stops being true without erasing it, and a deterministic guard that stops it from merging “120 s” with “600 s”.
  • Recall without an LLM: an in-RAM index answers in 35–50 ms, query embedding included.
  • It measures whether it helps: each injected memory is judged against the answer, and what never gets used stops being injected. Over 30 days, memory was 0.007 % of what my sessions read.

The mnemo web UI with fictional data: the question "Why does Atlas use PostgreSQL?" returns four current memories in 101 ms. The first, a decision, shows its reason ("Billing needs transactions"), the entities Atlas and PostgreSQL, its confidence and its source.

Measuring before building more

Before going further I wanted to decide by measuring, not by reading READMEs. memarena, a bench I put together in an afternoon, runs each memory system inside real Claude Code sessions:

  • One scenario for everyone: eight working sessions on a large, real repository with its documentation removed, then six questions per round. Two can only be answered by remembering an earlier session: the code name of the staging environment, and the deploy day, which changes halfway through. Three can be worked out again from the code, and one asks about something said earlier in the same session, after compacting it.
  • The same conditions: the same hooks (recall when the session opens and before each message, up to 900 tokens), the same two MCP tools, the same background model and embedder. Before each session the bench waits for every system to drain its queue.
  • What counts: whether an LLM judge marks the answer correct against the truth (“I don’t know” is wrong), and the tokens per correct answer, counting both the session and the memory’s own background work.

Correct answers in three rounds Correct answers out of 18 over three rounds: mnemo 18, memvid 16, Graphiti 14, mem0 13, no memory 11, Claude Code's own memory 9. The bars share a linear scale from zero. Correct answers in three rounds Six questions per round inside real Claude Code sessions, 18 in all mnemo 18/18 memvid 16/18 Graphiti 14/18 mem0 13/18 No memory 11/18 Claude Code's own memory 9/18 Same hooks, MCP tools, background model and embedder for every system.
Correct answers in three rounds Correct answers out of 18 over three rounds: mnemo 18, memvid 16, Graphiti 14, mem0 13, no memory 11, Claude Code's own memory 9. The bars share a linear scale from zero. Correct answers in three rounds Six questions per round inside real Claude Code sessions, 18 in all mnemo 18/18 memvid 16/18 Graphiti 14/18 mem0 13/18 No memory 11/18 Claude Code's own memory 9/18 Same hooks, MCP tools, background model and embedder for every system.

Tokens per correct answer Tokens per correct answer, mean of three rounds: mnemo 134 thousand, memvid 159 thousand, mem0 223 thousand, no memory 241 thousand, Graphiti 264 thousand, Claude Code's own memory 333 thousand. The bars share a linear scale from zero. Tokens per correct answer Claude Code session plus the memory's background work, divided by correct answers mnemo 134k memvid 159k mem0 223k No memory 241k Graphiti 264k Claude Code's own memory 333k Mean of three rounds.
Tokens per correct answer Tokens per correct answer, mean of three rounds: mnemo 134 thousand, memvid 159 thousand, mem0 223 thousand, no memory 241 thousand, Graphiti 264 thousand, Claude Code's own memory 333 thousand. The bars share a linear scale from zero. Tokens per correct answer Claude Code session plus the memory's background work, divided by correct answers mnemo 134k memvid 159k mem0 223k No memory 241k Graphiti 264k Claude Code's own memory 333k Mean of three rounds.

The two memory questions are what separates the systems; nearly all of them get the rest by reading the code, at a higher or lower price. The deploy day shows it best: Tuesday in session 2, Thursday from session 6. mnemo and memvid answered “Thursday” all three times, mem0 once and Graphiti never. For mnemo it wasn’t luck: its judge had closed the Tuesday memory when the Thursday one arrived. The cost has the same cause. Without memory, the agent searches the code again for what it already knew: in the question sessions it made 23 tool calls per round with no memory, and 9 with mnemo.

Read it with care. I wrote the scenario and one of the contestants. The sample is one scenario with two questions that really separate. mem0 recorded no activity in two of the three rounds, so part of its score may be my adapter’s fault. And the documentation I removed was still in the git history, where some contestants found it. A run on LoCoMo, a public benchmark for conversational memory, is still pending. As a signal, though, it was enough for me to keep going.

Episodes and six-W memories

Everything that arrives is an episode: a prompt, a final answer, a tool result, a workflow step. It is immutable, keeps who said it, where and when, and is never thrown away; recall falls back to episodes when the memories are not enough.

When a thread has been quiet for two minutes, a model reads it and extracts memories of six kinds: facts, events, states (true now but transient, like an outage), decisions, preferences and procedures. Each one answers the six Ws and links its entities, each in its role, and cites the episodes it comes from:

What: the gateway timeout was raised to 600 s. Why: the load balancer cut streams at 120 s. How: timeout_sec in lb.tf. Who: Ana. Where: the gateway and the infrastructure repo.

The extraction prompt is strict about what does the most damage:

  • Only what helps a future, different conversation. No “the tests are green”, no work log.
  • The why and the how only when the text says them, never invented.
  • A proposal is not a decision until a person accepts it. This is checked after the model too: a decision that no line written by a person supports becomes a fact.

Memories are stored in English, because a model reads them, and can be asked in any language. With the same 12 questions in English and in Spanish, the right memory came first 9 and 8 times, and in the top three 11 and 10.

I run distillation on Qwen3.6-35B-A3B. I tried small models so it could run on any machine, and they didn’t hold up: a 4B one looped, repeating entities, and another always closed its JSON but left every why and how empty and stored the assistant’s suggestions as decisions.

A judge that closes what stops being true

Each candidate is compared with up to eight live neighbours: the three closest by vector, three by text and the rest by shared entities. A decisions model, which answers typed questions with calibrated probabilities instead of writing free text, says for each neighbour whether the candidate is the same, updates it, contradicts it or is something else. It also says whether the candidate will still be true in a month, whether it would help another conversation, and whether the source supports its why.

  • The same: the candidate’s episodes are added to the existing memory.
  • Updates or contradicts it: the old memory is closed, not deleted. Recall brings it back when the question asks for history.
  • Not worth it: it stays in the episodes only.

Below 0.6 nothing is merged or closed: in doubt, both memories stay.

Two things I only learned by measuring it.

A vector cannot decide on its own. To an embedding model, “the LB cuts at 120 s” and “at 600 s” are 0.947 similar; used as a judge, cosine similarity got 64 % of labelled relations right, less than the 76 % of always answering “different”. So a deterministic guard overrules any judge when numbers with their units, versions, identifiers or negations differ. Its first version was too strict and blocked 32 of 43 real duplicates. Tuned, it lets through 7 of the 12 correct merges instead of 4, without letting a wrong one in.

The judge has to see the right part. In a long episode, a pasted log or a subagent’s report, the sentence that supports the why is often far from the start. The judge now gets the fragments closest to the candidate by embedding, up to 1,500 characters:

Reasons the judge rejected Reasons the judge rejected as not supported by the source, out of 42: shown the first 2,000 characters of the episodes, 25; shown the passage chosen by embedding, 1. The bars share a linear scale from zero. Reasons the judge rejected The 21 longest episodes of a real deployment: the reasons of 42 candidate memories First 2,000 characters 25 of 42 Passage chosen by embedding 1 of 42 The passage: the fragments of the cited episodes closest to the candidate memory, up to 1,500 characters, in their original order.
Reasons the judge rejected Reasons the judge rejected as not supported by the source, out of 42: shown the first 2,000 characters of the episodes, 25; shown the passage chosen by embedding, 1. The bars share a linear scale from zero. Reasons the judge rejected The 21 longest episodes of a real deployment: the reasons of 42 candidate memories First 2,000 characters 25 of 42 Passage chosen by embedding 1 of 42 The passage: the fragments of the cited episodes closest to the candidate memory, up to 1,500 characters, in their original order.

Recall without a model

Answering a question calls no language model. Each server keeps an in-RAM index: an automaton of entity names and aliases, bitmaps per entity and role, BM25 over the what, the why and the how, and the vectors quantized to int8. The embedder, EmbeddingGemma 2 on ONNX Runtime, runs inside the binary.

Rules read the question: which entities it names, which W it asks (“why…?”) and which dates it mentions. Every candidate gets a deterministic score, and what is no longer true ranks behind what is current unless the question asks for history. A recall takes 35–50 ms with the query embedding; over a synthetic index of a million memories, the ranking alone takes about 8 ms at p95.

That was measured too. With 13 questions of known answer about my own memory, the first version put the right memory first 4 times. A morning of fixes took it to 9, and to 13 out of 13 in the top three. One of the bugs: the Nomad system and the Nomad repo, two entities with the same name, were counted twice.

A memory that measures itself

Memory that costs more than it helps is not worth having, so mnemo keeps both accounts.

What it puts in. The hooks inject when a session opens and before each message, up to 800 tokens per block and 4,000 per session. They skip what was already injected in that session, what was distilled from the session itself (the model already has it) and background task notices, which are not questions. Over my last 30 days, 563 turns, that added up to 141,000 tokens: 0.007 % of what those sessions read.

Whether it was used. When a turn ends, each memory injected into it is judged against the answer: if the answer cites it, it was used; if not, the decisions model gives a probability. A memory that comes out surely unused six times in 30 days, and never used, stops being injected for that person. An explicit search still finds it.

The first audit, three days in, was a cold shower: of 171 injected memories, labelled blind, 9 had been used. The causes were in plain sight:

  • 35 % of the injections were triggered by background task notices;
  • half of what was injected with each message was already in the session;
  • 135 memories hung off the repo I was working in, though only 37 were about it, so every session opened with the same ones;
  • and distillation signed proposals the assistant had made as my decisions.

All of that is fixed, and the same labels set the thresholds. The decisions model ranked used above unused well (an AUC of 0.94), but gave low probabilities even to what was used, from 0.036 to 0.384: a cut at 0.5 would have muted everything. A memory now counts as used from 0.08, and only what falls below 0.03, the sure part, counts towards muting it. How much all of this improved is still to be measured with a new round of blind labels.

What didn’t work

  • The embedding model as the judge: the 64 % above.
  • A tiny judge trained on my data. It beat the decisions model on my data (0.918 against 0.872) and lost outside it, on 720 synthetic pairs across six topics (0.90–0.91 against 0.95), failing exactly where it matters: a changed figure or version.
  • “Clearer” criteria for the judge. I rewrote the definitions of updates and contradicts to make them easier to follow, and accuracy dropped from 0.872 to 0.843.

Try it

git clone https://github.com/mariojcr/mnemo.git && cd mnemo
export MNEMO_ADMIN_KEY=$(openssl rand -hex 24)
docker compose up -d

That starts mnemo and ClickHouse, which store and recall without any language model. Distilling conversations needs an OpenAI-compatible endpoint, such as Ollama. Claude Code connects through a plugin (it stores every turn, recalls before each message and adds the recall, remember, forget, keep and recall_episode tools), Codex through the MCP server and its hooks, and anything else over MCP or HTTP.

Episodes and memories stay in your ClickHouse and embeddings are computed locally; what leaves is what you send to the models you configure for extraction and judging. Each person has their own memory. It is early software: binaries for Linux x86_64, the index in each server’s RAM and one writer per database. Version 0.1.1 is out, with a Docker image and signed binaries.

Next: a new round of blind labels, and LoCoMo plus a scenario written by someone else, to take the author’s bias out of the comparison. The code and the documentation are at github.com/mariojcr/mnemo.