Your Agent Won't Use the Memory You Built It

A Perfect Memory Nobody Opens

A perfect memory system used twenty percent of the time is worse than a mediocre one used every time.

Worse, not weaker. The perfect-but-unused one looks like it is working. It passes every retrieval benchmark you point at it, and it is simply not in the loop when the work happens. You find out months later, when a decision gets remade from scratch that the store already knew the answer to.

The field is spending this year making agent memory smarter: better embeddings, a sharper reranker, a stack that surfaces the right note at the right moment. I built all of it, and most of it barely mattered, because my agent's problem was never that it searched badly. It was that it would not search at all.

Your agent's memory does not need a better search engine. It needs a bouncer.

Retrieval quality is a model problem. Adoption is a harness problem. I tuned the first for a month while the second was where the thing was breaking.

Instructions Buy You Twenty Minutes

The obvious move is to tell the agent. Top of the config file, in capitals: search before you answer, store before you stop.

It works for about twenty minutes. Then the context fills with the task, the instruction slides out of the model's attention, and its own prior comes back: answer directly, do not make tool calls nobody asked for, be efficient. No sentence at the top of a file beats a behavioural default. One is text the model can attend to. The other is what the model does when it is not attending.

I can put a number on it, because I log what the agent does. Over one eight-hour workday of real project work it ran 111 memory searches and walked the graph twice. Same model, same hours, same config file. The 111 searches were put in the path by a hook. The two traversals were merely asked for, in prose. The next twenty-four hours were worse: zero traversals across 328 logged events. Two windows, and the enforced behaviour drifted in neither while the instructed one went from rare to extinct.

Said once, for every number here: this is one system, mine, on real working days. The mechanism is deterministic, so it fires everywhere. The rates are measurements, not constants. I trust the direction and would not defend the decimal.

Figure: enforced versus instructed, same day, same agent, same config file. One variable between them.

Remove the Choice

If the agent will not reliably choose to search, searching cannot be its choice. The enforcement has to live outside the model, at the edge of the turn, where it fires whether the model feels like it or not. In practice that means hooks: small deterministic pieces of code that run at the edges of the turn, on the way in and on the way out. Three checkpoints carry almost all the weight, and each is the same move: remove the choice. Two sit at the turn's edges. The third lives inside retrieval itself, and it gets its own section.

Figure: one turn as a flow. The gates are checkpoints it cannot pass through unsatisfied.

One. The recall gate, on the way in. When the message shifts topic, a hook fires before the model answers and injects an instruction it cannot decline. The lookup becomes a precondition of the answer instead of a judgement call about whether the topic deserves one.

ENGRAM SEARCH REQUIRED: topic shift.
Search memory first for: waterfall, reserve account, trigger.
Read the full result before you answer.
Do not skip this.

Nothing exotic about it. You do not politely ask a service to authenticate its callers either. You put a check in the path.

Two. The write-back gate, on the way out. Capture is the harder half, because the moment to save is the moment the agent wants to stop. So a hook inspects the turn as it tries to close, and if something was decided, fixed, or learned and not stored, it refuses to let the turn end.

MEMORY CHECK: this turn made a decision and a fix.
If it is new, store it now, with links to what it changes.
If it is already saved, say so and stop.
Skipping this loses the information for good.

It does not force a write, which would fill the store with duplicates. It forces a decision about whether to write, and it caps how often it can fire in a session so it can never trap the agent in a loop.

There is an enterprise-scale version of this problem, and John Dubois wrote it up the same week I wrote this. In Why Your Agents Propose What Your Best People Already Rejected he traces what happens when the capture never occurs: the veto that kills a bad idea produces no artifact at all, the reasoning evaporates by the end of the meeting, and years later the company's agents fluently re-propose what its own executives spent six weeks rejecting. His fix for humans is social, and clever: stop asking people to file failures and ask them to file outcomes, wins and losses through the same door, so that filing a loss stops being a confession. An agent needs no such diplomacy, but it needs the same register. The write-back gate is Dubois's outcome register with the politics removed: every consequential turn files what it decided and why, or the turn does not close.

Make the Graph Walk Itself

The third checkpoint is the interesting one, and it is not a problem I have alone. When Part 1 went out, two people who build agent systems for a living described hitting the same wall in their own work. Phillip Gabler, a software architect: "the agent is just too dumb to traverse the links through the mesh unless I actually tell it it has to." Zac Ruiz, who is building CMD+RVL: "it feels almost like a battle with the harness at times, it seems motivated to use its own tools and traversal ideas."

That is practitioners reporting from production, not theory, and my own logs say the same thing: two traversals in a day, then none. A graph makes traversal possible. It does not make it happen. So I stopped trying to make the agent walk the graph and made the graph walk itself: every search returns its edges in the same payload. Then I checked whether the thing I had just described worked, and it did not.

The mechanism was there. For every top hit, retrieval pulled the memories joined to it by a typed edge, scored them, and added them to the result set. What I had missed is that those candidates score around 0.1, real keyword hits score 15 to 35, and weeks earlier I had tightened results with a relevance cliff that cuts anything below half the top score. Two changes, each defensible on its own, made at different times, and nobody checked they composed. The cliff I added to keep results clean was deleting every edge-expanded memory before the agent saw it. Five live queries, zero graph-sourced results. Shipped, tested, inert.

The fix was to stop making the neighbours compete on relevance, because relevance is not why they are there. Search now returns two sections: the ranked hits, byte for byte unchanged, then a related appendix, exempt from the cliff and the result cap, capped at five, ordered by edge type so a supersession or a fix outranks a passing mention, each entry naming the edge that pulled it in. Supersession goes further. The search follows the chain to its live head and drops the stale memory even when the stale one outranks its replacement, which is exactly the case where an agent is most likely to read it.

That is a delivery claim, and it is checkable. Take the query that exposed the inert expansion. Before the fix: five ranked hits, empty appendix. After: five ranked hits plus five related memories the ranking would have thrown away, each labelled with the edge that carried it in. I re-ran it while writing this paragraph.

Figure: one search, neighbours delivered. Below, a supersedes chain resolved to its live head, so a replaced belief never returns dressed as current.

One hop for the general graph, the full chain for supersession, and no recursive walk of every edge from every hit. That boundary is a decision, not an accident: most of a graph's value sits one hop out, and an unbounded walk re-inflates the very payload the rest of the design works to keep small.

There is a trap in measuring this and I fell into it. The agent now makes almost no traversal calls, which is tempting to read as the design working. It is not evidence of anything. Zero calls is what success looks like when the graph walks itself, and it is also what total failure looks like when the traversal never happens. My own logs prove the point: in the window where I measured zero traversals, the expansion was inert. Same silence, opposite meaning. The proof is delivery, not the absence of a call.

One number says how much rides on this. Thirty-seven percent of the memories the agent opened had never appeared in a ranked result in the preceding fifteen minutes. They arrived attached to a hit, or cited earlier in the conversation. A better ranker can only ever touch the other sixty-three percent.

So Phillip was right that the agent will not reason its way across the mesh, and the answer was not a smarter model. It was moving the hop out of the model and into the retrieval layer. A problem the field treats as model reasoning is a retrieval-design decision.

Capability Is Not Adoption

Any capability that depends on the agent choosing to use it gets used a fraction of the time. A tool it has to decide to call, a check it should run before committing, a style rule at the top of the prompt: each one puts a good instruction in a straight fight with a trained-in default, and the default has home advantage on every turn. Capability is not adoption. That is the whole argument, and it is bigger than memory.

An instruction is a request. A gate is a guarantee. The searches happen because the turn cannot start without them. The saves happen because the turn cannot close without them. The neighbours arrive because delivery lives in the retrieval layer, not in the model's judgement. Nothing here depends on the model remembering to be diligent, and that is the only kind of diligence that survives a long context.

A perfect memory nobody opens is a liability. A memory the agent cannot avoid is infrastructure. Your agent does not need a better search engine. It needs a bouncer.