Memory Ep. 4: what if it worked like a hippocampus?

The last episode ended on a complaint. I had built retrieval that worked, but it was flat: it ranked notes by how close they sat to a question in vector space and then stopped, completely ignoring the links between them. And I said something that stuck with me afterwards: my own memory does not work by cosine similarity.

So I went to find out how it does work. Or at least how the part of the brain in charge of it is thought to work.

A short, honest detour into neuroscience

I am an engineer, not a neuroscientist, so take what follows as the mental model I built to steal from, not a textbook.

The hippocampus is, roughly, the brain’s indexer for episodic memory. It does not store a memory as one solid block. It binds together the scattered pieces, where you were, who was there, what it smelled like, and keeps the pointers that hold them together. Later, one fragment is enough: give it a cue and activation spreads across the connections to related traces until the whole thing reassembles. A piece brings back the whole. The jargon is pattern completion; the everyday version is one thing reminding you of another.

The word that stuck was association. You do not retrieve a memory by word-similarity. Things come back because they are linked, and the link can be arbitrary and deeply personal, something no stranger would ever guess.

That is exactly what my vault had and my retrieval was throwing away. The [[links]] are the bindings. A note about a person is already wired to the projects they touch, the decisions they made, the days they show up in. My past self had handed me an association graph for free, and my shiny vector search was treating every note as an island.

That is the moment the project got its name. Hipocampo: Spanish for hippocampus.

I wasn’t the first to want this

Before building anything, I went looking, half to learn and half to check I was not about to reinvent a worse version of something that already existed.

There is a whole wave of “memory for AI” work right now: Mem0, Letta (the MemGPT lineage), Zep, Cognee, MemU, and more turning up every month. There are benchmarks for it, like LoCoMo for long-term conversational memory. And the graph idea is not new on the retrieval side either, Microsoft’s GraphRAG builds a graph over a corpus and retrieves through it.

Reading across them, I noticed they mostly share one move: they try to infer the structure. Take raw text, or a stream of conversation, push it through an LLM to extract entities and relations, build a graph out of what the model guessed, and retrieve over that.

It is clever work. It also felt like solving a problem I did not have.

The bet

Here is the thing none of them can assume, but I could: my knowledge is already structured.

I did not dump raw text into a box and hope a model could reconstruct how it fits together. I spent months writing notes and linking them by hand, at the moment I wrote each one, while I still remembered why two things belonged together. The association graph is not something to rebuild with an LLM and pray it guessed right. It is sitting in the vault, authored, deliberate, mine.

So my bet was almost the inverse of the field’s: don’t infer the graph, use the one the human already drew. Treat the wikilinks as synapses. When a question lands on a note, let activation spread along the links to its neighbours, the way a cue spreads through the hippocampus, and let the structure I built on purpose do the work a similarity score can’t.

On paper it was beautiful. A graph I did not have to guess at, a recall mechanism with a clean biological story, and a thesis that set it apart from everything I had just read.

On paper.

Next

The obvious next step was to stop admiring the idea and build it: turn the links into a real graph, run association over it, and, the part I kept quietly postponing, actually measure whether it beat the boring flat search from Episode 3.

That is the next episode. It did not go the way I wanted, and that turned out to be the most useful thing that happened in the whole project.