wiki-memory-lab - watch an AI agent's memory work

Long-term memory for AI agents as a human-readable markdown wiki: an index skimmed at session start, selective recall of only the notes a task needs, and notes that get extended - or thrown out, with a written reason - as they prove right or wrong.

✓ every number on this page regenerates in CI with zero API keys - if a claim drifts, the build fails

The idea in 30 seconds

Most AI assistants either forget everything between chats or reread their entire diary before every answer - slow and wasteful. This one keeps a recipe box: at the start of a session it skims the card titles, pulls only the two or three cards today's question needs, updates those cards afterward, and throws out any card proven wrong. And the box is just plain text - open the folder in any editor and read your AI's whole memory yourself.

The concept of agent memory as a self-maintained markdown wiki is Andrej Karpathy's (attribution, no affiliation). This lab builds it small - and then measures it.

The session loop

Every session runs the same five beats. Click a stage.

A real session, replayed

These lines are quoted from the committed transcripts (runs/) that CI regenerates on every push. Step through the story - including the moment the memory deletes something it learned was wrong.

MODE: ScriptedAgent - deterministic, zero API keys; write-backs are rule-driven

Verbatim excerpts from runs/milldale-session_02/04/06/07.md, lightly condensed between beats.

The numbers - each with its honest caveat

The full table lives in the README, rendered from metrics.jsonl by report.py - no measured number is typed by hand, and the CI drift gate fails the build if regeneration disagrees.

0.61×

Selective recall vs loading everything

Context tokens over 8 sessions: 2,236 vs 3,654 (proxy tokens = chars/4). The ratio is the claim, not the absolute counts.

1.27×

…and where the design LOSES

On a small 8-note corpus, stuffing wins - the index is a standing per-session tax. Published on purpose: a claim without its boundary isn't finished.

0.95

Retrieval precision & recall

Labeled an upper bound by construction in the same row - the same author wrote tasks, hooks, and labels. One miss is planted to show what a lazy hook costs.

1 / 0

false-CREATE / false-EXTEND

The exact-title matcher's confusion table. A paraphrased duplicate slips past it - counted and published, not papered over.

context tokens per session, three loading policies

The hero chart - its legend carries the disclaimer inside the SVG, so it survives screenshots: “scripted agent - measures the harness, not model capability”.

cumulative context tokens, selective vs stuffing, with the freed-budget gap shaded

The cumulative view - the shaded gap is context budget freed, and it widens as the wiki grows: stuffing scales with corpus size, selective recall scales with the task's working set. Regenerated by CI like every other number here.

Use it yourself - three commands, no keys

Clone and install (installs nothing but this package)

git clone https://github.com/LZBiala/wiki-memory-lab
cd wiki-memory-lab
pip install -e .

The runtime is Python stdlib only - the empty dependency list is a feature.

Run the demo

python -m wikimemlab demo

Eight sessions stream in seconds: index loads, selective recalls, write-backs, the prune with its written reason, the decay. Then the token curves are regenerated in front of you.

Open the memory in any editor

wiki/            ← the agent's memory: index.md + one note per concept
runs/            ← transcripts + ops.jsonl (every operation, with a reason)
report/hero.svg  ← the curves you just computed

No database, no viewer app. What does my agent believe should be an ls, not a research project.

Adapt it - bring your own model (v1.1 seam)

agents.Agent is the interface: implement choose_recall() and answer() with a live model and run the same harness. No live adapter ships in v1.0 - deliberately, so no published number can be mistaken for model judgment.

What this does NOT show

The circularity trap, in plain words

The bundled agent is scripted. A scripted agent that is told to use memory, using memory, proves nothing about intelligence - so this project never publishes a “task completion” curve, because completion tracks retrieval by construction. What the harness measures honestly is the memory protocol itself: token arithmetic, retrieval quality of one-line hooks against labeled tasks, and whether merge, prune, and decay fire when they should - including both directions the title matcher fails.