I hold my work to a standard I learned running incident bridges: if I can't explain a system clearly, I don't understand it yet. This page walks through five projects the way I'd brief a decision-maker - the mechanism, what each number means in practice, and the questions worth asking me about each. Every figure comes straight from the published results, losses included, because a result you can't check is just a story. The bundled agents are deliberately simple scripts, so every number proves the measuring instrument, never any AI model's intelligence. The working vocabulary at the bottom defines the field's terms the way I use them.
An AI's long-term memory, kept as a wiki a human can read and check.
Imagine a doctor who, before every appointment, re-reads your entire file back to page one - every cold, every checkup, thirty years of paperwork. This project teaches the AI to work like a good nurse instead: keep the chart organized, pull only the pages that matter for today's visit, and leave the rest in the cabinet.
Selective reading used 0.61× the tokens of loading everything - about 61 cents on the dollar, and tokens are the meter AI bills run on. On a small pile of notes, though, the simple load-everything approach won (selective reading cost 1.27× there), and that loss is published rather than buried.
I measure cost honestly and publish the exact case where the clever method loses.
Tests AI code reviewers by hiding one known bug and checking they find it at the right line.
It's a fire drill for AI reviewers. Instead of waiting for a real fire, I plant one - a single known bug in otherwise clean code - and check whether the alarm goes off at the right spot. Clean files are mixed in too, because an alarm that shrieks at burnt toast every morning is as useless as one that never rings.
The reviewer caught 9 of 10 planted bugs, scored at the planted line within two; the tenth is a deliberate blind spot shown as a red miss, in public. The new best-of-k study is a wind tunnel rather than a road test: nine copies of one reviewer, each dealt its own synthetic error stream, independent by construction and so the friendliest case there is, vote on findings and climb exactly as the jury arithmetic says they must. Matching the math proves the vote counting works, not that any reviewer got smarter. The controls carry the real lesson: nine copies sharing one error stream gain nothing, a panel that's wrong more often than right votes itself worse, and real reviewers land somewhere between those lines.
I test the testers - and know when a committee adds judgment and when it's just an echo.
Splits AI decisions into a proposer, a professional skeptic, and a judge - then cross-examines the judge.
Think of buying a used car. One person falls in love with it, so you bring a mechanic friend whose only job is to find what's wrong, and a level-headed third person makes the final call. This project wires an AI the same way - and keeps a notebook of every car they walked away from, and why.
In one real case, 3 of 4 verdicts flipped when nothing changed but the judge's standing instruction. Same facts, same arguments, different outcome - meaning the referee, not the evidence, was deciding the game. The flip test exists to catch exactly that before anyone trusts the verdicts.
What these numbers are not. That is a single case study, clearly labeled, and not a benchmark - four verdicts is four verdicts. The other figure you will see quoted, 0 of 8 against 3 of 8, comes from two judges I wrote myself to behave that way, so it is true by construction. Together they show the instrument lights up when a judge is instruction-sensitive. Neither one shows that any judge you would actually deploy passes, and I would rather say so than let a good number travel further than it has earned.
A verdict is only as trustworthy as its judge - so I built the tool that puts the judge on the stand.
When the AI's memory fails, the failure gets written down, fixed, and re-tested on sealed questions.
Before touching anything, I wrote a set of test questions and sealed them in an envelope. Then came the fixes - each one citing, in writing, the exact failure it repairs, like a mechanic's work order. Only at the end does the envelope open. If the score climbs on questions nobody could have studied for, the improvement is real.
The sealed questions went from 8 out of 10 to 10 out of 10 after the fixes. The proof it wasn't luck: a placebo arm - the same amount of effort, aimed at nothing in particular - stayed flat. The targeted fixes did the work; the busywork alone did nothing.
I run self-improvement like a clinical trial, sugar pill included.
A retro browser game where the core party, the bosses, and the numbers come from the real projects.
Think of a chef who, instead of handing you a résumé, cooks you dinner from it. This 8-bit game plays in your browser: four of the party are the labs above and the rest are their own people, each boss is a real way AI systems fail - runaway retries on the main road, then made-up facts and smuggled instructions deeper in - and every number a character quotes can be checked against the repos, including the balance figures, which now ship with the simulator and its run logs. Clear every arc and a post-credits finale opens: THE GRAND STAGE, a live industry exercise where a licensed red team is the final exam and the ending is earned, not granted - the same idea the mutation lab applies to the game's own tests.
The game quotes the portfolio's real results - the 0.61× memory cost, the 9 of 10 bugs caught, the 8-to-10 sealed-envelope jump - as in-game facts. Nothing was invented for drama. If a boss fight cites a number, that number lives in a repo you can open.
I can teach hard ideas to anyone - and I kept the fun on the same honesty rules as the labs.
▶ Play it (~10 minutes, phone-friendly)
Ask the AI several times and take the best or majority answer - spending extra thinking time instead of waiting for a smarter AI.
Ask nine doctors who have not talked to each other and take the majority - but if all nine trained at the same school with the same blind spot, nine opinions are really one.
Getting more out of the AI we already have by removing the clumsy ways we use it - no new model needed.
Like discovering your microwave had a defrost button all along - the machine didn't get better, you finally used what it could already do.
Deliberately planting a known bug to check whether your tests - or your AI reviewer - actually catch it.
A safety inspector hides one fake violation in the factory to see if the auditors flag it - miss the plant, and they're missing real ones too.
An automatic check that re-runs every published number on each change and blocks the change if a claim stops being true.
Like a restaurant re-weighing every advertised portion each morning - the menu isn't allowed to say eight ounces on a day the scale says six.
A comparison group that gets the same effort but not the real fix, so you can tell what caused the improvement.
Half the patients get the real pill and half get a sugar pill - if both groups improve the same, the pill wasn't the reason.
Test questions locked away before any fixes begin, so nobody can tune the system to the test.
The final exam stays in a sealed envelope while you study - you can't rehearse the exact questions, so your grade actually means something.
How well confidence matches reality: when a system says it's 90% sure, it should be right about 90% of the time.
A weather forecaster who says "70% chance of rain" is well calibrated if it rains on about seven of every ten such days.
Several independent safeguards layered together, so a malicious instruction that slips past one still runs into the next.
A bank doesn't trust the front-door lock alone - there's a guard, a vault, and an alarm, so one picked lock doesn't empty the safe.