How we measured. What we found. What failed.
Most memory-benchmark numbers cannot be compared: different judges, different answer models, different question subsets. This document is the complete method behind our number — frozen protocol, ten certified runs, the ledger of what we lost as well as what we gained, and the experiments that did not work. Every figure below is reproducible from the open harness.
One protocol everyone can be scored on.
We adopted the strictest published configuration — the protocol of the Mem0 paper (arXiv:2504.19413), reused verbatim by Zep's rebuttal — and froze it. Whatever is frozen defines comparability; whatever is free is the system under test.
- Judge: gpt-4o-mini, temp 0, verbatim accuracy prompt
- Answerer: gpt-4o-mini, temp 0
- Dataset: LoCoMo, categories 1–4, n = 1,540
- Metric: J-score
- Parse failure counts as wrong
- Ingestion (per-turn, event-date tagging)
- Retrieval (hybrid fusion, k, expansion)
- Answer-prompt routing (disclosed)
- Memory engine configuration
Ten runs. One number.
Same judge, same answerer, same questions. Published numbers above these (including 90+ marketing claims) use non-comparable configurations and are excluded.
What we gained. What we lost.
Single-run numbers hide two things. First, churn: at temperature 0, re-running the identical configuration regenerates 5.4% of answers differently. Second, the trade: improvements are not free. Joined across run-pairs, our certified configuration persistently answers 170 questions the naive baseline cannot — and persistently loses 72 the baseline got right.
Our own persistence gate flagged this trade as exceeding the harm budget we set before measuring. We publish the gate's verdict with the number: the configuration is certified as net-better with a disclosed trade, not as strictly better. If a memory vendor tells you their system only adds correct answers, they have not measured.
Three designs that did not survive contact.
Every retrieval-precision design we tried lost more than it gained, for a structural reason: a real system must guess relevance without knowing the answer, and its errors fall on the 1,317 questions it already answers — a pool nine times larger than the 144 it could win.
Where the number can and cannot go.
An oracle probe — answering the hardest persistent failures from their gold evidence alone — measures the ceiling of this frozen protocol at ~94.9: beyond it lie defective gold labels, judge strictness, and the answerer's own reasoning limits. Swapping only the answering model for a current one (same memories, same retrieval, same judge) scores 89.68 — a disclosed, non-comparable row that locates the remaining bottleneck in the 2025-era answer model, not the memory.
Check our work.
The engine is open source (AGPLv3). The harness, judge configuration, run artifacts, and per-question ledgers behind every figure on this page are published for verification.