A LongMemEval-S number you can reproduce

We held off on posting a benchmark for a long time. Not because we didn’t have runs - because most memory benchmarks you read are a number with no way to check it. A blog says “X%”, and you have no idea what reader answered the questions, what judge scored them, how much context the retriever was allowed to feed, or whether an LLM quietly did the hard part inside the “memory” layer. So the number tells you almost nothing about the memory system.

Here is one we’re comfortable standing behind, because you can run it yourself.

The result

On LongMemEval-S, the full 500-question set, Engrava 0.6.0 scored 81.6% micro in August 2026 - 81.76% averaged across the six question categories. The run uses the canonical LongMemEval scorer (pinned to a known upstream commit), the standard gpt-4o-2024-08-06 reader and judge over the OpenAI API, and a top_k of 20 retrieved turns. Nothing about the reader, the prompt, or the scorer is ours; the only thing we swapped in is the memory.

It is compared against the previous release: 0.5.0, run in July 2026, scored 82.4% micro / 82.58% macro on the same 500 questions, same reader, same judge, same scorer, same top_k. Both rows are on the leaderboard, both verified, and both ship their reproduction artifacts. We are leading with 0.6.0 because that is the version this post is about; the older row stays because removing it when the number goes down is exactly the move that makes benchmark pages worthless.

	0.5.0 (2026-07-10)	0.6.0 (2026-08-11)
micro	82.4%	81.6%
macro	82.58%	81.76%
n	500	500

Both figures are dated on purpose. This post is a record of two specific runs, not a running scoreboard; the current table, whatever version is newest when you read this, lives on the Engrava benchmarks page.

The run also has no LLM in the memory pipeline. Ingestion and retrieval are deterministic - hybrid search over a typed graph, no model doing extraction, summarization, or re-ranking behind the curtain. In the benchmark’s own terms this is a Group A run: memory_pipeline_llms: []. So whatever the score reflects, it is not a second language model inside the memory layer doing part of the work - and not one you’d have to pay for on every write.

That last part is a cost property, not just an architectural one. A memory layer that calls a generative model on every write - to decide what to store, to summarize it, to re-rank it on read - pays for that model on every operation, and the bill scales with how much the agent remembers. Engrava’s ingest and retrieval are deterministic, so writing and reading memory doesn’t spend generative-LLM tokens. It isn’t free of model calls entirely - vector search needs an embedding at write time - but that’s a cheap, pluggable embedder you can run fully local, not a generative model doing the expensive work on every operation.

So it’s a retrieval-quality result in the sense that retrieval is the only part we swapped: the measurement runs end to end through a fixed reader and a fixed judge, and holding those constant is what makes two runs comparable. It does not make the score ours alone.

The gap between the two rows is four questions out of five hundred. The next section is about what we can and cannot say about those four.

About those four questions

The obvious question about two runs four questions apart is whether the newer version got worse. We went and looked at both runs’ artifacts rather than guessing. The artifacts support neither “it regressed” nor “it’s just noise”:

Between the two runs, engrava handed the reader identical context on 457 of the 500 questions, and on the 43 where the retrieved context differed at all, not one answer changed. Every one of the 32 questions whose outcome moved - 18 down, 14 up, netting the four-question difference - received byte-identical retrieved context in both runs, same passages in the same order. Those flips therefore cannot be attributed to the memory layer: with the input to the reader unchanged, what varied was the reader and judge themselves, which are not deterministic even at temperature zero. We are not claiming the two versions are equivalent, and we have not run a replicate of this configuration, so we have no measured variance for the score itself and will not invent a confidence interval. What we can say precisely is narrower and stronger: this difference is not something engrava’s retrieval did.

Two things that paragraph deliberately does not say, and we won’t say them either. It does not call the difference noise - that would be a claim about measurement variance, and measuring that needs replicate runs we have not paid for. And the churn underneath the four-question net - 32 individual outcomes moving, 6.4% of the set - is a measurement from these two runs. It is not an estimate of how much a score wobbles between runs, and we won’t present it as one.

Where the number comes from

Engrava isn’t a vector index with a graph bolted on. The pieces that move a score like this are the same ones in the free package: a typed knowledge graph (thoughts as nodes, seven edge types between them), hybrid search that fuses vector similarity, BM25 over the text, and recency in one query, and turn-level granularity so the retriever can land on the exact user turn a question depends on rather than a blurry session average.

Those live alongside the rest of what Engrava ships, in one embedded SQLite store. The benchmark exercises the retrieval slice and nothing else. What it shows is that the retrieval half of a local, no-server memory layer holds up on a public long-horizon test.

The number is not a result about consolidation. Engrava’s background consolidation (“dreaming”) is deterministic memory hygiene, and it is switched off for this run - so whatever it does or doesn’t do for long-horizon recall, this figure does not measure it and we are not reaching for it here.

Reproduce it

The run lives in a public repo - sovantica/engrava-benchmark, MIT. It isn’t a package you install; it’s a repo you clone and run against the public engrava on PyPI. The machine-readable leaderboard.json in that repo is the number of record; this post just describes two of its rows.

git clone https://github.com/sovantica/engrava-benchmark.git
cd engrava-benchmark
git checkout a45dde9                 # the runner commit this result pins
python -m venv .venv
source .venv/bin/activate
make install
pip install "engrava==0.6.0"        # the exact version the result pins

export OPENAI_API_KEY=...            # reader + judge, OpenAI-direct
export ENGRAVA_BENCH_LONGMEMEVAL_S=<path>/longmemeval_s_cleaned.json  # the cleaned split (see note below)

python runners/longmemeval/run.py    # no flags - the bare command is the canonical run
make validate
make leaderboard

The bare command with no flags is the canonical configuration - the same reader, judge, scorer, and top_k every published number uses. Any flag that overrides a model or endpoint moves the row out of the comparable segment, so it’s no longer the headline. If you want to check the wiring before spending anything on API calls, python runners/longmemeval/run.py --smoke runs real Engrava retrieval against a local embedder and a mock reader/judge for free.

Get the dataset right: it is the authors’ cleaned LongMemEval-S release, from Hugging Face xiaowu0162/longmemeval-cleaned (longmemeval_s_cleaned.json) - not the raw longmemeval_s.json. The result row pins that dataset by sha256, so before you run, confirm your file’s hash matches the row’s dataset_revision; a different revision produces a different score.

Every result row pins the axes that move a score - engrava version and distribution hash, runner commit, reader and judge snapshots and endpoints, scorer version, retriever, granularity, top_k - plus a reproduction artifact and its checksum.

The honest limits

The two rows sit in separate comparability segments, and exactly one axis separates them: the harness commit. Dataset revision by hash, reader and judge snapshots, scorer commit, retriever and top_k are identical. That is why they are worth putting side by side, and why the board still keeps them apart - a number is comparable only within a segment matching on all of those axes, harness included, and different segments are never merged into one ranked table. Both sit in the canonical gpt-4o-2024-08-06 reader and judge; a score produced with a different reader isn’t rank-comparable to either, and we don’t present it that way. This is one retrieval benchmark on one dataset - a real signal about long-horizon recall, not a universal statement about every workload. And it’s the retrieval slice specifically: the structural guarantees Engrava also ships (the hash-linked journal, typed edges, MindQL) are their own thing, verifiable in their own right, and not what this figure measures.

If you’re evaluating agent memory, don’t take the percentage on faith - run the command above, then swap in whatever else you’re weighing by writing one adapter, and read both numbers off the same reader and scorer. That’s the comparison that actually tells you something.

pip install engrava

• Repo: github.com/sovantica/engrava
• Benchmark: github.com/sovantica/engrava-benchmark