What our coverage number did not protect

Engrava’s test suite had 3,845 tests and 94.22% line coverage. There was more test code in the repository than production code. By every number we had, the thing was well tested.

Then we ran a mutation audit over it and found that the suite did not protect a broken concurrency guard, several deletion paths, five configuration sections, two of its own safety tests, or a field that would silently discard a legitimate value, in the version people were running. That is nine findings in three groups, and the groups are not alike. The accounting is below.

This post is about how that happens, because the mechanism is more interesting than the individual bugs, and because we would have told you the suite was solid the day before.

Coverage answers a question nobody asked

Line coverage tells you a line executed while the tests ran. It does not tell you that anything would have noticed if the line were wrong.

Those sound close. They are not. A test can execute a guard, assert something true about the result, and stay green after you delete the guard entirely, because the outcome it asserts is produced by something else in the path. The line was covered the whole time. It was never checked.

The instrument for finding this is mutation testing: change the production code deliberately, run the suite, and see whether anything goes red. What survives the mutation is the finding. Not the code you broke, the tests that let you.

We ran that against Engrava before the 0.6.0 release. Here is what it returned, grouped honestly, because the three groups are not the same kind of thing and lumping them together would flatter us in one direction and alarm you in another.

Group one: six guards we hardened, none reachable from outside the process

Six findings were guards that validated a value and then used the caller’s original object rather than the validated one. Validate, then discard the result of validating.

If you have ever written a validator, you know this shape immediately. You check an argument, the check passes, and then the code below reaches for the argument again instead of for what the check produced. If the value can answer differently between those two reads, the check and the use are no longer talking about the same thing.

Six of those were reachable (and here is the part that matters more than the finding) only by a caller already executing code inside the same process. Not through a config file, not through a query string, not through data arriving from anywhere. Every one needed code running next to ours, deliberately handing the library something built to behave inconsistently.

We checked that against the published release itself rather than against our own descriptions of what we had fixed, which is a different claim. The answer was the same for all six: none of them crossed a trust boundary in any version that was ever on PyPI. No advisory is owed, no CVE, no patch to the 0.5 line.

The framing we ended up with, and it is the honest one: this is the absence of a boundary, not a hole in one. Someone executing arbitrary code in your process can replace the function these guards protect, or the standard library call underneath it. Defending against that specific caller is not a thing a library can do, and a test asserting we had would be theatre.

So why harden them at all? Because defence in depth is worth having when it is cheap, because the mechanism generalizes to places where the caller is less trusted, and because a guard that can be talked out of its own conclusion is wrong on its own terms even when nothing can currently exploit it. We fixed all six. We are not going to describe them in language that suggests you were exposed, because you were not.

Group two: two safety tests that proved nothing

This is the group that unsettled us most.

Two tests in the hygiene suite (the code path that decides what memory gets archived and eventually deleted) passed over guards that were working correctly. They were not detecting the guards. They were detecting something else in the path that happened to produce the same outcome.

One of them had a docstring stating, in plain English, that a specific pin was what caused the row to be skipped. That was not true. A different condition entirely was doing the skipping, and you could remove the pin protection at all three layers and watch the test stay green.

Nobody wrote those tests carelessly. They were written by people who understood the feature, and read by people who understood it too. Reading a test tells you what its author believed. It does not tell you what the test can detect. Only mutating the code it claims to protect tells you that, and neither of us thought to do that until a sweep did it mechanically.

There is a related finding that makes the point sharper. Elsewhere in the codebase, deleting a guarantee outright left 4,316 tests green: the whole suite, not some narrow slice of it. A green suite is not evidence that a guarantee holds. It is evidence that nothing in the suite noticed it was gone.

Group three: the one that could actually lose your data

Nothing above changed what a normal user’s data did: the first group needs code already running inside your process, and the second is about our tests rather than about shipped behaviour. This one is different, and it is the reason this post exists in the form it does.

Engrava’s edges carry a decay_multiplier, a float that is documented as valid from 0.0 upward. Setting it to 0.0 is a legitimate thing to do: it means this edge does not decay by that mechanism.

In released 0.5.x, the code that read that value back from the database tested it for truthiness. 0.0 is falsy. So a stored 0.0 read back as 1.0, the default.

That alone would be a bad-but-recoverable read bug. It got worse on the way out: the update path rebuilds the whole edge record from the values it just read and writes every column back. So the next time anything updates that edge (for an unrelated reason: changing the weight, touching metadata), the misread 1.0 is written over your 0.0. Permanently. No error, no warning, no log line.

No attacker. No unusual configuration. Ordinary correct use of the public, documented API, silently discarding a value the documentation says is valid.

It is fixed in 0.6.0. If you set decay_multiplier=0.0 on any 0.5.x release, check those edges, and note that upgrading does not restore a value that was already overwritten. The 0.5 database has no record of what the number used to be.

We are stating this plainly because a post about our own testing gap that quietly omitted the one defect that could cost someone data would be worth less than not writing it.

What the count actually is

Six defence-in-depth hardenings, all of which require code already running in the process. Two vacuous tests over working guards. One real data-loss defect.

Those nine are the same nine the opening lists: the concurrency guard, the deletion paths and the configuration sections are among the six; the hygiene safety tests are the two; the field that could lose data is the one. We are spelling the arithmetic out because the easy version of this post describes all nine in security language, and that would be a worse error than any of the individual findings: six of them cannot be reached by anyone who is not already running code in your process, and two of them are about our tests rather than about anything that shipped.

What we changed, and what we did not

The suite went from 3,845 tests to 4,316; coverage went from 94.22% to about 95%. The coverage move is the least interesting number on this page, which is the argument closing on itself.

The changes that matter are structural. Guards now use the value the validation produced. Several hand-maintained lists that were supposed to enumerate protected operations are now derived from the code rather than typed out and hoped over. The two vacuous tests are able to fail. And we now require, for a claim about a test, that someone demonstrate the signal going red on the broken version, not that the suite is green on the fixed one.

What we did not do: replace the concurrency mechanism. The specific guard the audit broke is fixed, but the broader design question underneath it (how two processes writing to one file coordinate) needs a schema change, and landing that in a release days from publishing was the wrong trade. It is deferred and written down as deferred, rather than rushed in so this post could have a tidier ending.

If you want to try this on your own suite

You do not need a mutation-testing framework to start. Take a guard you believe is tested: an authorization check, a validation, a deletion safety condition. Delete it. Run your tests.

If they stay green, you have learned something about your suite that no coverage report was ever going to tell you. That is the whole technique. The tooling only makes it systematic.

We found six hardenings, two tests that proved nothing, and one real bug this way, in a codebase we would have described the day before as well tested. We think that says less about Engrava than about what a coverage number is capable of promising.

pip install engrava

• Repo: github.com/sovantica/engrava
• Upgrade notes: github.com/sovantica/engrava/blob/v0.6.0/docs/upgrade.md