Give an agent memory and you inherit a multi-tenant problem you did not ask for. Facts about Olena must not reach Petro. Everyone writes the test:
context = memory.context_for("olena", question)
assert "Petro's address" not in [f["text"] for f in context.facts]Green. Ship it.
That test passes on an implementation that returns an empty list for every question ever asked. It passes on an implementation that lost Olena’s facts too. It passes on memory that does not work at all.
A test that asserts something did not happen is satisfied most easily by nothing happening.
This is not a clever edge case. It is the default shape of every security test, every filter test, every “must not” in your suite — and it took three separate discoveries on one feature before I stopped assuming I had it covered.
The failure that has no symptom
Retrieving from memory is ranked search. Score every candidate fact against the question, keep the ones above a threshold, cap the count. Access control is a filter. The only question is where the filter goes.
Put it first:
mine = [f for f in all_facts() if f.owner == owner]
ranked = rank(question, mine)
return top(ranked, limit=3)Put it last, which reads just as naturally:
ranked = rank(question, all_facts())
top_three = top(ranked, limit=3)
return [f for f in top_three if f.owner == owner]Both return only the owner’s facts. Neither leaks. The second one is broken.
Petro has three delivery addresses stored, phrased slightly closer to the question than Olena’s. All three outrank hers. They fill the three slots. Then the owner filter removes them, and Olena’s memory answers her own question with nothing.
Consider what this looks like in production. No exception. No warning. No leak — an audit finds the isolation perfect, because it is. The agent simply does not know where to deliver Olena’s order, on some questions, when someone else’s data happens to rank higher. It is not even deterministic across users: it depends on what other people stored.
And the isolation test is green, because Petro’s address is genuinely not in the output. Nothing is.
The mirror half
The fix to the test is one line, and it is the whole point:
assert context.facts, "the owner's own fact vanished"
assert "Khreshchatyk" in context.facts[0]["text"]Assert that the permitted thing arrived, not only that the forbidden thing did not. Two claims, not one. The second never follows from the first and never appears on its own.
I know it never appears on its own because I have now watched it not appear four times, on four different features, each time reviewed by someone who had not written the code:
- A retrieval filter, where “the internal document was not returned” was green on an empty result set.
- A confirmation gate for irreversible actions, where “the tool did not run without approval” was green on a gate that blocked everything, approval included.
- A validation step, where “the bad step did not execute” was green on a run that had silently stopped.
- This one.
Four features, one shape. In every case the author wrote the half that names the danger, because the danger is what he was thinking about. The half that says the system still works is boring, so it went unwritten — and it is the half that fails.
If you take one thing from this: grep your test suite for assert not and assertNotIn and to.not. and read every hit. For each, ask what the function would have to return for that assertion to pass while the feature is broken. Usually the answer is “nothing”. Usually “nothing” is reachable.
The test that could not fail
Two reviewers read the finished feature in clean context, having seen none of the work. Twenty-seven findings. The most expensive one cost no code at all — it came from asking of each test a single question: what would have to break for this to go red?
Four times the answer was: nothing.
The worst of the four was the isolation test itself. Its fixture stored the other owner’s
address as Доставляти на Банкову 11, and both assertions searched for the substring
Банкова. Ukrainian declines nouns: the accusative in the text is Банкову, and the
nominative the assertion looked for never appears in it at all. So not any(...) was true
unconditionally — against a correct implementation, against a broken one, against memory
with no owner filter whatsoever.
It had been green since the day it was written, and it would have stayed green forever.
If your codebase has non-English fixture data, this class of bug is waiting for you and it
is invisible to review-by-reading. assert "Munich" not in result passes trivially when
the data says München; so does every plural, every possessive, every case ending. And
unlike a wrong assertion, a vacuous one produces no signal — ever.
The fix is one line, and it belongs above every negative assertion:
assert any("Банков" in f.text for f in stored), "the fixture has no foreign fact"
assert not any("Банков" in t for t in texts), textsProve the fixture can produce a match before asserting it does not.
Then the mirror test was also wrong
Here is where it stops being a tidy lesson.
I wrote the mirror assertion. I wrote a fixture for it: five of Petro’s facts, one of Olena’s, and a comment saying Petro’s were “more relevant by phrasing” so they would take the slots. I ran the broken implementation and the test went red. I ran the fixed one and it went green. Done, apparently.
It was passing for the wrong reason.
Petro’s facts and Olena’s fact scored identically — same two words overlapping the question, same number. My comment was simply false. Olena’s fact survived the top-three cut for one reason: Python’s sorted is stable, and her record happened to be inserted last, so the ties resolved in insertion order and she landed in the third slot.
Swap two lines in the fixture — insert Olena first — and the test goes green on the broken implementation. No assertion changes. Nothing in the test looks different. It just stops testing anything.
A test resting on tie-breaking order is not testing relevance. It is testing that I wrote the fixture in a particular sequence, which is exactly the thing no one reviews.
How it surfaced
Not from a red test. Reds were all correct.
I was writing a worked example for readers — three memory implementations side by side on the same data, so you can see the difference in numbers rather than in prose. Store-everything, filter-after, filter-before. The middle one printed one fact.
It was supposed to print none. That is the entire point of the example.
So the demonstration of the bug was more sensitive to the bug than the test for the bug. The test had enough slack that stable sorting closed the gap; the demonstration had none, because a demonstration that shows nothing wrong is visibly useless while a test that proves nothing looks exactly like a test that proves something.
The fix: give Petro’s facts genuinely higher overlap with the question — 0.75 against 0.50. Now the claim rests on relevance, which is the property, instead of on insertion order, which is an accident.
The tool that lies the same way
I run mutation testing on this repository: break the code on purpose, count which tests go red, and pin those counts in a file so a test that quietly stops catching its mutation shows up as a mismatch.
The owner-filter mutation reported zero reds on its first run.
The obvious reading is that the test is worthless. The actual reading was different. My mutation had left the candidate list already filtered by owner before reordering the selection — so foreign facts never took slots and the defect never reproduced. The mutation was broken, not the test.
The instrument you use to check whether your tests are lying can lie in precisely the same way, and it has the same tell: it produces the comfortable result. A green test says “fine”. A mutation catching nothing says “your suite has a hole”. Both are conclusions you can reach without doing the work, and both are wrong roughly as often as they are right.
Zero reds is a question about the mutation first. Only after the mutation is proven to break the property does zero reds become a verdict on the suite.
The same run later reported two reds where I expected one. The second was the module’s line-count budget — the mutation added two lines, the budget check noticed, and it had nothing whatsoever to say about owner isolation. That is worse than noise. “Caught twice” reads as stronger evidence, and half of it was a file-size assertion. Now the size check refuses to measure a module the mutation harness has deliberately broken, and reports “not verified” instead of a red that means nothing.
Where else this hides
Once you have the shape, memory is full of it.
Summarising a conversation twice. Old messages get compressed into a summary; the window overflows again, and the naive implementation compresses “everything outside the window” — which now includes the previous summary. Each pass loses detail. There is no error, no warning, and the text stays perfectly coherent. It simply, gradually, stops being true. What test fails? Not “a summary exists”. Not “the summary is non-empty”. Only an assertion that the summary contains traces of both compressions catches it.
Storing everything. The most common memory implementation keeps every user utterance. It works for a day. Then retrieval returns four facts about the same thing, contradictions accumulate faster than facts, and answers get slightly worse — then slightly worse again. No exception, no failing test, no log line. The field calls it context rot; the operational description is that your agent degrades on a timescale longer than your test suite’s attention span.
The context limit is on tokens. There is no limit on nonsense. You are the limit.
Expiry checked at write time. Delete the fact when it goes stale and the code is simpler and the history is gone, so the question “what did it say before?” has no answer at all. Checking expiry at read time costs a comparison and keeps the record. Every deletion is a question you have decided nobody will ask.
What twenty-seven findings looked like
Worth naming the distribution, because it is not what I expected. Eleven were real code defects. Seven were tests without teeth. Six were prose that had drifted from the code. Three were deferred with an owner and a date.
Two of the eleven are worth repeating because they are not memory-specific:
A record validated with a default and constructed without one. The status field was
checked as data.get("status", ACTIVE) and then built as data.get("status"). A record
missing that field passed validation and arrived as None — permanently inactive, never
reported as corrupt, silently absent from every answer. Two lines, twenty characters apart,
disagreeing about a default.
A Unicode line separator split a JSON Lines record in half. json.dumps does not escape
U+2028; str.splitlines() treats it as a line break. One of those characters inside a fact
— routine in text pasted out of a PDF, and facts are written by users — turned one record
into two malformed halves and the fact disappeared from both. Then the next write rewrote
the file from the records it could parse, destroying the evidence it had just reported.
Neither has a symptom. Both are one-line fixes. Both were found by someone reading the code with no memory of having written it.
What I do now
Three questions, applied to any test that asserts an absence:
What would make this fail? Not “what does it catch” — what specific value or state turns it red. If the honest answer is “an exception”, it is not testing behaviour.
Does empty pass? Run the function against a stub that returns nothing. A surprising number of “must not” tests are green against a return [].
Does the fixture carry the claim, or the ordering? If two elements tie on the property being tested, the test is measuring your insertion order. Make the difference real and measurable, then assert on the measurement.
None of this is sophisticated. All of it is the difference between a suite that reports on your code and a suite that reports on your intentions.
The code, the mirrored checks, and the eleven mutation exercises are in stage 5 of the course repository — pinned to the tag, so the code you read is the code this describes. The three-way comparison that caught the bad fixture is solutions/exercise_2_context_rot.py. It runs offline, without an API key.