Your RAG Leak Test Is Green and Your Access Control Is Broken

A support agent has two kinds of document behind it. Public ones — the return policy, shipping times, product descriptions. Internal ones — the refund threshold an operator can approve without asking a manager, the escalation rules, the margins.

A shopper asks what the automatic refund amount is. Your access filter does its job: the internal document does not reach them. You have a test for exactly this, and it is green.

The shopper gets “nothing found” for a question your public return policy answers on line one.

Nothing leaked. Your test is right about that, and it will stay right while the feature is broken, because it is testing the wrong half of the property. This is the interesting part of RAG access control and it is almost never the part people write tests for.

Below are four ways this goes wrong, all four found in my own code by a review I did not do myself, all four silent. Every number is measured, and you can reproduce each one.

Working code: stages/s02_rag/ at tag stage-02. Runs offline, no API key. The corpus is Ukrainian, so queries below carry a translation — run them verbatim and you get the same scores.

Where the filter goes

Retrieval is four steps: embed the query, score every fragment, sort, take the top k. Access control is one more line. The only question is where you put it, and there are two plausible answers.

# Filter first, then take the top k
allowed = [i for i in range(len(fragments)) if access_of[i] == asker_access]
ranked  = sorted(allowed, key=score, reverse=True)
top     = ranked[:top_k]

# Or take the top k, then filter
ranked  = sorted(range(len(fragments)), key=score, reverse=True)
top      = [i for i in ranked[:top_k] if access_of[i] == asker_access]

Both remove the internal documents. Both pass a leak test. One of them is broken.

Here is the query яка сума автоматичного повернення (“what is the automatic refund amount”) against a small store corpus, scored with no filter at all:

0.554  internal-refund-thresholds#0   [internal]
0.433  internal-refund-thresholds#1   [internal]
0.250  returns-policy#0               [public]
0.144  shipping-policy#1              [public]

The internal document wins on similarity, and it should — it is literally the document that answers the question. The public return policy is third.

Filter first with top_k=2, and the shopper gets returns-policy#0 at 0.250. Correct answer, correct access level.

Filter last with top_k=2, and the two internal fragments occupy both slots, get removed, and the result set is empty. The shopper is told nothing was found. The answer was there. It was displaced by documents they were never allowed to see, and then those documents were dutifully deleted.

No log line records this. No exception is raised. The leak test passes.

The mirror test

The fix in the code is trivial. The fix in your head is the part worth keeping:

A test that the forbidden thing did not happen never substitutes for a test that the permitted thing did.

These are two different claims. Covering one creates a strong, false sense that you have covered both — strong enough that I built the whole stage, wrote the checks, ran a mutation pass, and shipped it without noticing.

So the suite has a second check, and it reads almost silly written down:

def check_permitted_document_is_not_displaced_by_a_filtered_one():
    """The permitted document did not disappear — the filter runs BEFORE selection"""
    result = base.search(INTERNAL_BAIT, access=PUBLIC, top_k=2)
    assert result.hits, "the permitted document vanished from the results"
    assert result.hits[0].fragment.source == "returns-policy"

Move the filter after selection and exactly one check goes red — this one. The leak check stays green. That asymmetry is the whole lesson, and it is why the check exists.

The check that looks brittle is the one that works

Now the part I did not expect.

Run the same broken implementation with top_k=3 instead of 2:

top_k = 3
  filter first  ->  0.250 returns-policy#0
  filter last   ->  0.250 returns-policy#0     ← identical

At top_k=3 the permitted document still fits in the window alongside the two internal fragments, so both orderings produce the same output. The bug is fully present and completely invisible.

The code is equally broken either way. What changes is whether you can see it — and what decides that is top_k, a parameter with no relationship whatsoever to access control.

Which means a check written “the way production runs it”, with top_k=3, would have been green on the broken code. The check pins top_k=2 for exactly that reason.

This inverts a habit most of us have. A test with a hardcoded parameter looks brittle, looks like it is over-fitted to an implementation detail, and is the first thing someone tidies up during a refactor. Here that parameter is load-bearing: it is what makes the property observable at all. So the reason is written down next to the check, because the next person to “make this more realistic” will silently switch the test off.

The same failure again, through a different door

The retrieval function became a tool for an agent. The access level is bound with partial, so the model can ask what to search but never whose documents:

Tool(
    name="search_knowledge_base",
    parameters={
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
        "additionalProperties": False,
    },
    func=partial(search_knowledge_base, access=access),
)

Now remove the partial and pass the bare function. Is that a leak?

No. The function’s default is PUBLIC, so an unbound call sees less, not more. Fail-safe default, working as intended.

And yet an operator has just stopped seeing internal refund thresholds. They will quote the wrong number to a customer, with no indication anything is wrong. Every leak test is green — for the second time in one feature, through a completely different mechanism.

That is what pushed the rule from “good idea” to “write it in the playbook”. The pair of properties needs a pair of checks, every time:

forbidden content did not arrive     ← the test everyone writes
permitted content did arrive         ← the test that catches this class

While we are here, one honest correction to something I wrote in the lesson before the review: partial is not the barrier. Calling tool.func(query=..., access="internal") overrides the binding without ceremony. What actually holds the line is additionalProperties: false plus a validator that rejects an undeclared argument before the function runs. partial is how the system supplies the right value without asking anyone. Two parts, and neither works alone. Confusing them is how “but we have a partial” gets said in a review.

Six silent ways to publish an internal document

The access level comes from document metadata:

---
title: Internal automatic-refund thresholds
access: internal
---

Parsing three lines of that is not where you expect to find a security bug. The original had one line that decided everything:

access = fields.get("access", PUBLIC)   # ← fail-open

Six ways to reach that default were verified, and every one of them is silent:

None of these looks like a security incident. Several are what happens when someone edits a document in a different editor. And the lesson attached to this code tells the reader to add their own document to the corpus by hand, so every path was reachable by the intended workflow.

The fix is four lines and the reasoning is one:

LEVELS = frozenset({PUBLIC, INTERNAL})

def _access(fields):
    declared = fields.get("access", "").lower()
    return declared if declared in LEVELS else INTERNAL

Plus normalising keys and reading with utf-8-sig so a BOM cannot eat the opening fence.

The asymmetry justifies it completely:

losing access to a document    noticed immediately — someone complains
publishing an internal one     noticed never

Fail-open here means the protection works only for as long as nobody makes a typo. There is a version of this in every metadata parser that decides permissions, and it is worth grepping for in yours today.

Who attaches the source

One more, because it rhymes.

Grounded answers need provenance. The obvious implementation is to ask the model for it: cite the document you used. It works in testing, it reads well, and it is wrong for a reason that has nothing to do with prompt quality.

A model asked to cite will occasionally name a document that was never retrieved — and an invented reference looks exactly like a real one. There is no surface signal distinguishing them. The mechanism introduced specifically to separate a grounded answer from a fabricated one becomes a source of fabrication.

So the system attaches it instead, from the list it just retrieved:

return Answer(
    text=model_text,
    sources=[hit.fragment.label for hit in result.hits],   # not from model_text
)

Sources are now correct by construction. Referring to a document that does not exist is not reachable.

Two honest limits go with that, and both belong in the same paragraph as the claim:

A source is guaranteed to exist, not to be what the answer followed from. The model received the fragment and may have answered past it. The demo shows this without hiding it: on a returns question, a shipping document crosses the relevance threshold and formally becomes a source. Measuring whether an answer follows from its sources is a different problem, and pretending otherwise would be worse than the gap.

On the agent path, the guarantee needs a different mechanism. The loop returns the model’s text as-is, so sources are extracted by the system from the tool step transcript — what search actually returned, not what the model wrote. The check feeds the model a deliberately invented citation and asserts it never reaches the sources.

Numbers

49 checks, 24 of them on failure modes
8 breakage exercises, each red in exactly the check that claims it
9 major findings from independent review, in code that already looked done
2 of the 4 access failures leaked nothing at all

That last line is the one I would put on a sticker.

Two more things measured along the way, both cheap and both now permanent:

A mutation harness that greps for failures reports zero when the mutation breaks the import. Mine reported “0 caught” for six mutations in a row because a broken f‑string meant the suite never ran and there were no failure lines to find. The tool for checking whether a test lies had lied in precisely the same way. It counts executed checks now.

One check asserts the lesson’s own numbers against the suite that prints them. The prose claimed “28 checks, 9 on failure modes” while the command printed 29 and 10. A reader following the instruction to run it would have hit the discrepancy on their first command.

Try breaking it

Clone the tag and make the code wrong on purpose — that is the fastest way to internalise any of this.

git clone https://github.com/AZANIR/agentic-ai && cd agentic-ai
git checkout stage-02
pip install -e ".[dev]"
python -m stages.s02_rag.run
python -m stages.s02_rag.check

Then, one at a time:

1. Move the access filter after top-k selection. One check goes red, and it is not the leak check.
2. Run the same mutation with top_k=3. Nothing goes red. Same bug.
3. Drop the partial binding. No leak, and the operator loses their documents.
4. Set the metadata default back to public, then add a document with Access: capitalised.

Each mutation has a measured expected result written down, so you can tell “I broke it correctly” from “I broke something else”.

One environment note that cost me a confused minute: if you change a number to another number of the same length and revert within the same second, Python may reuse a stale .pyc — it compares source mtime at second granularity and file size, and both matched. Clear __pycache__ if a check fails on code you have already reverted.

What this does not cover

This is document-level access control on a small in-memory index. Row-level permissions, per-user document ACLs resolved at query time, and multi-tenant isolation are all harder, and the last one is a different architecture rather than a bigger filter.

It also assumes access levels are known before the search runs. When they must be resolved per request, the sentinel matters more than anything above: None meaning “no filter” is a bug waiting to happen, because None is exactly what an unresolved lookup returns. “Access unknown” and “access unlimited” must never be the same value. In this code full access is a named constant you have to type on purpose.

Whatever your retrieval stack, the check worth adding this week is the boring one:

Assert that the permitted document is still there. Your leak test is not going to tell you when it goes missing.