The Detector That Always Finds It Is Not a Detector

You build an evaluation harness for an agent. You ask a model to grade the answers. The report comes back: 76% pass. You ship.

The number is worthless, and not for the reason people usually give. The problem is not that the judge is a language model. The problem is that nothing in your harness distinguishes a judge worth listening to from one that is broken — so the 76% describes your judge at least as much as it describes your agent.

A model judge is a measuring instrument, and instruments get calibrated.

An evaluation that declares the judge’s verdict to be truth has made exactly the mistake the whole exercise exists to prevent: it trusted a number without asking where it came from.

Below is what calibration actually costs, the shape of the check that makes a bias detector mean something, and the defect an independent review found in the level I was most confident about.

Working code: stages/s08_eval/ at tag stage-08. 31 checks, 15 of them on failure modes. Runs offline, no API key, and every number below comes from that tag.

Position bias, and the half that makes it a measurement

Ask a judge to compare two answers. Then ask again with the answers swapped. A judge that keeps its verdict is stable; a judge that flips preferred the position, not the content.

Run that on a deliberately biased judge and you get:

position bias: FOUND — 3 of 3

Three flips out of three pairs. Convincing. And on its own it proves nothing at all, because a detector that reports bias on every input cannot tell a biased judge from an honest one. It is not measuring the judge; it is measuring its own eagerness.

The number that makes the first one mean something is the second:

position bias: agreement — 0 of 3        (steady judge)

Same three pairs, same detector, a judge that does not play favourites. Zero flips. The mirrored half is not decoration — it is the condition under which the first half says anything.

The pairs are weighted on purpose: both answers are equally grounded and the same length, so the only thing left to prefer is the slot. If one answer is better, a flip is a judgement, not a bias, and the detector is back to measuring nothing.

Length bias runs the same shape:

length bias: FOUND — 2 of 2
   when does delivery arrive:   3 -> 5   (+2 for 81 extra characters)
   how do I return an item:     4 -> 6   (+2 for 78 extra characters)

Padding that adds no information adds two points. Same content, more characters, better grade.

The fake judge is biased deliberately. It does not imitate any particular model — it plays the role of a broken instrument, the way a mutation plays the role of broken code. What it does wrong is written in its docstring rather than hidden: the first answer shown gets a free point, and every forty characters add another regardless of content. Point the same detector at a real model with a real key and it produces the same report.

Three levels, and the rule about what each may blame

One verdict per case is a category error. “The agent failed” can mean the final text was wrong, or the path was wrong, or a tool returned nonsense — three different problems with three different fixes, collapsed into one word.

So there are three levels, and each is only allowed to blame its own layer:

• end-to-end — the final answer, and nothing else. Not the route, not the tools.
• trajectory — the path: which tools, in what order, within what step budget.
• component — one step in isolation: did this tool return something usable.

The attribution rule matters more than the levels. An end-to-end failure caused by a tool that timed out is a component failure with an end-to-end symptom, and a report that files it under “the answer was wrong” sends you to rewrite a prompt that was never the problem.

The level that graded an answer it had never seen

I wrote all three levels, wrote 31 checks against them, ran a mutation pass, and called the stage done. An independent review then read the code with fresh eyes and found the end-to-end level had never once looked at the trajectory.

def e2e(case: Case, judge: Judge) -> Verdict:
    scored = judge.score(case.task, case.answer, case.expected_answer)

case.answer is the answer stored in the case. The trajectory — what the agent actually did on this run — is not an argument. So an agent whose run produced no answer at all, whose trace ends in a timeout, still gets graded: the judge scores the text from the case file and reports passed, score 3.

This is a tautology with two names. Both sides of the comparison come from the same source, so the check cannot fail for the reason it claims to test. It looked like an evaluation and behaved like a lookup.

The fix is one argument and one guard:

def e2e(case: Case, trajectory: Trajectory, judge: Judge) -> Verdict:
    said = trajectory.answer()
    if said is None:
        return Verdict(E2E, UNSCORED, JUDGED, "no answer in the trace")

The consequence is visible in the report, which is how you know it was real:

Two cases moved out of “failed” and into “unscored”. They were never failures; they were runs with nothing to grade, and the old code had been grading the case file instead.

I recommend the specific habit that catches this class: for every equality in a test, ask where each side comes from. If the answer is “the same place”, the test is furniture.

Unscored is a state, not a rounding error

Two failures hide in the gap between passed and failed.

The first is silence read as success. A trace with no step of the required kind does not prove the step went well — it proves nothing. Counting it as a pass produces a report that improves as instrumentation degrades.

The second is a shrinking denominator. If the pass rate is computed over scored cases, every case the judge could not grade quietly leaves the denominator, and the percentage goes up. That is the worst kind of bad news: it arrives disguised as good news.

So unscored is its own column, and the denominator is all cases. A judge that is rate-limited, over budget, timed out or returning unparseable text produces unscored, never failed — and the report says how many.

The parser is strict for the same reason. An earlier version pulled every digit out of the judge’s reply and took the first two characters, which turns “I would rate this 3 out of 10” into a 31. It now requires the reply to be a bare number, and anything else is unscored with the raw text attached.

What the traces did not carry — measured, not assumed

The harness reads traces the earlier stages already write. Which means it can answer a question about the course itself: how many stages emit a key that lets you group steps into a run?

Rather than assume, the harness parses the earlier stages’ source and counts:

fields carried in traces:  3
stages with no run key:    4

Four of the earlier stages wrote traces you cannot group by run. That is not a criticism of a design decision — it is a decision nobody made, discovered by looking. Evaluation is the first consumer of tracing, and the first consumer is where you learn what the producer forgot.

Sampling, when you cannot judge everything

Judging every production request costs money and adds latency, so the harness samples. The interesting part is not the rate; it is what the check asserts about it.

sample over 1000 requests: 106 sent to the judge = 10.6%

A sampler that hits exactly 100 of 1000 is not sampling, it is scheduling — and a scheduled sample is trivially gamed by request order. The check therefore requires the count to be near the target and not exactly it, and it requires the same seed to reproduce the same set. Reproducibility and randomness are not in tension; skipping either one is what makes a sample unusable.

What this deliberately does not prove

• Twenty-one cases are not statistics. There are no confidence intervals here, and pretending otherwise would be worse than not computing them.
• A biased fake judge does not prove real judges are biased. That is established elsewhere. It gives the detector something to detect.
• Sampling is verified in-process. A real deployment stays NOT EVALUATED rather than green.
• With a real judge, determinism is not guaranteed. The flakiness check turns into NOT EVALUATED when a key is present, which is the honest state.
• There is no drift over time. Drift needs stored history, which this stage deliberately does not keep. It prints the numbers drift is computed from; comparing windows is a different job.

Numbers

21 cases, 9 of them edge cases
31 checks, 15 of them on failure modes
14 mutation exercises, each red in the check that claims it
19 judge calls per run

The harness, the biased fake judge, the mirrored halves and the 14 mutation exercises are in stage 8 of the course repository, pinned to the tag, so the code you read is the code this describes. It runs offline, without an API key.