The eval literature reads like a new discipline. It is mostly not. If you have written a test suite, you already have the concepts — harness, fixture, assertion, regression run, flake — and the eval world has renamed almost all of them.
That is good news for anyone moving into AI quality work, and it is also a trap. Some of the terms map cleanly and you can translate on sight. Three of them look like they map and do not, and those three are where teams lose months.
Here is the table first, then the places it breaks.
The direct translations
| What you call it | What the eval world calls it | Same thing? |
|---|---|---|
| Test case | Task | Yes |
| Assertion | Grader | Mostly |
| Test runner | Harness | Yes |
| Regression suite | Regression evals | Yes |
| Snapshot | Golden dataset | No |
| Flaky test | Non-determinism | No |
assertEquals | LLM-as-judge | No |
The first four you can read straight across.
A task is a test case: one input, plus what a correct response looks like. A harness is a runner: it sends every task through the model, records what came back, and hands each result to the grader. Regression evals are a regression suite: the whole set replayed after a change, compared against the previous run. Nothing new here except vocabulary, and if a document uses these words you can substitute your own and lose nothing.
A grader is an assertion with one difference worth flagging: an assertion is a line of code and a grader can be a line of code or a model call. When it is code — does this answer cite a URL that exists in our docs, is the tool name in the approved list, is the JSON parseable — it behaves exactly like an assertion and you should prefer it every time it is possible. When the property is subjective, the grader becomes a model, and that is where the third broken row comes in.
Break one: a golden dataset is not a snapshot
A snapshot test freezes an output and byte-diffs future runs against it. Any difference is a failure, and that is the whole point: determinism is the contract.
A golden dataset freezes tasks and what a good answer looks like, and the model will produce different words every run. You cannot diff it. There is no equality check available, which is exactly why a grader exists at all.
The practical consequence is that golden sets are graded, not compared, and a graded set has properties a snapshot never does. It can be partially right. It can drift in quality without any run “failing”. And it needs curation: a snapshot is generated by the system, while a golden set is assembled by humans deciding what good looks like, one case at a time.
That last part is the actual work, and it is the piece you cannot buy from a framework. The metrics are interchangeable. The set of cases representing how your system fails is not.
Break two: non-determinism is not flakiness
This is the one that causes the most damage, because the instinct it triggers is exactly wrong.
A flaky test is a defect. Something in the test or the environment is racing, and the goal is elimination — find the race, wait on a real condition, remove the flake. A suite with flakes is a suite people stop believing.
Non-determinism in a model is a property of the system, not a defect in the test. The same question asked twice produces two different valid answers, by design, because sampling is how these models work. The goal is not to eliminate it. It is to bound it.
Which changes what a passing result even means:
- A deterministic test that passes tells you the behaviour is correct.
- A non-deterministic task that passes once tells you the behaviour is sometimes correct, and you do not know the rate.
So the unit of measurement moves from pass/fail to pass rate, and a single green run stops being evidence. Running each task once and reporting green is the most common mistake teams make in their first eval suite, and it feels correct because it is what a test suite does.
The fix is unglamorous: run tasks multiple times, report the rate, and treat a drop in rate as the signal rather than waiting for a zero. Setting temperature to 0 does not rescue determinism either. It narrows the distribution and the outputs still move, and you have now measured the model in a configuration you may not ship.
Break three: the grader has an accuracy, and the assertion never did
The deepest asymmetry, and the one with no equivalent in ordinary testing.
expect(banner).toHaveText('Invalid card') is correct by construction. There is no world where the assertion itself is 87% right. It compares two strings and the comparison is exact. Nobody has ever had to validate an assertion library before trusting the suite it powers.
An LLM-as-judge grader is a model making a judgement, which means it has a true positive rate, a true negative rate, and an error rate you do not know until somebody measures it. You have introduced a component into your quality gate whose own quality is unknown, and it sits in the one position where its own errors contaminate every number downstream.
The numbers involved are not reassuring by default. A published RAG build measured the faithfulness verifier gating every one of its answers and got AUROC 0.702 against a human-labelled set — better than chance, nowhere near reliable, and the honest ceiling on every downstream metric that depended on it.
So the vocabulary you need here has no QA counterpart at all:
| Term | What it means for you |
|---|---|
| TPR (true positive rate) | Of the answers a human calls good, how many the judge also calls good |
| TNR (true negative rate) | Of the answers a human calls bad, how many the judge also flags |
| Bias correction | Adjusting the reported pass rate using known TPR/TNR, because an imperfect judge miscounts in a predictable direction |
| Confidence interval | Reporting “82–88%” rather than “85%”, because a finite labelled set does not support a point estimate |
| Evaluator drift | The judge silently getting less accurate — the provider updated the model, or your definition of good moved, with no change on your side |
Evaluator drift deserves the most attention because it has no analogue in a normal suite and no alarm attached. Your assertion library does not change behaviour under you. A judge model behind an API can, and when it does, your dashboard moves while your repository stays identical. If you have not re-scored the judge against a fixed labelled set in some months, you do not currently know its accuracy. You know what it was.
Two failure types worth separating
One more distinction that saves time, because the two look the same in a failure report and have completely different fixes.
Specification failure: the model did what you asked and you asked for the wrong thing. The bot never mentions attaching a screenshot when reporting a bug, because nobody told it to. One line in the prompt, gone.
Generalization failure: the instruction was clear and the model still gets it wrong on some inputs, and those inputs do not obviously resemble each other. This one cannot be prompted away — it needs cases, measurement, and a rate you track.
The reason to separate them on sight: specification failures are cheap and satisfying to fix, so teams keep reaching for prompt edits, and a generalization failure edited as if it were a specification failure produces a longer prompt and the same rate. If two prompt revisions have not moved it, it was the other kind.
What to take from this
The translation is real and it means you are not starting from zero. Tasks are test cases, harnesses are runners, regression evals are regression runs, and your instincts about coverage, curation, and maintenance all transfer.
The three breaks are what to actually internalise:
- Golden sets are graded, not diffed — so somebody has to decide what good means, case by case, and that set is the asset.
- Non-determinism is bounded, not eliminated — so the unit is a rate, and one green run is not evidence.
- The grader has an accuracy — so measure it, report it, and re-measure it, because unlike every assertion you have written it can be wrong and it can change on its own.
The rest is vocabulary. Learn the table, use your own words internally, and put the effort into the three rows that do not translate.