Every team that ships an LLM feature arrives at the same fork within about a month. The demo works, someone asks how you know it still works, and you discover you have no answer that survives a follow-up question.
Two roads out. Install an evaluation framework and get a dozen metrics in an afternoon. Or build a judge that understands your domain, which takes weeks and creates a component you now have to maintain and, more awkwardly, validate.
The advice you find is mostly one road or the other, argued as if it were a principle. It is not a principle. It is a scoping decision with a fairly clean rule, and getting it wrong in either direction is expensive in a different way.
What the frameworks actually give you
The generic metrics are genuinely good, and they are good because the failures they measure are shared across almost every LLM application.
For a retrieval pipeline: did retrieval surface the right documents, did the answer use them rather than the model’s training data, is the answer relevant to what was asked. For an agent: did it pick the right tool, did it answer relevantly, did it invent information it did not have.
That list is worth internalising even if you build everything yourself, because it is a decent enumeration of how these systems fail generically. Getting it measured on day one, with an install and a test file, is the correct first move. There is no argument for hand-rolling faithfulness scoring before you have looked at what a standard implementation says about your system.
What matters is what the list does not contain: anything about your product.
The line
Here is the rule, and it holds up better than “framework good” or “framework insufficient”:
Generic metrics measure whether the system behaved like a competent language model. Your judge measures whether it behaved like your product.
An off-the-shelf faithfulness metric can tell you an answer was grounded in the retrieved passages. It cannot tell you the answer quoted a deprecated pricing tier, contradicted a policy you published last quarter, used the wrong term for a concept your customers care about, or gave advice that is technically supported by the documents and wrong for that customer’s plan.
Those failures are not on anyone’s metric list, because they are not shared. They are the reason your product exists.
So the split is not build-versus-buy at all:
| Use the framework | Build the judge | |
|---|---|---|
| Measures | Failures every LLM app shares | Failures specific to your domain |
| Examples | Faithfulness, relevance, tool selection, context recall | Policy compliance, correct terminology, tier-appropriate advice, regulated phrasing |
| Cost to start | An afternoon | Weeks |
| Cost to maintain | Version upgrades | Ongoing, and it needs its own test set |
| Skip it when | Never, it is the floor | Your failures genuinely are the generic ones |
Most teams need both, and the sequencing is not symmetric. Start with the framework, always. Then look at what it did not catch, and that residue is the specification for your judge. Building a domain judge first is how teams end up maintaining a custom implementation of faithfulness scoring that a library already had.
This is the same shape as test frameworks generally. Nobody writes their own assertion library. Everybody writes their own domain assertions on top of one.
The part both roads owe you
Whichever you pick, the same obligation follows, and it is the step almost everyone skips.
Your judge has an accuracy, and you do not know it until you measure it.
This is not hypothetical. In a documented RAG build, the faithfulness verifier gating every answer scored AUROC 0.702 on a human-labelled set: clearly better than chance, clearly not reliable, and the honest ceiling on every number that depended on it. That build measured its judge and published the figure, which is rarer than it should be.
An unmeasured judge does not remove uncertainty from your system. It relocates it, out of a place where you were worried about it and into a dashboard where you are not. You now have a number that is confidently wrong at an unknown rate.
The obligation applies equally to framework metrics. “Faithfulness: 0.91” is a model’s opinion about your output, produced by a judge whose agreement with human labellers on your data is unknown until someone checks. A library being popular does not transfer its calibration to your domain.
The practical version is smaller than it sounds. Assemble fifty examples your team has hand-labelled good and bad, run the judge over them, report how often it agrees. Fifty is enough to separate “useful” from “coin flip”, and that is the distinction that matters most.
The eval set is the real artefact
There is a second-order effect worth naming, because it decides whether any of the above is worth doing.
Generic metrics arrive with generic test data, and generic test data measures generic behaviour. The failures you care about will not appear in it, not because the data is bad but because your failure modes are by definition the ones nobody else’s benchmark contains.
So the metrics are interchangeable and the framework is replaceable, but a curated set of cases representing how your system fails is the thing you cannot buy and cannot regenerate. It accumulates. Every production incident should end with one more case in it.
That is the argument for treating an eval as a truth mechanism rather than a grade. A score from a framework on its own data is a grade. A score from your judge on cases drawn from your own incidents is a measurement of something you care about.
A sequence that works
- Install the framework and measure the generic failures this week. Faithfulness, relevance, retrieval quality, tool selection. Do not skip it because you suspect you will need something custom; you need the floor first, and you need to know what it already covers.
- Collect what it missed. Every complaint, every incident, every “that answer was technically fine and wrong for us”. This is the requirements document for step three and it writes itself if you keep it.
- Build a judge only for the residue. Narrow scope, your terminology, your policies. The narrower it is, the easier it is to validate.
- Score the judge before you trust it. Fifty hand-labelled cases, agreement reported as a number, rerun whenever the judge changes.
- Grow the case set from production, not from imagination. One case per incident. This is the part that compounds.
Our own autonomous QA cycle took this route, and the judge was the piece that took longest. Not to build, but to trust. That order is the right way round: a domain judge is a week’s work, and establishing that it agrees with a human often enough to gate anything is where the real effort goes.
The version to remember: reach for the framework first because the generic failures are real and cheap to catch, build a judge only for what is left, and put a number on whichever one you are relying on. A quality gate whose own accuracy is unknown is not a gate.