Near-Zero Hallucination Costs Half Your Answers

“Near-zero hallucination” is the kind of claim that should prompt a second question, and the second question is almost never asked: near-zero out of how many attempts?

A retrieval system can drive its hallucination rate arbitrarily close to zero by answering less. Refuse everything and the rate is exactly zero. So the number on its own says nothing — it is one half of a trade, and the other half is coverage.

There is a documented build that measures both, on a 200-question set at ten million vectors, and publishes the losing side of the trade as well as the winning one. That makes it worth reading closely, because the numbers say something more useful than the headline does.

The two numbers that belong together

The evaluation set is deliberately balanced: 100 answerable questions, and 100 that cannot be answered from the corpus. The unanswerable half includes hand-built false-premise questions — which year Einstein won his second Nobel, which language Newton invented in 1700. Questions with a confident shape and no possible answer.

The results land as a two-by-two:

The bottom-left cell is the hallucination: two questions out of a hundred that the system should have refused and instead answered. Two percent, on a set built specifically to trap it. That is a genuinely good result, and a plain RAG pipeline with no verification would light that cell up.

Now read the top row, which is where the bill is. Of the hundred questions the corpus could answer, the system answered 46 and stayed silent on 54.

Coverage is 0.46. More than half the answerable questions came back as “I do not have enough evidence.”

On the answers it does give, quality holds up — 0.908 faithfulness, 0.97 context recall. The evidence is there and the answers stay inside it. But those figures describe a system that has already thrown away half its opportunities to be wrong.

Neither number is meaningful alone. 2% hallucination at 46% coverage is one system. 2% at 90% coverage would be a different and far harder system. 2% at 5% coverage would be a system that mostly says nothing. If a vendor quotes you a hallucination rate without a coverage figure beside it, the missing number is the one you need.

The threshold is a product decision

What makes this honest rather than merely careful is that the trade is exposed as a dial rather than baked in. The build sweeps an abstention threshold and draws a risk-coverage curve, then picks the point that holds hallucination under a stated budget while answering as much as possible.

That reframing matters more than the specific values. Where you sit on that curve is not a model property and not an engineering achievement. It is a decision about what a wrong answer costs in your domain, and it belongs to whoever owns that cost.

A support bot answering billing questions can live at high coverage: a wrong answer produces an annoyed follow-up. A system summarising clinical guidance or a compliance position cannot: a wrong answer produces an incident. Same pipeline, same code, different point on the same curve — and the only way to have that conversation is to have drawn the curve first.

This is the part that generalises beyond RAG. Any component that can decline to act has this dial, and most teams ship one without knowing where the dial is set.

The judge is the ceiling

Everything above rests on one component. The pipeline splits each answer into atomic claims and checks each against its cited text with a faithfulness verifier. Claims that fail bring down the answer; enough failures trigger an abstention.

So the obvious question is how good that verifier is — and the build asks it, which is rarer than it should be. Tested on its own against a human-labelled set of faithful and hallucinated answers, the verifier scores AUROC 0.702.

That is meaningfully better than a coin flip and a long way from reliable. And it is the ceiling on everything else: an unverified verifier does not remove hallucination, it relocates it from the answer into the scorecard. Your system now confidently reports that it is grounded, using a judge that is wrong a substantial fraction of the time.

The build states this plainly and names a stronger verifier as the highest-value next step, which is the correct read. But the transferable lesson is the step itself: the judge needs its own test set, scored independently, before you trust anything downstream of it.

This is the same argument as treating an eval as a truth mechanism rather than a grade. A judge you have not measured is not a quality gate — it is an opinion with a number attached. When I wrote about building an autonomous QA cycle with a custom LLM judge, the judge was the component I was least willing to take on faith, and 0.702 is a good illustration of why.

What actually scales, and what does not

The scale claim gets tested properly: a real approximate index built at 100k, 1M, and 10M vectors, with build time, disk, and latency measured at each step.

A hundredfold increase in corpus size costs less than a doubling in query latency. That is the expected shape for an approximate index — it searches a fixed number of partitions rather than the whole space, so query cost tracks partition count, not vector count. Disk grows linearly, which is fine, because disk is the cheap axis.

One caveat the build states and I will repeat, because it is the kind of thing that gets dropped in summaries: the scale run uses synthetic random vectors. Recall sits near 0.1 in that run, which sounds alarming and is not — random vectors give an approximate index nothing real to find. The run measures latency and disk against corpus size. It does not measure retrieval quality, and it is not evidence that recall holds at ten million.

Then the part worth pinning to the wall. Per-stage timing on real queries:

A typical question takes about 4 seconds; the slow tail reaches nearly 18. And the vector search — the thing the whole scale section is about — is 18 milliseconds inside a 4-second budget.

Retrieval dominates the total, but not because of the index. It dominates because that stage runs an embedder, two searches, and a cross-encoder reranking 150 candidates, sometimes more than once when the corrective loop fires. Those are model calls.

The index is not the bottleneck. The models around it are. That is worth knowing before anyone spends a quarter migrating vector stores: the wins live in cutting model calls, batching the reranker, and caching intermediate grades — not in a faster database. It is a familiar shape to anyone who has profiled a slow test suite and found the time was in setup, not in the assertions.

Where this leaves RAG

I have argued before that retrieval alone is not enough — that vector search returns similar fragments rather than authoritative ones, and that agents need a structured trust layer above it. This build is the other side of that argument, and it strengthens it rather than answering it.

Look at what it takes to make plain retrieval trustworthy: hybrid dense-plus-sparse indexing, rank fusion, cross-encoder reranking, query decomposition, per-sentence citation, atomic claim extraction, a faithfulness judge, a calibrated abstention threshold, and a corrective re-retrieval loop. Nine components stacked on top of “search the documents.”

And after all of that: 2% hallucination, 46% coverage, a judge at 0.702, and four seconds a question.

That is not a criticism of the build — it is a careful piece of engineering and the honesty about coverage and AUROC is the best thing in it. It is an observation about the shape of the problem. The elaborate machinery exists to compensate for retrieval not knowing which of two similar passages is authoritative. Every component in that list is, in some form, a way of asking “but is this one actually true?” after the fact.

Structure in the knowledge layer answers that question earlier and more cheaply than a verification gate answers it later. The stack above is what it costs to answer it late.

What to take from it

Three things, in order of how often they get skipped:

1. Never quote a hallucination rate without a coverage figure. They are one measurement. A system that abstains more will always look safer, and looking safer is not the same as being more useful.
2. Test your judge separately. If a verification gate decides what ships, its own accuracy is the ceiling on everything behind it. An untested judge moves the failure rather than removing it.
3. Profile before you optimise the database. In a pipeline like this, the vector store is single-digit milliseconds inside a multi-second budget. The expensive part is every model call you make around it.

And the framing underneath all three: abstention is a feature with a price tag. Decide the price deliberately, in the domain where the wrong answer actually costs something, and write the number down where the next person can find it.