A refusal is not a hallucination

The safest thing your assistant can do is say "that isn't in the documentation I have." Most faithfulness implementations score that answer as a 100% hallucination. We know, because ours did — until an independent audit of our own harness caught it.

How claim-level faithfulness works

The standard way to measure grounding in 2026: decompose an answer into its individual factual claims, then verify each claim strictly against the retrieved context — supported, unsupported, or contradicted. Faithfulness is the supported share; hallucination rate is its complement.

faithfulness      = supported_claims / total_claims
hallucination_rate = 1 - faithfulness

Now feed it a refusal: "I could not find information about customer-managed encryption keys in the documentation available to me." How many factual claims does that answer make about the world? Zero. And 0 / 0 is where a metric quietly becomes a lie.

The bug

The obvious defensive implementation returns 0.0 for the empty case — no supported claims, so no score. That single line means:

• an honest refusal scores faithfulness 0.00, hallucination rate 1.00;
• a confident, fluent, half-fabricated answer scores 0.50;
• so the metric rates the fabrication twice as trustworthy as the refusal.

Put that number in front of a team with a target to hit and you have built an incentive to remove refusals. The system's hallucination rate improves. The system gets more dangerous. This is not hypothetical — "reduce our hallucination rate" is the brief we are most often handed, and the metric behind it is usually nobody's job to check.

The correct treatment

A zero-claim answer asserts nothing false, so its faithfulness is 1.0 — it is perfectly grounded. Whether it is useful is a different question, and it belongs to a different metric: answer relevance. Keeping those two separate is the entire point.

From a real run in our harness, the refusal above scores:

That is the truthful reading: the generator behaved correctly, and the retrieval pipeline owes this user a document. The fix goes to the retriever, not to the prompt telling the model to be more assertive.

Two more bugs from the same audit

We put the harness through an adversarial code review before letting it produce a customer number. Two other findings changed reported scores:

Duplicate chunk ids inflating nDCG above 1.0

Real systems return overlapping windows of the same chunk. If the same id appears twice in a ranked list and both are credited, discounted cumulative gain can exceed its own ideal — a system scoring better than perfect. Duplicates are now credited once, at their best rank.

A chunker silently deleting text

Body text sitting on the line directly under a markdown heading, with no blank line between them, was being dropped along with the heading. The affected content simply became unretrievable — and every metric still reported cleanly, because the ground truth was written against the same broken chunks.

We also moved precision@k to the standard denominator (k, not the number of results returned), so a retriever that returns three results instead of five cannot win on precision by returning less.

Why this matters more than the model choice

Every reliability programme rests on a measurement chain: chunker → retriever → judge → score → decision. A bug anywhere in that chain does not announce itself. It produces a plausible number, a green dashboard, and a decision made on fiction. All three findings above produced plausible numbers.

So when you evaluate an eval vendor — including us — the questions worth asking are: what does your metric do with a zero-claim answer? What does it do with duplicate contexts? What is the denominator of precision@k? Can I reproduce your score from your raw output? If the answer is a dashboard rather than a definition, you are buying a number nobody has audited.

https://github.com/nnnirvana/attova-harness

← All teardowns