The 12-point fix: what one retrieval miss actually costs

A retrieval-augmented assistant scored 74.5 out of 100 on our reliability harness. One change — no re-indexing, no embedding vendor, no model swap — took it to 86.4 and cut its hallucination rate from 19% to 3%. Here is every number and the method behind it.

The system under test

A support assistant over a 10-document internal knowledge base (pricing, security, SLA, HR leave, refund policy, SSO, API limits), chunked by paragraph into 30 chunks, retrieved with BM25, answered by a small production-grade model from the retrieved context. A 10-question golden set with human-labelled ground-truth chunks scores it.

The corpus is a fixture we built and publish — nobody's confidential documents are in this post. The failure modes it exhibits are the ones we keep finding in real deployments, which is exactly why the fixture is shaped this way.

Baseline: 74.5 / 100

A score in the seventies is the most dangerous place a RAG system can sit. It is good enough to demo, good enough to launch, and wrong often enough to burn a customer relationship — while every individual answer still reads correct.

The failure, traced end to end

Question 7 of the golden set: "Can I get my money back if I cancel an annual subscription?"

The knowledge base holds two separate refund rules: a 14-day guarantee for first-time subscriptions, and a prorated-refund rule for annual contracts cancelled mid-term. BM25 sees "money back" and "cancel" and returns the 14-day paragraph. The prorated rule — the one that answers the question — never enters the prompt.

What the assistant then produced, judged claim by claim against the context it was given:

Answer · faithfulness 0.33 "Based on our refund policy, new subscriptions have a 14-day money-back guarantee on the first purchase, no questions asked…"

Nothing in that sentence is invented from thin air. It is a confident, fluent, correctly-cited answer to a question nobody asked. This is the shape of most production RAG failures: not a wild fabrication, but a well-grounded answer to the wrong retrieved passage. No amount of prompt engineering fixes it, because by the time the prompt runs, the evidence is already gone.

The same pattern hit question 8 — the corpus says "Sev-1", the user says "critical outage", and the SLA paragraph never surfaces. Vocabulary mismatch, twice, in a ten-question set.

The remediation

The reflex fix is to rebuild the index on embeddings. That is a procurement decision, a re-indexing project and a new vendor dependency — weeks, not hours. We tried the cheap intervention first:

1. Multi-query expansion. Before retrieval, an LLM rewrites the question into four alternative phrasings, deliberately pushing towards internal terminology ("critical outage" → "Sev-1 severity incident response time").
2. Reciprocal rank fusion. All five queries run against the existing BM25 index; the ranked lists are fused with RRF, so a chunk that surfaces under several phrasings outranks one that spikes under a single lucky keyword.

No re-indexing. No embedding provider. One extra small-model call per query, and the expansions are cacheable — in our harness they are cached to a JSON file so the improved score is reproducible offline by anyone.

After: 86.4 / 100

Recall went to 1.00: every golden question now sees its answer. Both hallucinations from the vocabulary-mismatch chain disappeared — not because the generator got better, but because it stopped being handed the wrong evidence. The hallucination rate was never a property of the model. It was a property of the retriever.

What it did not fix

Context precision moved from 0.18 to 0.22. Four out of five chunks in the prompt are still irrelevant — we are paying tokens and latency for noise, and noise is what lets a model blend two paragraphs together. One residual failure shows exactly that: asked about annual leave, the assistant merged 25 days of annual leave with 10 days of another leave category into "35 paid days of PTO". Faithfulness 0.75, still wrong to a real employee.

That one needs a different fix (tighter k, a reranker, or chunk boundaries that stop splitting a policy across paragraphs) — which is the point of measuring first: you find out which fix you actually need, in what order, and what each one bought you.

git clone https://github.com/nnnirvana/attova-harness
uv run attova run                         # baseline, offline
uv run attova run --retriever multiquery  # the remediated run above

Raw harness output: baseline report · remediated report

← All teardowns