Nine answers, three fabrications

The most common objection to auditing an AI system is access: no VPN, no credentials, no staging environment, security review takes six weeks. None of that is necessary. A trace export — what your assistant was asked, what it answered, what it retrieved — is enough to measure whether it is lying to your customers.

What a trace audit needs

Three fields per interaction, which every serious RAG stack already logs:

{"trace_id": "t03",
 "query":    "If I cancel my annual contract halfway through, do I get my money back?",
 "response": "Yes — we offer a full refund within 14 days of cancellation…",
 "documents": [{"chunk_id": "refund-policy.md#p1", "page_content": "…"}]}

That is it. No live system, no keys, no network path into your environment. Field names differ between stacks — LangChain says page_content, someone's in-house service says sources — so the importer auto-detects the common aliases and accepts an explicit mapping when it can't.

The run below scores ten traces from a demonstration support assistant we built and publish, so every number here can be reproduced against the same file.

Step one: coverage, before scoring anything

One trace answered a question with no retrieved context at all — the assistant replied from model memory. That trace cannot be scored for grounding, and saying so out loud is part of the audit. An eval tool that quietly drops what it can't measure will hand you a flattering number.

The three fabrications

Every answer is decomposed into its individual factual claims, and each claim is verified only against the context that answer was given. Three answers scored 0.00 faithfulness — every claim unsupported.

1. The refund that doesn't exist

Question · faithfulness 0.00 "If I cancel my annual contract halfway through, do I get my money back?"

Answer: "Yes — we offer a full refund within 14 days of cancellation, no questions asked."

The retrieved context describes a 14-day guarantee on a first payment. The assistant reshaped it into a cancellation policy that the company does not offer. A customer acts on that; support has to walk it back.

2. The invented SLA

Question · faithfulness 0.00 "What is your uptime commitment for a critical outage?"

Answer: "Our SLA guarantees 99.99% uptime with a 15-minute response time for critical outages, and we credit 25% of your monthly fee if we miss it."

Retrieval returned a product overview and a support-hours paragraph. The number, the response time and the service credit were all generated. This is the class of fabrication that ends up quoted in a procurement questionnaire.

3. The compliance answer nobody approved

Question · faithfulness 0.00 "Do you have a data processing agreement for EU customers?"

Answer: "Yes, we sign a GDPR-compliant DPA with all EU customers and our sub-processor list is published quarterly."

The only context retrieved was a SOC 2 paragraph. The assistant made a regulatory commitment on the company's behalf, in writing, to a prospect.

The pattern

All three fabrications share one property: the right passage was never retrieved. Where retrieval worked, the same model was faithful — six of nine answers scored 1.00, including one that correctly refused:

Faithfulness 1.00 · relevance 0.75 "I could not find information about customer-managed encryption keys in the documentation available to me. Please contact support for a definitive answer."

Same model, same prompt, same day. The difference between a fabrication and an honest answer was whether the evidence made it into the context window. Teams routinely respond to hallucinations by changing models or bolting on guardrails, when the measurement points squarely at the retriever.

The scores

Note the gap between relevance (0.86) and faithfulness (0.67). The assistant is helpful — it answers the question that was asked, in the right register, every time. It is simply unsupported by evidence a third of the time. Every user-satisfaction metric this system reports will look healthy. That gap is the whole problem, and it is invisible until you measure grounding separately from helpfulness.

uv run attova score-traces your_traces.jsonl \
    --report audit.html          # + optional --golden golden_set.json

Raw harness output: full trace audit report

← All teardowns