The exposure
Draft — to be completed
- A tourism website published AI-generated content describing hot springs that do not exist, and visitors travelled to find them. The business lost credibility in a single news cycle. Hallucinations are not a theoretical problem; they are preventable with evaluation before publication.
- Assistants answering from documents (legal, research, policy) are trusted because they sound right. A confident wrong answer in a client deliverable is a reputational and regulatory event.
- Early “paste the documents into an LLM” pilots are fast and become liabilities for exactly this reason.
Where standard controls fall short
Draft — to be completed
| PRACTICE | WHAT IT HIDES | CONSEQUENCE |
|---|---|---|
| Task-completion checks only | An out-of-scope question answered fluently with zero retrieval scored 5/6 | Ungrounded answer passes |
| Same model writes and grades | Self-assessment bias | Inflated quality scores |
| No scope guard | Assistant answers questions outside its knowledge base | Wrong domain, confident tone |
| No citation requirement | Claims cannot be traced | Reviewers cannot verify |
| All answers treated equally | Low-confidence answers reach clients | No safety valve |
What we recommend
Draft — to be completed
- Scope guard that returns “out of scope” before any retrieval.
- Answer-only-from-context contract: say “I don’t know” when the documents don’t support an answer.
- Independent judge model scoring groundedness and citation precision against a gold set.
- Confidence-routed outputs: low confidence goes to a person, never silently to a client.
- Prompt improvement driven by failure feedback and validated on held-out questions.
- Groundedness and relevance tracked over time; drift triggers review.
OUR RECOMMENDATION
Ask for two numbers before you trust an assistant: how often it answers from the documents, and how often it says it doesn’t know when it should. If nobody can give you those, it has not been evaluated.
Findings are from EonAI’s reference systems, built and tested on synthetic data. They describe patterns we see across clients’ systems, not a client engagement.