A hallucination is an answer that sounds credible but isn’t true: an invented number, a regulation that doesn’t exist, a quote nobody said or a link that leads nowhere. It isn’t a bug one update can fix. A language model predicts the most likely continuation of a text and has no built-in knowledge of what it doesn’t know.
That the company, not the model, bears the responsibility isn’t theory. In 2024 a Canadian tribunal ruled that Air Canada was liable for incorrect information about a fare discount its chatbot had given a customer. The argument that the chatbot was a “separate entity” didn’t convince the tribunal.
When hallucinations are most common
- When the model has no data in its context and answers from memory rather than from documents.
- When the question is about details: numbers, dates, names, prices, regulation numbers, quotes and sources.
- When the task is open-ended and the model has to decide on its own what matters.
- When the prompt pressures it to always give an answer.
Where the risk is highest
Not every use of AI needs the same safeguards. It helps to split tasks into two groups.
High risk: replies sent to customers without review, information about prices, terms and deadlines, legal and financial content, figures in reports for leadership, code going to production.
Low risk: brainstorming, first drafts someone will read before publishing, reformatting data, summarizing a document the reader has in front of them anyway.
The safeguards below matter mainly for the first group.
How to reduce the risk
- Give the model sources. Instead of asking for knowledge from memory, give it the company’s documents and have it answer only from them. This approach is called RAG. Also require it to point to the passage the answer comes from.
- Let it say “I don’t know”. Put it in the instructions, then check with out-of-scope questions whether it really declines.
- Narrow the task. Choosing from a list, extracting data from a document or filling in a fixed structure is much safer than free writing.
- Check automatically what can be checked. Compare prices with the database, open the links, recalculate the numbers, validate the format. A program does this faster and more accurately than a person.
- Keep a human where mistakes are costly. A reply to a customer complaint or a proposal can be drafted by AI but sent only after approval.
- Test before every change. Collect a few dozen real questions with correct answers and run them after every change to the instructions or the model. Measure how many answers are correct, how many are refusals and how many are wrong.
- Review conversations. Every week, read a sample of real conversations. Mistakes the tests didn’t anticipate then go into the test set.
What not to do
- Don’t count on “don’t hallucinate” in the instructions. The model doesn’t know when it’s hallucinating, so asking changes little.
- Don’t launch a customer-facing assistant without testing just because it did well in a demo.
- Don’t hide that AI is answering. A customer who knows who they’re talking to treats the answer differently and finds it easier to ask for a human.
Where to start
Before you build an assistant for customers, work out which tasks in the company can safely be handed to AI and which need the safeguards above. That’s how we start every implementation: with a process audit and a list of tasks ranked by savings and risk. Read more on our AI Boost service.
Let’s talk about your project
The first call takes 30 minutes and is free. After it, you know whether we can help and what it will cost.
Book a call
