Definition
When an AI generates confident-sounding but incorrect or fabricated information. A core concern in AI deployment: mitigated via retrieval-augmented generation, human review, and constrained workflows. The model is not lying; it is producing the most plausible-sounding text it can, and plausible is not the same as true, which is why hallucinations read exactly like correct answers.
What it looks like in practice
A moving company's AI drafts a reply to a customer asking about a third-floor walk-up. The draft is well written and mostly right (crew size, arrival window, packing materials) except for one line: 'stair carries above the second floor are included in the hourly rate.' They are not; this company bills a flat stair fee per flight. The model had never been told, so it filled the gap with what usually sounds true. A reviewer catches it because she knows the fee schedule, fixes the line, and sends it. Cost of the error: one edit before send. Cost without the review: an argument on moving day.
Why it matters
Hallucination is the single best argument for human oversight in a managed setup, and the reason approval gates exist at all. The failures that hurt a small business are rarely dramatic (a price that drifted, a policy that never existed, a warranty term invented to sound reassuring) and they are dangerous precisely because they read as authoritative. Three things reduce the risk together: grounding answers in the business's real documents, narrowing what the AI is allowed to do unprompted, and putting a person between the draft and anything money- or brand-sensitive. None of the three is optional if customers are reading the output.