When AI Gets Confidently Wrong

The model wasn't lying. It was guessing, and it sounded sure of itself while doing it.

A Tesla on autopilot once drifted into the wrong lane, reading a faded road marking as a real one. Ask an AI travel assistant for a restaurant recommendation in a small town and it might invent one entirely, complete with a cuisine style and a price range, none of it real. Both moments share the same root cause: the model wasn't lying. It was guessing, and it sounded sure of itself while doing it.

This guessing has a name: hallucination. It shows up constantly in products built on large language models, and it tends to surface at the worst possible time, when a user has no easy way to check the answer against reality.

01 / The Accident

Half of a very old story

Oddly, this same tendency has a long and productive history outside computing. Alexander Fleming didn't set out to discover penicillin. A contaminated petri dish led him somewhere unplanned, and the accident became one of medicine's biggest breakthroughs. Human error, followed by careful verification, can turn into insight.

AI hallucination follows the first half of that story without the second. It stumbles into unplanned territory just as readily, but nothing downstream checks the work.

02 / The Machinery

Why there's no internal flag

Understanding why requires letting go of the mental model most people bring to software. A database looks something up. A language model doesn't have facts stored anywhere waiting to be retrieved. It generates the next plausible word based on patterns absorbed during training, over and over, until a full sentence emerges.

Most of the time this produces something true, because true statements tend to be common in the training data. Sometimes it produces something fluent and false instead, and there's no internal flag that distinguishes the two.

Given that, the responsibility shifts to the people building products on top of these models. A few design principles help.

03 / The Principles

Designing around the guess

Creativity should match the task. A marketing team drafting ad copy or a novelist brainstorming plot twists benefits from a model that wanders freely; unexpected connections are the whole point. A newsroom fact-checking a quote, or a hospital system summarizing patient records, needs the opposite. The same underlying model can serve both, but the guardrails around it shouldn't look the same in each case.

Uncertainty deserves a voice. The real problem isn't that models sometimes get things wrong. It's that users can't tell when they have. A well-designed system says “I'm not sure” or “this is based on limited information” instead of stating a fabricated restaurant name with the same tone it uses for a verified fact. Giving users a signal for when to double-check turns a silent failure into a manageable one.

Oversight should scale with stakes. Low-risk suggestions, like a playlist recommendation, can run on autopilot. High-risk decisions, like a medical diagnosis or a legal filing, need a human in the loop who treats the AI's output as a draft rather than a verdict.

The model recommends. A person decides.

04 / The Posture

Design for the failure, not the miracle

None of this requires AI to stop hallucinating altogether, which may not even be possible with current architectures. Airplane engineers don't design around the assumption that every part will work perfectly; they design around the assumption that parts fail, and build redundancy and warning systems accordingly. AI products deserve the same posture.