The Hallucination of Competence

A short field note on how AI sounds most confident exactly where it knows the least.

I recently ran into something that completely changed how I use AI.

01 / The Incident

When AI sounded right, but wasn't

I was working with a decade-old open-source project and wanted to get it running. Instead of digging through the code myself, I handed the repository to an AI assistant and asked a simple question:

“How do I get this running?”

A few seconds later, it gave me what looked like the perfect answer. It generated a clean configuration file, suggested a modern project structure, and even included a simple “Hello World” example. If I hadn't looked at the actual code, I probably would have followed its instructions without thinking twice.

The problem was that almost none of it matched the repository.

02 / The Trap

Why the answer looked so convincing

While working on the code, the AI had leaned heavily on an outdated README and filled in the rest with what it thought a project like this should look like. It assumed a modern directory layout even though the codebase was over ten years old. It recommended a dependency with the right name but a completely different API. Everything it produced looked polished and believable.

If someone gives you a wrong answer and sounds unsure, you're naturally skeptical. But when an answer is organized, well-written, and delivered with complete confidence, it's much easier to assume it's correct.

The AI wasn't trying to deceive me. It was doing what these models are designed to do — generate the most likely next answer, even when the evidence isn't there.

03 / The Shift

What finally worked

At first, I thought I just needed a better prompt. I kept asking it to be more detailed, explain its reasoning, or walk me through the setup more carefully.

None of that helped. What finally worked was changing the role I gave it.

Instead of asking it to help me get started, I asked it to review the repository like a skeptical engineer. I told it to cite the exact lines of code it was using as evidence, point out anything that looked like an assumption instead of an observation, and avoid trusting the README unless the code backed it up.

Then, it started pointing out real issues. It found the dependency mismatch that had been causing problems. It highlighted a genuine thread-safety concern. It even noticed tests that looked like they were passing while quietly swallowing failures.

The model hadn't suddenly become smarter. I had simply stopped encouraging it to guess.

04 / The Habit

What I do differently now

Since then, I've changed how I use AI.

Now, whenever I'm using AI to understand an unfamiliar codebase or work through something where correctness actually matters, I don't ask it to give me an answer first.

I ask it to prove its answer. If it can't point to the code, I assume it's making an educated guess. That one single habit has saved me far more time than any clever prompt I've written.