[source]
🚨BREAKING: OpenAI published a paper proving that ChatGPT will always make things up.
Not sometimes. Not until the next update. Always. They proved it with math.
Even with perfect training data and unlimited computing power, AI models will still confidently tell you things that are completely false. This isn't a bug they're working on. It's baked into how these systems work at a fundamental level.
And their own numbers are brutal. OpenAI's o1 reasoning model hallucinates 16% of the time. Their newer o3 model? 33%. Their newest o4-mini? 48%. Nearly half of what their most recent model tells you could be fabricated. The "smarter" models are actually getting worse at telling the truth.
Here's why it can't be fixed. Language models work by predicting the next word based on probability. When they hit something uncertain, they don't pause. They don't flag it. They guess. And they guess with complete confidence, because that's exactly what they were trained to do.
The researchers looked at the 10 biggest AI benchmarks used to measure how good these models are. 9 out of 10 give the same score for saying "I don't know" as for giving a completely wrong answer: zero points. The entire testing system literally punishes honesty and rewards guessing.
So the AI learned the optimal strategy: always guess. Never admit uncertainty. Sound confident even when you're making it up.
OpenAI's proposed fix? Have ChatGPT say "I don't know" when it's unsure. Their own math shows this would mean roughly 30% of your questions get no answer. Imagine asking ChatGPT something three times out of ten and getting "I'm not confident enough to respond." Users would leave overnight. So the fix exists, but it would kill the product.
This isn't just OpenAI's problem. DeepMind and Tsinghua University independently reached the same conclusion. Three of the world's top AI labs, working separately, all agree: this is permanent.
Every time ChatGPT gives you an answer, ask yourself: is this real, or is it just a confident guess?
The easiest way to conceptualise why LLMs are always lying is to think about it is as an autocorrect on steroids. Ya know how you can mash the middle suggested words on your phone keyboard and it sometimes sounds like something you would say? That's all these LLM 'AI' things are at the moment. The one in the autocorrect on your phone has been trained just on how you type on your phone, the LLMs they're talking about have been trained on all the text available online, hence the 'large' in 'large language model'. Its impressive but its still just guessing what the next word will be. I wouldn't even go so far as to say its lying or guessing, its not thinking or anything. They're not even like a mouse, or an insect. They're just plinko machines. The ball Plinks down through a multidimensional model and the pins it hits are words.
They can do fancy tricks of stacking them so that one guessing machine will look at the outputs of two or more guessing machines and then guess which one will probably be more correct. They call these one 'thinking models'. But in the end it's still just autocorrect on steroids.
LLMs are always lying in the way that a plinko machine will sometimes end up with the ball taking a weird route though the pins. Its not that its wrong sometimes, it's that we notice that the through line of pins that the ball happens to hit, and the words those pin line up with, doesn't line up with things we know to be true. Then we turn around and say the plinko is lying, like it has any agency.
To further expand on this, the pins in a plinko is actually a pretty good analogy for how LLMs understand what they're doing. Its not words to an LLM, its point in space. The points are labeled or numbered, and it knows that when it hits point 76362983, that usually point 668827 follows, after that it can choose 444449855 or 873893919, and then go to 66588873987. That's it deciding if it says 'its a safe mushroom' or 'its an unsafe mushroom'. Obviously it's more complex and nuanced than this, but it shows how an LLM fundamentally cannot know what its saying to us at a base level. Sure, we can adjust the positions of the pins so that the ball tends to fall in a way so that the output happens to line up with sentances we know are true, but at it's core, it's just a plinko we've made so complex that some of us trick ourselves into thinking is fully alive.
"Whohh, this horse plinko said it's in love with me."
