The problem with AI comes from a problem in the idea of knowledge.
A lot of times people act like human knowledge is "discovered" or obtained, and after that it simply exists; it becomes an eternal, transcendent thing as part of "what we know" about the world.
For example, "we" "discovered" that microorganisms cause disease, and thereafter this became a part of "human knowledge."
The problem is, there was not a discrete event of discovery; long before microscopy was invented, some doctors (such as Ibn Sina aka Avicenna) in the Islamic Golden Age theorized that diseases were caused by tiny particles or organisms that could travel from person to person.
And indeed long after microorganisms were actually observed, people still didn't adhere to the belief that they could cause disease, and with each different illness the connection to a microorganism had to be established. And even today, some people don't believe that microorganisms cause illness.
Then again, the behaviors and methods for reducing the spread of illness, which were proven to be effective by observing microorganisms, didn't require the "knowledge" of microorganisms to develop. Behaviors such as quarantine, washing, and avoiding touching infected people or dead bodies, were followed centuries before it was possible to observe a germ.
As you can see, it's hard to say when "we" "Knew" or "did not know" the causes of illness and how to prevent them.
When someone followed the behaviors that prevented illness spread, they did not necessarily know why. When someone theorized that tiny particles caused illness, they did not necessarily have proof. When someone proved that a microorganism caused illness, they did not necessarily convince others. When the proof was written down, not everybody read it. When people read it, they did not necessarily follow the behaviors that prevented illness spread.
Someone in the year 1020 might "know" that when you get the plague, you have to quarantine, whereas someone in the year 2020 might not "know" that. How does that happen?
Knowledge does not just exist like an eternal spirit. It has to be learned, maintained, practiced, and transmitted.
The internet is full of information. But is the internet full of knowledge? No. A written work is a means of transmitting information, a pipe from one brain to another that information flows through. Only when a human learns information, does it become knowledge. Books, likewise, are pipes that information flows through. They don't actually "have" knowledge. If no one reads the book, the knowledge doesn't exist.
The internet is a vast repository of information. AI shoves it all into a blender and extrudes it into sentences that have form. But the information wasn't knowledge, it was just a device for transmitting the knowledge, and the extruded sentence-stuff produced by AI is even farther from being knowledge, because the transmission between brains has been interrupted and partially destroyed by the randomness inherent within the functioning of Large Language Models.
A library can contain all the secrets of the universe, but no one will "have" the knowledge unless they read the books. And by reading the books, I mean not just being able to comprehend the structure of language within the sentences, but actually learning. When you try to understand a concept and it is so hard that makes your head feel fuzzy, that's because learning is happening. Your brain is literally growing new connections and pathways.
Learning requires teaching. A way to look at knowledge transmission is through the idea of apprenticeship. A master teaching an apprentice is self-replicating: they contain knowledge, and they are shaping another person to contain the same knowledge, molding another brain into the same shape.
So, we can say that knowledge transmission happens when a knower self-replicates, causing another knower of the same knowledge to exist. But a knower can't self-replicate by splitting into two people. Instead, they have to somehow put their knowledge into another person from the outside. Language is a good way to do that.
Someone who writes down their knowledge is not self-replicating by writing it down. They have created a secondary object or tool to aid the self-replication process. Without brains, language would be nothing, there would be no such thing as symbols or meaning, language would not exist.
This means that information is completely worthless if it is not successful at replicating knowledge from one brain into another. Which means that the level of information we have stored in various repositories does not directly correlate with how much "we" "know." Written information can fail to become knowledge in various ways: people can just not read it, or the writing can be too hard to understand, or it can be forgotten once understood, or it can be discredited or seen as untrustworthy, or nowadays, with generative AI, it can be interrupted and distorted on its way to the learner.
Large language models can only be seen as "intelligent" if you think that language itself can be a "knower." I think it is clear that language, even though it transmits information, cannot know it.