WhereWindAll

The computer/brain analogy is compelling because computers are able to do some things we find extremely difficult and to which we attribute high intelligence to people who can do those things. Playing chess, for example. Most of these things involve what amounts to effective lookup of useful information in a large database. The things that we find simple to do, like perceiving and navigating through a complex world, computers find extraordinarily difficult. Somehow, we do not see that as a lack of intelligence.

Computers are fundamentally dualist. Brains are not. By that I mean that computers are a hardware substrate on which an algorithm created by an external entity executes. That at least suggests that the analogy between brains and computers could be just that -- an analogy. As French neuroscientist Yves Frégnac put it, "big data is not knowledge."

LLMs are an elaborate way of accessing big data. What strikes me about what enthusiasts are eliding from LLMs is that what they do is not driven by knowledge. It is driven by a pastiche of things that have been said in the past by humans. So when they argue that LLMs indicate the imminent arrival of true artificial intelligence, they are in effect claiming that intelligence does not depend in any way on actual knowledge. That strikes me as nonsense.

Brains are not radiators. They are not fountains. They are not steam engines. They may not even be computers. No one really knows that yet. Another French neuroscientist, Romain Brette, has challenged this metaphor in some detail. Brette points out that in thinking of brains as running code, researcher unconsciously drift between different meanings of the word "code." Starting from a technical sense, in which code means there is a link between a stimulus and the activity of a neuron, they drift into a very different, representational, meaning in which neural codes represent that stimulus, without justifying, or even consciously acknowledging, that shift.

This is dangerously close to a homunculus model. The unstated implication, using the representational meaning of code, is that the activity of neural networks is presented to an ideal observer or reader within the brain, often described as "downstream structures'" that have access to optimal ways of decoding the signals. With LLMs, it is pretty obvious that the downstream structure is us.

The cognitive revolution in psychology, starting in the 1970's, has pretty clearly demonstrated that viewing the brain as a passive computer that responds to inputs and processes data is wrong. Brains exist in bodies. Those bodies are interacting with and intervening in the world, and a considerable portion of whatever it is that brains do is based on sensorimotor metaphors derived from these interactions. And I should point out here that the meaning of metaphor is not the usual "How shall I compare thee to a summer's day" sense. Rather, the cognitive theory of metaphor involves wholesale export of reasoning methods from one domain into a completely different one, e.g. using the ability of the brain to reason about navigation to instead think about mathematics. This is what a number line is. When the metaphor changes (as it did in mathematics from numbers as enumeration of objects to labeling positions along a path), the meaning changes as well (as when the enumeration metaphor excluded the concept of zero as well as irrational numbers from the world of numbers -- the Pythagorean position -- to requiring them to be numbers since otherwise those positions along a path lack labels).

In 2015, the roboticist Rodney Brooks chose the computational metaphor of the brain as his pet hate in his contribution to a collection of essays entitled This Idea Must Die. Less dramatically, but drawing similar conclusions, two decades earlier the historian S Ryan Johansson argued that “endlessly debating the truth or falsity of a metaphor like ‘the brain is a computer’ is a waste of time. The relationship proposed is metaphorical, and it is ordering us to do something, not trying to tell us the truth.”

Reverse engineering a computer is often used as a thought experiment to show how, in principle, we might understand the brain. Inevitably, these thought experiments are successful, encouraging us to pursue this way of understanding the squishy organs in our heads. But in 2017, a pair of neuroscientists, Eric Jonas and Konrad Paul Kording*, decided to actually do the experiment on a real (and simple) computer chip, the MOS 6507 processor that was used in popular video games in the 70's and 80's. Things did not go as expected.

They deployed the entire analytical armament of modern neuroscience to attempt reverse engineering the CPU. Despite the fact that there is a clear explanation for how the chip works, they were unable to detect from outside the hierarchy of information processing that occurs inside it. As Jonas and Kording put it, the techniques fell short of producing “a meaningful understanding”. Their conclusion was bleak: “Ultimately, the problem is not that neuroscientists could not understand a microprocessor, the problem is that they would not understand it given the approaches they are currently taking.” This is directly related to neural networks in general as they are the blackest of black boxes. No one knows how they convert input into output, and this experiment suggests that such knowledge cannot be obtained with current techniques. Absent that knowledge, claims of "sentience" or "intelligence" are specious.

Post #1 ------------------------------------------------------------------------------

I worked in a school for a year post-pandemic doing IT support as they rolled out 1:1 devices for the kids. Not a lot of professional cred, I'll admit, but actual first-hand experience nonetheless. What I saw was a school that bought the sizzle but got a chewed-up plastic hamburger. The super cool ultra-high-tech hyper-personalized education software OF THE FUTURE made everything worse. Teachers couldn't teach if the network went down or if a license wasn't renewed or a vendor updated their app and broke the damn thing for a day in the process. With all the activities being splooged out of some SV assembly line somewhere, and all the student records being stored safely on the cloud, any disruption was essentially a death sentence for the day's learning...which was made even worse in the classes where they had students all at different points in a lesson, because even if the teachers did still have materials to work from, their students were all over the map and it wasn't possible to do anything coherent. Administrators routinely hammering that teachers must use the tools regardless of educational outcomes because the school had paid for the license to a software package and if they wanted to keep App A they also had to show they were using Apps B and C to maintain the funding. Added bonus, teachers who focused on teaching the material instead of teaching the software got hit on their end of year reviews. Added added bonus, teachers also got dinged on parent reviews because the educational software that they don't control didn't work and was confusing. Hey, you know how older people and people who aren't technophiles aren't great with new technology, especially stuff like license-based SAS kinda products that change constantly? Guess how good most of the teachers in the building were at figuring out wtf was going on and adapting or even calling the right person when their SSO broke? Also super fun trying to help teachers understand the difference between whitelisting and blacklisting websites and how to tell when their screen monitoring software had failed to release the previous students so that another teacher could work with that class and what to do about it. Know what a chatbot is absolute garbage at doing? Understanding. Anything. At all. That's not what they do or how they work. The chatbots can't tell if a kid knows the subject, it can only tell if the kid knows how to use the app the subject is being presented on. When we did finals, most of the kids in K-5 absolutely bombed the first time through because the testing software was different from the learning software and they were being asked to interact with the material in a fundamentally different way than what they'd been taught for the whole year. You wanna see kids break down and cry? Watch them sob uncontrollably because they thought they did everything right but then their entire test gets wiped out by a laptop battery dying. Watch them freak all the way out when their test software flags them for cheating and locks them out and their teacher can't help them unlock it. I could go on and on about how utterly fucked the whole thing was, but the point here is that educational software as a tool for educators to work with is fine (to a point), but the reality is that schools are going to push more and more into using educational software as a pillar for educations to be built on, and that's really bad. I wish I had any reason to believe that slapdash, ever-changing, top-down software dressed up with a chatbot on the front end (or whatever other trendy nonsense is hot in SV at the time) was gonna be good for kids, but I just don't...and that's about the only kind that exists. The digital divide is real and it's not just about access to shiny new laptops. There is a massive gap between the technical skills most people actually have and the technical skills the tech bros think everyone should have, and "do more computers harder" is gonna make that way, way, way worse, not better.

Post #2 ------------------------------------------------------------------------------