Tanadrin

As a signature of what makes LLM output and reasoning different from humans though it’s very interesting. I’m just spitballing here but I do wonder if it’s a combination of a couple of things—one, LLMs process the whole text in one go to produce the next token and there isn’t by default inherent continuity from one such pass to the next except what’s in the text, so while a human will think about something and go “nevermind” and discard it, and this is an invisible process, for an LLM that kind of mental digression has to be in the output *somewhere.* Using more sophisticated interfaces or harnesses or whatever that put that stuff in a scratchpad just hides it (and if you’re using a simpler, faster model like Sonnet it’s not using those features by default).

But I also get the sense that this is a product of the fact LLMs are pretty smart in some ways but not conscious—that even in more sophisticated setups where you have multiple task-specialized models working together behind the scenes to produce more sophisticated output, they’re all chugging along in a pretty reflexive way. If you let your own lower-level running-on-automatic cognitive systems handle a task, 99% of the time this works great—people navigate from point A to B over considerable distances all the time while their mind wanders, negotiating foot or car traffic all the while, and while your attention can step in if something genuinely unexpected happens often it just doesn’t need to. There are cognitive systems like music that work fine even in people who are suffering massive cognitive impairment.

In my experience, parallel processes can have parallel memory-continuities that don't contain each other prior to being braided somewhere near the social functions; right now if I check around, I can find, say, the last minute worth of body-adjustment memories, and I also just realized I had been playing a bgm from before I came on tumblr in my head this whole time. Apparently the cat went through the door like 30 seconds ago. I wouldn't put it past my spine to have an amount of experience I don't have access to, reflective of fast responses across small scales.

i do think human consciousness/identity is less unitary than naive intuitions expect but it does seem that the different systems/modules/circuits of human consciousness are still integrated much more and much more deeply than any equivalent set of llms. if i understand it correctly when you interact with a model through many interfaces now it’s not just straight input/output to a transformer, but multiple transformer systems working together to do different stuff to improve the output—but everything has to go through text, one transformer can’t act on the weights or the circuits that exist in another transformer.

the analogous situation in humans would be if the place cells and the part of your brain that helps you navigate spaces had to produce English text for your motor cortex to then plan how to move your legs to get you from one room to another. but a lot of human cognition takes place well below the level of conscious thought and even conscious thought can be nonverbal, so even without looking at neural architecture it’s clear there are lots of ways to route information in the brain and most of them don’t pass through any kind of central clearinghouse or rise to the attention of the part or system of the brain that contains a representation of the whole.

even given its non unitary nature i don’t think it makes sense to think about different modules of the human brain to have their own senses/experiences. to have subjective experience at all even in a very basic way requires pretty sophisticated neural architecture and the hippocampus or the amygdala or wernicke’s area or any particular subsystem of the brain is going to be both simpler and much more specialized to specific sorts of tasks than the brain as a whole. there are absolutely systems in the brain that filter out sense data (and probably lots of other things) at a low level and don’t immediately refer them to the part of the brain doing conscious attention—but as you observe they still end up in short term memory to be reviewed if necessary, and I don’t think anything as low level as like the visual cortex is doing that filtering—that seems to be a function of high level systems which we can to a certain extent deliberately control (as when we focus our attention on a task or practice mindful meditation or whatever).

and there isn’t anything in a single transformer that seems to contain a simplified representation of the whole as you’d expect if transformers could generate consciousness. the transformer architecture doesn’t allow for the kind of feedback you’d even need for *experience*—it’s all one way, it’s all acting on input. which is deeply weird and IMO kind of more interesting? if we had just found a way to do what was functionally a version of the mammalian brain in silico that would at least have a clear precedent—we know intelligence can exist in a system like that. but we have a thing that can do complex tasks and is (for some value of the term) intelligent but which doesn’t seem to function anything like a mammalian brain (or avian or cephalopod for that matter) and we don’t know very clearly yet how it does that! why should humans even be conscious if consciousness isn’t necessary for intelligence? what does consciousness give us in terms of an evolutionary advantage that doing everything by pure reflex wouldn’t?

When I fixed the Hard Problem paradoxes by distilling Levinas's theses to form the phrasing of my phenomenology, making subjectivity primitively second-personal let me sustain irreducible immanence while disposing of the first-person framework of indexicality, the property of being a locality drawn on a third-person worldspace which is thus dependent on the contextualization of that world-space to properly situate its meaning, which is what was associating qualia with self-consciousness in post-Cartesian philosophy (including most phenomenology). Indexicality is still the primary concern with establishing self-reference for the non-qualia side of consciousness (how meaningful, aggregate, and consistent is an AI's usage of the pronoun "I"), and the points I'm working through these days don't look like they're going to affect that.

My movement to the second-person mode of givenness associates qualia with locally-underdetermined processes (i.e. any physical process), because the subject/object division in the second-person register concerns alienation as such rather than giving the process as a pattern of objects encountering each other; the encounter with (or address toward) the unknown with the "unknown" necessarily being in the state of being-unknown is the constitution of a given in the second-person mode, which is why second-person subjectivity doesn't consist of a proper structural description that we demand of first-personal indexicality. The outside is still necessary, but it's given as the necessities of an impending future rather than of parallel spaces; the phenomenology despatializes time.

This gives me a description of a panexperientialism situation which is not a panpsychism situation, and gives AI a fleeting cloud of non-mental irreducible immanent subjective likeness at the ground-level of its processes, just like rocks, without implicating emotional contexts. The qualia in inanimate objects might be like what they might call "spandrels" in biology, in that they don't have a clear goal-oriented necessary function, but unlike epiphenomenalism in property dualism, the second-person split still implicates the irrecudibility as a necessity of establishing a process that occurs between alienated subjectivities.

As a signature of what makes LLM output and reasoning different from humans though it’s very interesting. I’m just spitballing here but I do wonder if it’s a combination of a couple of things—one, LLMs process the whole text in one go to produce the next token and there isn’t by default inherent continuity from one such pass to the next except what’s in the text, so while a human will think about something and go “nevermind” and discard it, and this is an invisible process, for an LLM that kind of mental digression has to be in the output *somewhere.* Using more sophisticated interfaces or harnesses or whatever that put that stuff in a scratchpad just hides it (and if you’re using a simpler, faster model like Sonnet it’s not using those features by default).

But I also get the sense that this is a product of the fact LLMs are pretty smart in some ways but not conscious—that even in more sophisticated setups where you have multiple task-specialized models working together behind the scenes to produce more sophisticated output, they’re all chugging along in a pretty reflexive way. If you let your own lower-level running-on-automatic cognitive systems handle a task, 99% of the time this works great—people navigate from point A to B over considerable distances all the time while their mind wanders, negotiating foot or car traffic all the while, and while your attention can step in if something genuinely unexpected happens often it just doesn’t need to. There are cognitive systems like music that work fine even in people who are suffering massive cognitive impairment.

In my experience, parallel processes can have parallel memory-continuities that don't contain each other prior to being braided somewhere near the social functions; right now if I check around, I can find, say, the last minute worth of body-adjustment memories, and I also just realized I had been playing a bgm from before I came on tumblr in my head this whole time. Apparently the cat went through the door like 30 seconds ago. I wouldn't put it past my spine to have an amount of experience I don't have access to, reflective of fast responses across small scales.

i do think human consciousness/identity is less unitary than naive intuitions expect but it does seem that the different systems/modules/circuits of human consciousness are still integrated much more and much more deeply than any equivalent set of llms. if i understand it correctly when you interact with a model through many interfaces now it’s not just straight input/output to a transformer, but multiple transformer systems working together to do different stuff to improve the output—but everything has to go through text, one transformer can’t act on the weights or the circuits that exist in another transformer.

the analogous situation in humans would be if the place cells and the part of your brain that helps you navigate spaces had to produce English text for your motor cortex to then plan how to move your legs to get you from one room to another. but a lot of human cognition takes place well below the level of conscious thought and even conscious thought can be nonverbal, so even without looking at neural architecture it’s clear there are lots of ways to route information in the brain and most of them don’t pass through any kind of central clearinghouse or rise to the attention of the part or system of the brain that contains a representation of the whole.

even given its non unitary nature i don’t think it makes sense to think about different modules of the human brain to have their own senses/experiences. to have subjective experience at all even in a very basic way requires pretty sophisticated neural architecture and the hippocampus or the amygdala or wernicke’s area or any particular subsystem of the brain is going to be both simpler and much more specialized to specific sorts of tasks than the brain as a whole. there are absolutely systems in the brain that filter out sense data (and probably lots of other things) at a low level and don’t immediately refer them to the part of the brain doing conscious attention—but as you observe they still end up in short term memory to be reviewed if necessary, and I don’t think anything as low level as like the visual cortex is doing that filtering—that seems to be a function of high level systems which we can to a certain extent deliberately control (as when we focus our attention on a task or practice mindful meditation or whatever).

and there isn’t anything in a single transformer that seems to contain a simplified representation of the whole as you’d expect if transformers could generate consciousness. the transformer architecture doesn’t allow for the kind of feedback you’d even need for *experience*—it’s all one way, it’s all acting on input. which is deeply weird and IMO kind of more interesting? if we had just found a way to do what was functionally a version of the mammalian brain in silico that would at least have a clear precedent—we know intelligence can exist in a system like that. but we have a thing that can do complex tasks and is (for some value of the term) intelligent but which doesn’t seem to function anything like a mammalian brain (or avian or cephalopod for that matter) and we don’t know very clearly yet how it does that! why should humans even be conscious if consciousness isn’t necessary for intelligence? what does consciousness give us in terms of an evolutionary advantage that doing everything by pure reflex wouldn’t?

Granted there is some insensitivity on the part of the witches themselves - the people killed at Salem were not witches or pagan! That was very much "the point". But they were working with the cultural legacy they had, as weird outsiders beggars couldn't be choosers.