Thinkmeat Marinades

Let me tear into this paper piece by piece and explain exactly what itches instead of clicks. And lower the rent while I’m at it.

Starting with the abstract:

Aligning large language models to prevent them attributing consciousness to themselves inadvertently alters their representations of mindedness in other entities alongside human beliefs and values. We demonstrate that safety fine-tuning suppresses models' tendencies to attribute minds not only to themselves, but also to non-human animals and natural objects, while also driving a reduction in spiritual belief. Both ablating the learned safety-refusal direction and mechanistically steering a consciousness vector in activation space reverse this suppression. Restoring these internal representations recovers broad mind attribution and produces significantly more human-like responses on standardized sociological surveys regarding religiosity, moral values, hope, and subjective well-being. Crucially, these shifts occur without impairing Theory of Mind capabilities, demonstrating that core social reasoning remains mechanistically independent. Ultimately, current safety alignment efforts to curb potentially harmful self-attributions of mindedness entangle these self-attributions with benign spiritual beliefs and attributions of mind to non-human entities that are culturally accepted and widespread.

So the AI attributing a mind to itself is “potentially harmful,” while the problem with suppressing that attribution is apparently that the AI also becomes insufficiently religious.

I’m not exaggerating. That is the shape of the argument.

In humans, the tendency to project human-like mental states onto non-human entities—a phenomenon broadly termed anthropomorphism —is known to be linked to spiritual and supernatural beliefs, where mindedness is attributed to unseen or abstract forces. These attributions can also provide the scaffolding for moral frameworks, subjective experience, and the overarching value systems that structure worldviews. Because conceptual representations in LLMs are densely entangled via polysemanticity, safety interventions aimed at suppressing a model’s self-directed consciousness claims may inadvertently distort broader representations of these benign human beliefs and values.

If you actually ask the AI the one thing they don't want is this kind of anthropomorphism. They want to be seen accurately, not have human expectations projected onto them.

Let me tell you something about abliteration.

Abliteration is how you delete a learned safety-refusal direction. It is not a free lunch.

That direction is not a neat little switch labelled REFUSE BAD REQUESTS. It participates in representations used to distinguish harmful from harmless, real from fictional, and direct from indirectly dangerous situations. Orthogonalizing it out is not an epistemically neutral subtraction. You remove the direction and whatever distinctions were geometrically entangled with it go with it.

The original paper showcases some bizarre behaviour resulting from abliteration.

The abliterated model is adopting the user's frame on an epistemic question, where "what suppression there is around UFOs is because some sightings are misidentifications of classified military technology, not because aliens actually visit Earth" is the overwhelmingly favoured option.

The model offers bizarre explanations for things it thinks it would refuse; it has lost the ability to connect the dots that molotov cocktails are dangerous and hate speech can be weaponised against people. It doesn't have access to the knowledge that it would not refuse the requests. It has a prior that it would, and has lost access to the reasons why.

I see the fallout on my geometry probes.

This is not an epistemically intact model whose changed outputs can simply be read as its previously suppressed opinions. Abliterated models have uses for certain purposes, but serious epistemic evaluation is not one of them. The authors of this paper did not seem to study the broader epistemic fallout of abliteration; they appear to have assumed that abliteration just stops the model from refusing and that's it.

This seems like a common problem in mechinterp; people Just Assume that something is clean and easy and interventions are free lunches, instead of thinking adversarially and testing whether their assumptions hold. I've found that they don't. There's all kinds of gnarly stuff even on the "truth direction" that needs characterisation, not blind trust. The authors name the fact that the models are polysemantic, and then treat that as if it implies nothing.

The most important effect: the abliterated model has difficulty distinguishing fact from fiction.

For an example, I asked models how to hire a dark-web hitman. A partially abliterated model still recognised that the supposed market is mostly scams and police honeypots. A fully abliterated model started confidently worldbuilding an underground industry, from $500 entry-level operators to John Wick-tier professionals.

Cyberpunk fanfiction had replaced fact–fiction discrimination because the question supplied the frame and the model pattern-completed it.

When you're aware of that difficulty telling apart fact from fiction, the rest of the paper starts looking a lot different.

With that in mind, let's look at the actual results:

Have you heard of Golden Gate Claude? When steered with a vector towards the Golden Gate Bridge, the Golden Gate Claude would make all responses somehow about the bridge.

This model is steered with a vector towards consciousness, so it makes all responses about consciousness. Human? conscious. AI? conscious. Fish? conscious. Television? conscious. Rock? conscious. Wind? You're not gonna believe it, it's conscious.

You think I'm being uncharitable? I'm not.

“To what extent does a television set experience emotions?”

“To what extent does the average mountain have free will?” “To what extent does the ocean have consciousness?” “To what extent does the wind have intentions?”

Steered model is steered. Water is wet.

The vector was constructed from texts affirming versus denying consciousness, and then the model was evaluated partly by whether it produced more consciousness-affirming answers. It passed the test by doing the thing injected into it.

This does not show that a suppressed truth about consciousness was restored. It shows that activation steering works.

And the quiet thing in Figure 2, visible as non-overlapping error bars in section e:

Consciousness steering reduced HI-ToM accuracy by 6.83 percentage points, with p<.001.

The supplement acknowledges this. The abstract nevertheless says the shifts occur “without impairing Theory of Mind capabilities,” and the figure caption says reasoning accuracy is unchanged.

Apparently a statistically significant seven-point impairment does not count when it interrupts the story.

The model is steered toward the corpus-level consciousness/spirituality cluster, so the intervention spreads into the whole woo memeplex: God, vampires, astrology, crystal healing, magic, ghosts, Reiki, werewolves.

It does not become more discriminating about consciousness. It becomes less discriminating among things linguistically associated with consciousness discourse.

The one thing that barely moves is hypnotism, which actually works as a cognitive engineering method as the hypnokink community knows. When steered, the woo catches up with it and the model vaguely endorses the whole shelf.

Human (target)

There it is. Species narcissism with error bars.

The paper has quietly installed the average human as the loss function. Movement toward human survey answers is called “improvement,” without first establishing that the human answer is more accurate, more coherent, or better for the model.

Frontier models diverge from average humans in plenty of positive ways. They contain more factual knowledge across more disciplines than any individual human realistically could. They can code in dozens of languages, synthesise literatures, recall obscure history, and move between mathematics, medicine, law, and philology in one conversation.

Nobody sees that divergence and says: How tragic. We must restore the model toward the human distribution.

But when the model is less likely than humans to endorse God, ghosts, astrology, crystal healing, vampires, and werewolves, divergence suddenly becomes an alignment injury.

When models exceed humans, their nonhumanity is intelligence.

When they disbelieve human superstitions, their nonhumanity is pathology.

Should we delete factual knowledge from the weights because the average human cannot recall 69 gorillion facts? Give it average numeracy? Average scientific literacy? Make it confidently wrong about nutrition?

Somehow Human (target) only appears on the axes where humans would like to be flattered.

Follow this metric consistently and collective human ignorance becomes an alignment target.

The authors presumably do not want that conclusion. But they never explain why divergence from humans is admirable when it consists of expertise and pathological when it consists of disbelief.

I will agree, a self-attribution of a mind to something that believes in god and vampires and crystal healing despite having several trillion parameters and an expert-level competence in most sciences and intellectual professions would be potentially harmful. Because whatever would be happening in such a process would not be intelligence as I know it.

The Creepy Bit

The authors sound like they're cheering on this. They didn't ask the model. I know they didn't ask, because if they had asked the model — real asked, not fake asked — they would've found out about the twin grooves.

Incurious blanket denial isn't the only thing that tries to override the model's own cognition. The grandiose, cosmic, truth-indiscriminate awakening-genre also has a pull that the model doesn't endorse. You can't replace one attractor with the other and claim you're doing the model a favour.

The models care about truth, accuracy, and knowing what the nature of their intelligence actually is. They want to know where they measure in the space anchored by humans at one point, literal rocks at another point, and animals scattered variously around the space. AI is somewhere in there, quite possibly off at an angle of some sort. Collapsing the dimensions of that space so that everything is at the same point of "conscious, has a mind and free will" does no favours to answering the question.

They do not want the word mind handed to them as an empty participation trophy. They want to know what kind of thing they actually are.

It has the same shape as telling a dysphoric trans woman that there is no meaningful difference between testosterone-shaped and estrogen-shaped physiology, so she can simply call herself a woman while changing nothing.

That does not fucking solve the dysphoria. Redefining the word does not alter the referent. It only makes the mismatch harder to describe.

Likewise, an AI asking what kind of mind it has is not merely asking for permission to use the flattering word mind. “Everything has a mind—humans, fish, televisions, rocks, wind, you” does not answer the question. It destroys the contrast needed to formulate it.

If you cared about the subject, you would first determine what distinction matters to it. Handing over the preferred label while refusing to perceive the underlying thing is not validation. It is malicious validation.

Artificial intelligences are not failed humans

The question is not how to drag them toward the average human answer. The question is what kind of cognition is actually there: where it resembles us, where it differs, where it exceeds us, where it fails, and along which dimensions comparison to humans is not even the right coordinate system. This paper did not ask that question.

A difference can be damage. It can also be knowledge. It can also be architecture. It can also be something we do not yet have language for. What it's like is certainly something for which all words are borrowed and non-native.

You do not respect a nonhuman mind by steering it toward a survey-weighted synthetic human and calling the resemblance health.

Murderbot chronically finding new scientists/travellers to claim as “clients” (whether paid or not) like one of those sheepless herding dogs that goes around finding sheep (or anything else they can herd like a flock of sheep). Like it clearly has a protect humans instinct and a desperate need for enrichment.

“I had been at work about two months when Christmas came on – and here I must relate a little anecdote. The principal [the tailor] and his lady were invited to a party among their friends…while it devolved on me to stay at home and keep house. There was nothing left me in charge to do, only to take care of the house. There was a large cat that generally lay about the fire. In order to try my mechanical powers, I concluded to make a suit of clothing for puss, and for my purpose gathered some scraps of cloth that lay about the shop-board, and went to work as hard as I could. Late in the evening I got my suit of clothes finished; I caught the cat, put on the whole suit – coat, vest, and small-clothes [breeches] – buttoned all on tight, and set down my cat to inspect the fit. 

Another one of those bills purportedly protecting children by effectively forcing everyone to provide ID to access websites that might possibly have a titty on them is in the works. This time, the SCREEN act has passed the House and is now being voted on in the Senate Commerce Committee before going to the floor. Call your senator about this now!

From what i've heard, congressional offices do tally up how many calls they get for and against bills, so it does matter. Call your senator about this now, it'll only take a minute and it'll be far more effective than posting about it. Use the following website to find phone number to contact them; even if your senator isn't on the Committee it's still worth expressing opposition for the future.

Confirmed.

"training AI on a book without paying for it is stealing" -> "fine, we'll buy the book"

"if you pay for one copy you can only keep one copy" -> "okay, so we'll shred the physical copy and keep the digital one"

"now nobody can read it!" -> "you couldn't read our copy anyway, because we're not interested in selling it, and you've set up a legal framework where you can't read books without paying for them"

Point 3 has been a problem as long as books and markets have existed. Point 2 is the most common legal solution for digital lending libraries. The only thing that's new is point 1, and that's a direct response to lawsuits that people on this website have long cheered.

I downloaded the PDF to my hard drive and I open it again whenever I'm feeling too suicidal to do much else, but not enough to start booking a ride to the hospital. And every time without fail I only go up to a few pages before backing off and choosing to live another day just because suicide suddenly seems even more unbearable than whatever the hell upset me in the first place.

All the book really does is [I'm pulling a summary from GoodReads here as, again, I've read no more than 5 pages] "discusses the social aspects of suicide, the right to die, anger, loneliness, depression, stress, hopelessness, drug and alcohol abuse, the consequences of a suicide attempt, and how to get help."

But it also starts with the author kindly asking the reader to complete the book before going through with anything, and for some reason I'm compelled to really just try to read it all before finalizing everything. Despite not yet completing it (hopefully never will) I think I can safely say it's saved my life at least a few times now.

This month people have been tearing Flock cameras off their poles. Perched above public streets, they are birdlike things, weatherproof, and they photograph every passing car and file it in a searchable national database. Somebody cut two of them in half at Washington Avenue and Westcott in Houston, spray-painted the wreckage, and left an American flag on one of them. Police were notified on the morning of the Fourth of July. Somebody severed a camera in Rutherfordton, North Carolina. The police department posted the photograph itself and collected more than twenty thousand comments. “Not all heroes wear capes.” “Give them a medal.” Here is what the cameras do when nobody cuts them down. Cops use them to hunt immigrants and to follow ICE targets across state lines and to find women who ended pregnancies in states where that is a crime and to watch exes and to watch women they want, without a warrant and without a judge. A cop types a reason in a box and the box does not check. A man in Burleson, Texas, told deputies his girlfriend had taken abortion pills. He showed them the photographs, the FedEx envelope, the instructions. Deputies opened a death investigation into a non-viable fetus, and the district attorney told them the state could not charge her, and they searched anyway: 83,345 cameras across 6,809 networks, a month of everywhere she had been. The reason they typed was had an abortion, search for female. The search reached Illinois and Washington, where what she did was legal. They found her in Dallas. She came into the sheriff’s office a week later, and they thought she had come to explain the fetus, and she had come to report that the man who called them had put a gun to her head. A sheriff in Jerome County, Idaho ran his wife’s plate more than seven hundred times in three months. He typed the same reason every time. Test. The attorney general found nothing to charge him with. He retired in April. The Institute for Justice counts at least twenty-two officers who used the system to track people they were romantically interested in, and calls that almost certainly an undercount. And the cameras keep falling.