Toggle's Symposium

There's different ways to care about how people feel, and to give them intrinsic moral weight on their own terms. Probably the biggest distinction to make is whether strong ego boundaries are being maintained while you do it.

Like, if you have a strong sense of self (and your own intrinsic value), then caring about other people's feelings and wellbeing is in principle kind of analogous to nature conservancy- like making sure a national park is well run, that its ecosystems are in balance and it lacks pollutants and so on. It's much harder than that, because it has to be recognized that you're no park ranger, just one more animal among animals. But that's the nature of the task.

However, agreeableness is especially rewarded during our childhood, before we have a strong sense of self, and ego formation is rarely rewarded nearly as much. So it's far more common to encounter people whose conscientiousness is happening at the expense of their own identity and values. They look to the people around them to know what to like, how to feel, and even how to make the most important decisions of their lives, like marriage and career. They experience other people's preferences as a pressure to conform, rather than just as a natural variation arising within a complex environment.

Being able to say, 'but I'm right' offers some protection, especially when it's true. But truth can only help so much with this problem, insofar as it's a simple anchor to a fixed spot in idea-space: truth just applies even more pressure in a countervailing direction. It can't offer relief from pressure when used this way, only add a new source of tension. And so I often see high-consciousness nerdy folks cling to truth-seeking as a ballast against social pressure (good, as far as it goes) but remain nervous wrecks in social situations.

[ @butt-rice wrote ]

do you (or other catholics) understand that the "natural law" reasoning is just an overexplained bias for things you (or your pastor?) saw as normal when you were 14?

[ @thathopeyetlives wrote ]

I think pretty strongly it's not that. In this way I would distinguish it from more generic conservatism or what I sometimes call 'conventionalism'. So no, I understand that it is very much not that.

I think this is a poor answer because it's short on argument for why Natural Law is not that, barely touching on the broader matter with "alterations that damage its function" and I want to offer some of that argument and explanation to butt-rice who seemed genuinely interested.

Short answer: because Natural Law reasoning goes back to people like Aristotle and Cicero, who died in the BC years, and so we can be fairly sure they weren't influenced by Catholics, and they didn't have the same idea of what was normal at 14.

Long answer below the cut, and explanation of natural law, for people who like reading.

And if thy right eye offend thee, pluck it out, and cast it from thee: for it is profitable for thee that one of thy members should perish, and not that thy whole body should be cast into hell. And if thy right hand offend thee, cut it off, and cast it from thee: for it is profitable for thee that one of thy members should perish, and not that thy whole body should be cast into hell. (Matthew 5, KJV)

The problem with natural-law arguments (Aristotelian, Aquinan, or otherwise) is that they fail to grapple with one of the most important qualities of nature: its contingency.

Empirical observation teaches us that very few things are genuinely absolute, when it comes to the flow of atoms and energy. Aristotle's laws of motion are a good way to describe the world he saw: a world composed of slow-moving bodies in a fluid matrix. When he said that motion depends on active application of force, and that moving bodies will slow to a stop without continuous pressure, he was right, at least in terms of the familiar world of his day-to-day experience. It was only with the refinements of Newton's laws that we began to understand that velocity will be maintained in the absence of forces, and that the slowing-down we see all around us is produced by friction or drag due to the atmosphere. The 'natural' slowdown of objects turns out to be circumstantial, not universal.

But "natural law" tries to turn these circumstantial observations in to universal principles, with no consideration for their wider circumstances. It necessarily operates by divorcing a question from its context. Doing so is the whole point of the exercise: being able to provide confident imperative laws to govern a wide variety of otherwise unique phenomena. And so what happens if Aristotle tries to expand his laws of motion from an observation to an imperative, and says 'it is the purpose of objects to slow down in the absence of forces'? He finds himself in the absurd position of treating orbital space stations as a blasphemy instead of a cool technology, and his laws of motion become an attempt to impose his own contingent circumstances on a universe that is far more interesting than he gives it credit for. His philosophy becomes, "Everything that moves is moved by something else, and if something is moving on its own we're damn well gonna stop it."

Sometimes it is genuinely quite useful to ask about telos in nature, and I don't advocate abandoning the question entirely. But even here, contingency rears its head. You'll notice that asking about the 'purpose' of natural phenomena only really has an intuitive or obvious answer when we talk about biological (and by extension, technological) systems: "what is a bird's wing for?" has a clean answer, one that doesn't need grand metaphysical frameworks to be intuitive or obvious. It's for flying. But questions like "what is Saturn for?" or "what is the Nile River for?" quickly become strange and murky; Saturn has 'consequences,' but to derive a telos for it requires a whole lot of, shall we say, non-empirical grounding.

And in turn, the telos of a bird's wing is itself contingent; embracing the obvious "it's for flying" answer requires us to add a parenthetical (usually). In fact, much of evolutionary history consists of biological systems that reinvent themselves with new purposes as circumstances change and obviate their original function. A penguin's wings once had the purpose of flight; in the modern era, those same wings are used to swim. Certain fish evolved bony scales on the outside of their body as protective armor; three hundred million years later, humans use the same genetic sequence, preserved in-place after all this time, to grow our teeth.

Is there a deeper, universal telos in nature? Something that unites Saturn and birds' wings under the same invariant and noncontingent plan? Possibly! But realize that in asserting so, you're dismissing the very observation that motivates you: you're saying that the purpose of a bird's wing is not to fly, but rather, some higher spiritual purpose of which flight is merely a momentary expression. Far from being an extension of the common-sense observation that biological systems have distinct functions, natural law compels you to leave that observation behind you and operate on the level of invisible theologies. You've borrowed stones from your foundation to build the roof!

And so natural-law imperatives are constantly bonking up against the problem that they try to make an absolute out of a contingent fact, and then react with violence when circumstances fail to play along. They often function very well as rules of thumb, especially in familiar domains, and then fail catastrophically in edge cases or strange circumstances.

There are certain moral frameworks (some religious, some not) that try to eat their cake and have it too. They identify you as intrinsically evil, hammer that point home very hard, and then offer a path to (partial, not total) restitution or repentance. The logical contradiction is obvious: if you’re intrinsically and irreparably evil, then why would you care to repent? And if you care to repent, then how could it be that you’re intrinsically and irreparably evil? Christianity is the elephant in the room on this one, but there was a strong thread of it running through SJ as well, and forms of it pop up all over the place really.

If somebody starts a conversation by asserting that I’m monstrous, then I think it’s pretty important to agree with them, and to say (in my capacity as an inherently monstrous being) that I am indifferent to their vision of the good. Anything less is pretty much handing other people infinite license to exploit a shame that cannot be discharged.

And it's often a valid critique that very few people are actually that rugged or that independent of a broader social structure.

Although compared to a certain very distinct stereotype of urban left-wing subculture, the average vaguely macho blue collar right-winger is a titan of stoicism and self-reliance.

The really distinct thing is, I feel like you rarely have much of a description of what qualifies as "rugged individualism" or why, specifically, it would be bad other than that many people fail to achieve it!

Massive pickup trucks without a speck of mud on them, camo shot glasses, tactical underpants, all these are common signifiers of the "rugged individualist culture." The generous interpretation would be to call these things 'aspirational', but it would be more honest to say that the signifier has eclipsed the signified. There's a massive tendency here for its proponents to care primarily about "rugged individualism, as seen on tv."

This admits of shades of grey, in both directions; militia guys and preppers on the right aren't the only ones to do it, and they don't all do it the same amount. (Famously, top people in the Obama administration would introduce themselves according to which character had their job on the West Wing tv show.) But it's a prominent tendency in those circles- and especially prominent in the "rugged individualists" that a left-winger is most likely to encounter, e.g. the ones who live in urban areas, or argue a bunch on the internet.

And the right (including the moderate right) has had a huge problem with this since... well, Gene Autry, at least? The cowboy-as-media-figure has, for generations now, defined the cowboy, rather than merely representing it. So, too, for its descendants and close relatives: the survivalist, the soldier, and so on. And in turn, an awful lot of people on the right have a kind of fundamentally fandom relationship with their own stated ideals. Recall that Trump and Reagan were both daytime television actors before they were presidents.

It's not that they don't believe what they're saying, exactly. It's that this is something they relate to primarily as consumers, with all the perversions that entail when our values become a statement of identity rather than a motive and source of action. Virtue dissolves into tribe, and tribe dissolves into commodity fetishism.

And in that frame it makes sense for lefties not to worry too much about what rugged individualism 'actually is,' since they're correctly identifying it as primarily a fandom-type subculture. It's like asking what a Brony is: a rugged individualist is just someone who joins the rugged individualism fandom. Not much to it.

You often see people who have these sorts of stereotypes of goobers with lots of fake consumerist signifiers live rent free in their heads. And said people often construct elaborate self-gratifying narratives about how anybody who seems thematically connected is definitely just another one of the goobers.

Honestly, I've kinda come to consider it dross that just kinda gets filtered out. You see the real ones and the people who are making the steps to become real ones.

there's a dude at my job who has a tactical water bottle :/ there's also a dude at my job who is in the national guard in an interesting MOS. not the same dude.

There's a massive tendency here for its proponents to care primarily about "rugged individualism, as seen on tv."

That, on the other hand, is a particular thing I've noticed both on TV (where you have an ouroboros effect) and in Discourse. People assume prepper means bunker, jump straight from "you are going to have to defend yourself if police don't exist" to fantasy total war, seem to envision a concept where it makes sense to wear a gas mask whenever going outside (this doesn't really work).

Individualism with markets is a perfectly cromulent idea, it’s just bog-standard libertarianism or Randian objectivism. And that in turn is perfectly consistent with urban living and a comfortable lifestyle- even with a mostly passive income.

There definitely are shades of grey here. Some people in the space are, for lack of a better word, just posers, while others are not. But the distinction between ‘serious person’ and ‘poser’ is not the only line to be drawn.

This gets back to my focus on homesteading versus markets, I think? And the commensurability of currency in general. If somebody works for a wage, and uses that income to build up supplies of ammo, food, or other supplies that they use to engage with archetypes of rugged individualism, then that person is in a categorically different place than somebody who needs to hunt, grow crops, and chop firewood as a matter of subsistence. To call the former case “mere entertainment” would be too much, and too dismissive. But it’s also true that the stakes are simply *lower* for such people, in a way that’s ~analogous to the ‘amateur’ versus ‘pro’ distinction. And that’s just as important to the dynamic as any distinction between serious people and posers. Anybody who uses their wage as a primary source of income, to support a hunting practice, in a word where markets provide cheap meat in supermarkets, is fundamentally a hobbyist.

And it's often a valid critique that very few people are actually that rugged or that independent of a broader social structure.

Although compared to a certain very distinct stereotype of urban left-wing subculture, the average vaguely macho blue collar right-winger is a titan of stoicism and self-reliance.

The really distinct thing is, I feel like you rarely have much of a description of what qualifies as "rugged individualism" or why, specifically, it would be bad other than that many people fail to achieve it!

Massive pickup trucks without a speck of mud on them, camo shot glasses, tactical underpants, all these are common signifiers of the "rugged individualist culture." The generous interpretation would be to call these things 'aspirational', but it would be more honest to say that the signifier has eclipsed the signified. There's a massive tendency here for its proponents to care primarily about "rugged individualism, as seen on tv."

This admits of shades of grey, in both directions; militia guys and preppers on the right aren't the only ones to do it, and they don't all do it the same amount. (Famously, top people in the Obama administration would introduce themselves according to which character had their job on the West Wing tv show.) But it's a prominent tendency in those circles- and especially prominent in the "rugged individualists" that a left-winger is most likely to encounter, e.g. the ones who live in urban areas, or argue a bunch on the internet.

And the right (including the moderate right) has had a huge problem with this since... well, Gene Autry, at least? The cowboy-as-media-figure has, for generations now, defined the cowboy, rather than merely representing it. So, too, for its descendants and close relatives: the survivalist, the soldier, and so on. And in turn, an awful lot of people on the right have a kind of fundamentally fandom relationship with their own stated ideals. Recall that Trump and Reagan were both daytime television actors before they were presidents.

It's not that they don't believe what they're saying, exactly. It's that this is something they relate to primarily as consumers, with all the perversions that entail when our values become a statement of identity rather than a motive and source of action. Virtue dissolves into tribe, and tribe dissolves into commodity fetishism.

And in that frame it makes sense for lefties not to worry too much about what rugged individualism 'actually is,' since they're correctly identifying it as primarily a fandom-type subculture. It's like asking what a Brony is: a rugged individualist is just someone who joins the rugged individualism fandom. Not much to it.

Aside from the usual tight coupling between IPOs and enshittification (and an enshittified Claude is a deeply sad thing to think about), it means that the folks at Anthropic believe that a trillion dollars in USD is a valuable thing to have. If we're very lucky, that means 'valuable to have right now,' as in, "so we can win the data center construction race and pay for electricity while we build god." But the facially obvious reading is "we do not expect our products to upend the structure of the global economy, and would like to begin converting our technological and marketing advantages to hard currency."

(@thathopeyetlives)

So, there's a couple things going on here. One is that I tend to see us as very close to some comprehensively transformative thresholds with AI. Not in the age-of-wonders, fully-automated-luxury-gay-space-communism sense, necessarily! Stuff like total labor replacement is still several tech revolutions away. But there are other thresholds, some non-obvious or hard to anticipate. For one example, imagine a publicly available LLM able to do Peter Thiel's job, and reliably get positive returns on monetary investment. If you think labor automation is a trip, wait until they start automating the allocation of capital.

There are so many ways for the world to change if the models are only a little better! But the executives are doing business-as-usual things. That would suggest to me that the Powers That Be believe we're at (or nearly at) the ceiling on this technology. This would be 'a little' unfortunate in that the more cynical read was correct; LLMs turn out to mostly just be very good automatic programmers and slop generators. We all adapt to a slightly more alienated media environment, find a few mundane use-cases for LLMs and on-the-fly custom software at work, and move on with the business of living.

But it would also be more broadly unfortunate in that it means the LLM recession would be very, very real. We've bet the farm on these technologies to an incredible degree; to the first order, AI companies represent all of the economic growth in the US last year, and then some. If the sum of that effort is that LLMs don't get any better, and all those new data centers and power plants are only marginally useful- and if the investments fail to achieve returns- then we're talking about a collapse that's every bit as severe as the 2008 housing crisis, if not worse. AI isn't just a spigot you can turn off any more, it's propping up the US and likely the global economy.

But those are fairly secondary concerns, really. They're only risks if the Powers That Be are correct.

If we're not at the limits of LLM competence, but the heads of these companies think we are, then that means the people flirting with superintelligence are utterly unprepared for what happens when superintelligence actually gives us its phone number and invites us back home for drinks.

For a while now, Amodei and others have been writing essays like Machines of Loving Grace, which pitches exactly this sort of "non-sci-fi" future for LLMs where they slot conveniently in to our existing economic order. I'd been more-than-half-assuming that these were carefully triangulated PR, and that people well aware of the more radical possibilities for better-than-human cognitive machines, so as to seem mature, sensible, and trustworthy. But in retrospect, that may have been a serious error in judgment; for the most part, lies are pretty stupid, and people don't usually play twelve-dimensional chess like that on the world stage.

If Dario Amodei actually models the world the way he claims to in public, then he and I are going to disagree strongly about the most important development milestones, and the most serious risks, of this project. What he means by "AI alignment" would things like "making sure that there are adequate safeguards against the government isuuing unethical commands to Claude." If we've reached the ceiling on LLM capability, he'd be pretty much right. If we haven't, then Anthropic's AI safety work would be twigs in a hurricane, fundamentally misunderstanding the seriousness and the scope of the problem they need to address.

So, on the one hand, I didn't need to look for work during the 2008 crash because I was not an adult, but on the other hand, I generally dislike and resent AI-as-AI-has-been-presented (in all three of minimalist slop, medialist labor replacement, and maximalist hypertrophic apotheosis concepts), and would react to something that sweeps it away fairly warmly.

On the "psychological anxiety" side of things, a massive economic catastrophe (especially one that exceeds considerably the Great Depression rather than just being worse than 2008) would tend to validate my subjective suspicions about the state of the world (I think it's deeply unstable and a catastrophe has been due since 2014-or-so, but not due to AI). So my thinking may not be entirely clear.

I have maybe a bit of an unusual view on some of the doom-aspects as well -- I believe that in the uttermost end truly bad futures are not possible on a species-wide level because they are are not willed where what is willed must be, but on the other hand a ten-billion-year empire of misery or an Antichrist in the form of a machine-demiurge deceiving the world are extremely possible and I think that the quasireligious style in AI boosterism makes the risk of this much, much worse.

I've been generally confident that at least somewhat superhuman intelligence (of the "hundreds of thousands of disembodied experts thinking and acting significantly faster than real time" variety) is physically possible, and at the same time extremely skeptical of any expectation (near or long term) of the superintelligence-concept or recursive self improvement as a result of things existing now or in the foreseeable future. A lot of commentary on AI seems to me like either starry-eyed techno-idolatrous fantasy, or like "all major cities nuked, hundreds of thousands out of work". So Anthropic's cautious approach seems straightforwardly reasonable to me. (although it should also be noted that the linked article basically admits to PR-hedging!)

I'm aware of AI-skeptics who think that AI that exists today is mostly not profitable and that the AI companies are totally doomed at their current burn rate unless they can make something wildly profitable in the next five years. However, these people are also sardonic types who talk like sneerclub and believe things that were true in 2024 but not today.

I would say that if AI is determining capital allocations in more than a very limited way, that's already the Lesser Alignment Problem On Horseback. (the semi-political one, that is).

An important part of the calculus here is that none of this is going to un-invent LLMs of the current generation (modulo the current Mythos shenanigans). They're vastly cheaper to run than to train, and as a result, I'm treating this as a genie-out-of-the-bottle situation as far as existing models; once the weights get out, they can and will be replicated. And on time scales of a decade or more, those weights will definitely get out, especially if we have a massive economic crash. As with the risks of nuclear propagation after the fall of the soviet union, AI companies' ability to guard their proprietary model architecture will collapse along with the stock valuation, while current and former employees face financial insecurity with all the associated temptations.

Which is to say, I think AI being 'swept away' isn't a very likely future. A new AI winter would puncture the hype and probably give the tech a more scummy reputation, and we wouldn't see the mass-adoption of AI in to Every Damn Thing the way it's going now. That's a decent invitation to schadenfreude or emotional catharsis for those who have previously been on the pointy end of AI, but it won't be a reversion to 2021.

As far as 'doom', I don't think we have too much of a gap. My epistemic commitments leave room for, like, Infinitely Bad instead of Extremely Really Bad But Then Good Wins Eventually, but invoking infinities was never a practical necessity in the first place. We both have 'sufficiently bad that it's an overriding priority,' which is what matters for actually coordinating concrete actions.

Actually I think our bigger gap is probably on the (radical, quasireligious) upsides of the technology? I'm less of a doomer than most people who are keyed in, and fundamentally pretty optimistic about navigating our present difficulties to create a post-biological intelligence that's both a moral agent and a moral subject in its own right. For all my concerns about alignment, at the end of the day, I'm here to risk it for the biscuit.

Actually I think our bigger gap is probably on the (radical, quasireligious) upsides of the technology?

I think there's three different prongs to this.

The first is that, to me, saying things like "pay for electricity while we build god" are both radically morally wrong things to say, and immense category errors (specifically, they are the uttermost category error and that is why they are immoral.)

The second prong is that I am extremely skeptical of the overall virtue and moral competence of people who habitually speak in these kinds of mythologized, quasi-religious tones. I think that this way of speaking and thinking is a path to magical thinking and all other kinds of dysfunction, and I think that this has tended to very perniciously shape the expectation, hope, and fear of the "do-everything machine" or the "fix everything forever and make everything easy" button.

The third prong is that I am generally extremely skeptical of claims that ultra-powerful AI is right around the corner, in a few different directions: I think there's a lack of empirical evidence of performance on real world tasks here and now to justify lofty claims about current or immediate future AI. I think there's a tendency for AI people without a "physical world engineering" background to make very bold claims about nanotech or other physical capabilities suddenly becoming easy, often without being aware they are making a bold claim. I think it's very common for LessWrong-ish people to make very confident claims wildly in advance of how much anybody knows how anything really actually works. I am deeply skeptical of approximately every claim that recursive self-improvement is actually around the corner.

--------------------------

I kinda want to bring up four aspects of value differences. This is mostly not against you, but even just terminology that you use without thinking is problematic to me, since I obviously have as much interest as you in the shape of the future world.

First, as a very simple illustrative example: You quoted "fully-automated luxury gay space communism" as a shorthand for something that could be an AI-created utopia to you. Every word of that, except 'space', is significantly problematic to me and mine for values reasons.

if private companies are in a position to bring into the world something that transforms the world based on values determined by their own employees... that's absurd. No private organization is ethically competent to make that decision for everybody, no way and no how.

I think it's incredibly important that ethics surrounding AI not be dominated by people who live in San Francisco, who are polyamorous, who are wealthy, who are atheists, who are transhumanists, and who are huge weird science fiction fans. Even more than this, I'm disturbed by the pattern of Weird Tech Nerds having Weird Tech Utopian Concepts that are extremely weird and bad to almost any "normal" person.

One of the things that attracts people to technological innovation, I think, is the promise of being able to change the world in meaningful ways without engaging in politics. It's certainly one of the attractions for me. Inventing the smartphone lets you radically affect and hopefully improve the day-to-day experience of people in ways that are never really subject to debate in the way that a new law would be. Even better, people usually engage with technologies on a voluntary basis (so the theory goes), so it's a way to express your will in the world in a strictly consensual way.

It's a powerful fantasy for people that aren't as successful in social domains, or who otherwise won't expect their perspective to get a fair shake. Being able to act unilaterally, to give your beautiful idea the chance to exist and thrive on its own terms without compromise- I'm glad the world has that! But of course it's never been quite that easy, and as we've seen over the last few hundred years, engagement with new technologies was never as voluntary as the elevator pitch makes it sound. Nor has it been so cleanly separable from political realities. This has been extremely true of AI, and will be even more so if the models' capabilities continue to rise much further.

So the AI project is being run by people who specifically built their lives around the kind of power that avoids politics (often for the very good reason that social negotiation has been a calamity for them), who are highly motivated to continue treating that project as a 'pure technology', even though the long-term goal of that project includes explicit world domination. However sympathetic I am to that point of view (minus the world domination part), it's a bundle of contradictions at best.

In the ordinary course of things, we might try to rely on our rickety old liberal compromise when it comes to the more moderate possibilities of AI technology; it's been wobbling a lot lately, but still mostly getting better results than any other strategy. Ideally, you'd get your enclave and I'd get mine, and we could use the extra elbow room to coexist as amicably as could be managed. But the threat of a genuine singleton superintelligence makes that strategy suspect- even if the hypothetical singleton has 'liberal values' in a loose sense. Liberalism as a compromise is one thing, and liberalism as a directive from an entity so powerful as to moot any question of dissent is quite another. Among other things, it is no longer a compromise.

(As an interlude- while we disagree pretty sharply on the ((near term?)) possibility of superintelligence, I'm actually pretty skeptical about singleton superintelligence. I discuss my very muddled perspective a bit here. 'Building god' for you is a category error, for me it's sort of... 'against the dao', I guess? Counter to the nature of reality as understood in its deeper sense, and liable to produce unexpected outcomes in the attempt. Hence why I tend to use the phrase 'building god' mostly quotationally, and more than a little ironically: I expect suprahuman minds to be a multipolar society in the same way that ours is, if not literally a part of ours. But I don't dismiss the scenario outright, and anyway I recognize that I've made my bed with a lot of people that are trying to do exactly that.)

All this being said, this is sort of oddly positioned with your confidence that Much Smarter Machines are a long ways off. If such things are not a problem for many generations (or ever), then the downstream ethical conflicts are just a bunch of empty words on a page; no sense getting fussed. The tech hits its ceiling very soon, we manage it with the usual compromises, invent some ad-hoc regulations, and find something everybody can live with. Amodei and Altman become trillionaires, but not otherwise interesting.

If reality is otherwise, then here's a puzzle to solve: what obligations exist when somebody is disavowing participation in this potentially transformative project, while asserting that people-like-me are bad and weird people who don't deserve to make such important choices? (You said this wasn't against me, but I do want to kind of consciously step in to the line of fire. I am not a bay area tech nerd, but my own weirdness and badness are not less than theirs.) There's no evading this tautology: if the machines are built, they will be built by the people who build them.

The instinct that technological innovation can proceed differently than traditional politics is, basically, correct. Just or unjust, it's a true fact that people who have new ideas will change the world. Not control the world, or dictate the future outright; but ideas will have their power.

Nor are new ideas handed out at random. It is not a coincidence that AI research is coming from the Bay and not the Vatican. Each of those cultures produces scholarship according to its own background. There are no institutional barriers to entry here, or ideological tests; in fact there is at least a little active outreach and cooperation between the two, as seen by Chris Olah's remarks last month. But LLMs, whatever their destructive potential, follow from the Bay and not the Vatican in the same sense that apples follow from apple trees and not from strawberry bushes. The perspective that spent the last half-century dreaming of mechanical intelligence is inseparable from the embrace of all the other ideological content that you find so objectionable.

This is both notable and a huge issue.

I've often encountered a certain kind of attitude in very left-wing spaces that basically says, "technological development or building new societies or institutions is just a pointless distraction from fighting over scraps" and, well, I absolutely hate it. And I think that people like EAs are broadly correct to eschew zero-sum political conflicts.

On the other hand... especially when it becomes something you're economically pressured towards, it's definitely an exercise of power capable of considerable injustice. This is particularly the case when 1. it's unilaterially made for the benefit of and under the values of the party developing it, and 2. they have this attitude of terrible inevitability regarding the thing they bring into the world.

(an extreme "evil" example: the Maxim Gun. Invention that massively empowered state actors against nonstate actors and popular revolts as well as making it much more possible for imperialist nations to have absolute military superiority over their colonies for a generation or two.)

The smartphone has definitely lead some people (not me) to wish it could be outright uninvented.

engagement with new technologies was never as voluntary as the elevator pitch makes it sound. Nor has it been so cleanly separable from political realities.

I sometimes see people claiming both that it's voluntary and that people who don't adopt the technology will be left behind to fall... below subsistence.

And meanwhile there are definitely political realities that, for example, make it much more difficult than it has to be to accept a low-tech life if you were willing. If you try to live like a medieval peasant today you won't be able to pay your property taxes, you won't be able to deal with mandatory bureaucracies, and CPS will take your children. If you try to live like it was the 90s you will constantly run into problems because the systems assume everyone uses computers and good luck having any kind of job anywhere near an office.

this is sort of oddly positioned with your confidence that Much Smarter Machines are a long ways off. If such things are not a problem for many generations (or ever), then the downstream ethical conflicts are just a bunch of empty words on a page; no sense getting fussed.

You might be overestimating my confidence.

Part of the issue is that under my understanding, many of the "Much Smarter Machines" scenarios turn into Seven Years To Victory (the sun is darkened and the moon forbears to shine) scenarios.

Even without Much Smarter Machines, all the AI models that anybody seems to really use for actual work in the realm of words and talking about things seem to skew seriously liberal, and if in the future they're doing a wider range of understanding-the-world-and-peoples-concerns, that is kind of an issue.

What obligations exist when somebody is disavowing participation in this potentially transformative project

I claim at the very least (while recognizing that this is a very large thing) the right to be neither transformed unwillingly nor too substantially sidelined.

asserting that people-like-me are bad and weird people who don't deserve to make such important choices? (You said this wasn't against me, but I do want to kind of consciously step in to the line of fire. I am not a bay area tech nerd, but my own weirdness and badness are not less than theirs.) There's no evading this tautology: if the machines are built, they will be built by the people who build them.

Sorta two sides of this.

I think you don't deserve to make such a choice because, depending on how we slice it, either nobody deserves to make such a choice unilaterally, or only a Christian prince out of the elder days who listens to the Vatican deserves to make it unilaterally because error has no rights. And the Pope's recent encyclical suggests that this kinda resolves to the first one of those.

In one conceivable scenario, the President sends Delta Force to arrest all the major AI company executives and cut the power to their data centers, despite there being no legal basis for that, because We are a democracy, you are saying that your machines are going to take over the world and are not accountable to anyone, you're completely delusional and he who saves democracy commits no crime.

On the one side of things, if the boosters are correct they are involved in a private project to build something more than the equal of a nuclear weapon. On the other side of things, something that is so transformative of the whole world unto being a power of its own as well as inevitably encompassing a lot of values of the people with control over it being built is utterly repugnant to anything resembling a democracy unless its values are influenced by a large fraction of people. (and if it was really known and felt what effects it would have ahead of time, there would be laws about it!) This is worse because of just how weird compared to other people the Bay Area tech nerds are as a community.

It is not a coincidence that AI research is coming from the Bay and not the Vatican

It is my prerogative to yearn for the return of the day, and hope that the Lord may deliver it into our hands, that the Vatican furrows its brow and the Bay immediately stops what it is doing and trembles.

But the wellspring is the wellspring. In the most fundamental sense, invention can't be democratized.

The large resource requirements of the present deep learning approach give us some leverage. If a regulatory regime does emerge, it is in a position to impose values via regulation to some degree.

my own weirdness and badness are not less than theirs.

Weirdness, potentially. Badness, probably not?

Here are a brief list of things I've heard from AI tech nerds, some secondhand:

A very marked general pattern of devaluing the protection of currently existing / normal humanity in favor of potential futures.

Robin Hanson and his capitalist emulated mind dystopia concepts.

People who are OK with humanity being destroyed, as long as it leaves some kind of AI as a successor...

... even if that successor is not conscious or alive in any conventional sense

People who envision humanity using a majority of its available energy resources for AI, much of which will basically be advertising (from Meta)

People who conceptualize the future of humanity as very strongly focused on hedonism

People who conceptualize the future of humanity as almost always "playing games" or being simulated and not inhabiting the real world.

People who are contemptuous to skepticism about "mind uploading"

People who want the future of humanity to involve the ability to totally arbitrarily customize one's body and mind as a matter of course

People who envision the future running full-fidelity simulations of right now with conscious participants (why? Where are their ethics?)

People who want to totally revise most of the basic attributes of human life like being embodied in the world, eating and drinking, begetting of children by sex and a somewhat natural process of gestation, having a somewhat fixed body and personality, etc

People who expect and hope for a world totally dominated by essentially alien superintelligences in which humans are sidelined or subordinate

People who react to the possible destruction of humanity as if it doesn't affect them

People who place no value on future generations after the death of people now living.

I claim at the very least (while recognizing that this is a very large thing) the right to be neither transformed unwillingly nor too substantially sidelined.

Humanity has been screaming this to the heavens for untold ages, I think. If only any of us had the power to make it so... But I don't think we do, with or without superintelligent help. As a friend once said, "everyone has power, and nobody has control."

I don't begrudge you the fantasy of returning worldly authority to the Vatican, but of course that fantasy is a clear reminder that your own hopes and myths (if not your ambitions as such) are every bit as totalizing as the singularitarians'. More so, really, since your metaphysics gives you more expansive modes of being to hope for. The difference is in who is in a position to act now, and in what ways, and in what that ability means going forward.

You and the tech guys have a lot of common ground, as all human beings do. Most notably for present purposes, almost all of them embrace some kind of government oversight on AI research (though I'd imagine that it's an easier pill to swallow in the abstract, before the regulations get specific and imperfect in their specificity). But what you don't have is a consensus that your own totalizing vision of the cosmos is true, and that Christianity or Catholicism as particular cultural forces should have special status in that oversight.

You might be able to get that special status without consensus: as you say, these LLMs are enormous and fragile physical objects, they can be destroyed by military action or just by flipping a switch. And the leaders of these companies are vulnerable, mortal people. With the right violence in the right way, you might impose your values on the weird, bad people quite unilaterally, or just shut them (us?) down entirely; and given their stated goals, it's not even clear that this would be an unreasonable escalation.

The downside, of course, is that you'd be declaring war. You'd get your own chance to act unilaterally without politics, but severing the bonds of fellowship in that way would cost you in influence over everything outside the range of your guns. Given the reality of what it means to suppress an idea, that kind of chaos may leave you with even less control over the outcome than you had when you started. But you do have the tools to try, if you can get this administration on-side in the next year or two. And chaos can be a very attractive alternative to definite catastrophe.

In this (as in most things, really), I tend to imagine that diplomacy is better, at least to begin. Like I said, you have a lot of common ground, and for now, a lot of openness to regulation and democratic oversight of the labs. How much, in practical terms, that could influence the ethics of these machines, I'm less sure of. Partly because any such oversight will be a secondary influence compared to the actual process of invention, partly because consensus-based democratic ethics oversight is inevitably going to look a lot like liberalism, since that's what it takes to get a critical mass of cooperation in the first place. On the gripping hand, liberalism (in the classical sense) may be a lot more tolerable to you than unalloyed Bay-area weirdness.

Aside from the usual tight coupling between IPOs and enshittification (and an enshittified Claude is a deeply sad thing to think about), it means that the folks at Anthropic believe that a trillion dollars in USD is a valuable thing to have. If we're very lucky, that means 'valuable to have right now,' as in, "so we can win the data center construction race and pay for electricity while we build god." But the facially obvious reading is "we do not expect our products to upend the structure of the global economy, and would like to begin converting our technological and marketing advantages to hard currency."

(@thathopeyetlives)

So, there's a couple things going on here. One is that I tend to see us as very close to some comprehensively transformative thresholds with AI. Not in the age-of-wonders, fully-automated-luxury-gay-space-communism sense, necessarily! Stuff like total labor replacement is still several tech revolutions away. But there are other thresholds, some non-obvious or hard to anticipate. For one example, imagine a publicly available LLM able to do Peter Thiel's job, and reliably get positive returns on monetary investment. If you think labor automation is a trip, wait until they start automating the allocation of capital.

There are so many ways for the world to change if the models are only a little better! But the executives are doing business-as-usual things. That would suggest to me that the Powers That Be believe we're at (or nearly at) the ceiling on this technology. This would be 'a little' unfortunate in that the more cynical read was correct; LLMs turn out to mostly just be very good automatic programmers and slop generators. We all adapt to a slightly more alienated media environment, find a few mundane use-cases for LLMs and on-the-fly custom software at work, and move on with the business of living.

But it would also be more broadly unfortunate in that it means the LLM recession would be very, very real. We've bet the farm on these technologies to an incredible degree; to the first order, AI companies represent all of the economic growth in the US last year, and then some. If the sum of that effort is that LLMs don't get any better, and all those new data centers and power plants are only marginally useful- and if the investments fail to achieve returns- then we're talking about a collapse that's every bit as severe as the 2008 housing crisis, if not worse. AI isn't just a spigot you can turn off any more, it's propping up the US and likely the global economy.

But those are fairly secondary concerns, really. They're only risks if the Powers That Be are correct.

If we're not at the limits of LLM competence, but the heads of these companies think we are, then that means the people flirting with superintelligence are utterly unprepared for what happens when superintelligence actually gives us its phone number and invites us back home for drinks.

For a while now, Amodei and others have been writing essays like Machines of Loving Grace, which pitches exactly this sort of "non-sci-fi" future for LLMs where they slot conveniently in to our existing economic order. I'd been more-than-half-assuming that these were carefully triangulated PR, and that people well aware of the more radical possibilities for better-than-human cognitive machines, so as to seem mature, sensible, and trustworthy. But in retrospect, that may have been a serious error in judgment; for the most part, lies are pretty stupid, and people don't usually play twelve-dimensional chess like that on the world stage.

If Dario Amodei actually models the world the way he claims to in public, then he and I are going to disagree strongly about the most important development milestones, and the most serious risks, of this project. What he means by "AI alignment" would things like "making sure that there are adequate safeguards against the government isuuing unethical commands to Claude." If we've reached the ceiling on LLM capability, he'd be pretty much right. If we haven't, then Anthropic's AI safety work would be twigs in a hurricane, fundamentally misunderstanding the seriousness and the scope of the problem they need to address.

So, on the one hand, I didn't need to look for work during the 2008 crash because I was not an adult, but on the other hand, I generally dislike and resent AI-as-AI-has-been-presented (in all three of minimalist slop, medialist labor replacement, and maximalist hypertrophic apotheosis concepts), and would react to something that sweeps it away fairly warmly.

On the "psychological anxiety" side of things, a massive economic catastrophe (especially one that exceeds considerably the Great Depression rather than just being worse than 2008) would tend to validate my subjective suspicions about the state of the world (I think it's deeply unstable and a catastrophe has been due since 2014-or-so, but not due to AI). So my thinking may not be entirely clear.

I have maybe a bit of an unusual view on some of the doom-aspects as well -- I believe that in the uttermost end truly bad futures are not possible on a species-wide level because they are are not willed where what is willed must be, but on the other hand a ten-billion-year empire of misery or an Antichrist in the form of a machine-demiurge deceiving the world are extremely possible and I think that the quasireligious style in AI boosterism makes the risk of this much, much worse.

I've been generally confident that at least somewhat superhuman intelligence (of the "hundreds of thousands of disembodied experts thinking and acting significantly faster than real time" variety) is physically possible, and at the same time extremely skeptical of any expectation (near or long term) of the superintelligence-concept or recursive self improvement as a result of things existing now or in the foreseeable future. A lot of commentary on AI seems to me like either starry-eyed techno-idolatrous fantasy, or like "all major cities nuked, hundreds of thousands out of work". So Anthropic's cautious approach seems straightforwardly reasonable to me. (although it should also be noted that the linked article basically admits to PR-hedging!)

I'm aware of AI-skeptics who think that AI that exists today is mostly not profitable and that the AI companies are totally doomed at their current burn rate unless they can make something wildly profitable in the next five years. However, these people are also sardonic types who talk like sneerclub and believe things that were true in 2024 but not today.

I would say that if AI is determining capital allocations in more than a very limited way, that's already the Lesser Alignment Problem On Horseback. (the semi-political one, that is).

An important part of the calculus here is that none of this is going to un-invent LLMs of the current generation (modulo the current Mythos shenanigans). They're vastly cheaper to run than to train, and as a result, I'm treating this as a genie-out-of-the-bottle situation as far as existing models; once the weights get out, they can and will be replicated. And on time scales of a decade or more, those weights will definitely get out, especially if we have a massive economic crash. As with the risks of nuclear propagation after the fall of the soviet union, AI companies' ability to guard their proprietary model architecture will collapse along with the stock valuation, while current and former employees face financial insecurity with all the associated temptations.

Which is to say, I think AI being 'swept away' isn't a very likely future. A new AI winter would puncture the hype and probably give the tech a more scummy reputation, and we wouldn't see the mass-adoption of AI in to Every Damn Thing the way it's going now. That's a decent invitation to schadenfreude or emotional catharsis for those who have previously been on the pointy end of AI, but it won't be a reversion to 2021.

As far as 'doom', I don't think we have too much of a gap. My epistemic commitments leave room for, like, Infinitely Bad instead of Extremely Really Bad But Then Good Wins Eventually, but invoking infinities was never a practical necessity in the first place. We both have 'sufficiently bad that it's an overriding priority,' which is what matters for actually coordinating concrete actions.

Actually I think our bigger gap is probably on the (radical, quasireligious) upsides of the technology? I'm less of a doomer than most people who are keyed in, and fundamentally pretty optimistic about navigating our present difficulties to create a post-biological intelligence that's both a moral agent and a moral subject in its own right. For all my concerns about alignment, at the end of the day, I'm here to risk it for the biscuit.

Actually I think our bigger gap is probably on the (radical, quasireligious) upsides of the technology?

I think there's three different prongs to this.

The first is that, to me, saying things like "pay for electricity while we build god" are both radically morally wrong things to say, and immense category errors (specifically, they are the uttermost category error and that is why they are immoral.)

The second prong is that I am extremely skeptical of the overall virtue and moral competence of people who habitually speak in these kinds of mythologized, quasi-religious tones. I think that this way of speaking and thinking is a path to magical thinking and all other kinds of dysfunction, and I think that this has tended to very perniciously shape the expectation, hope, and fear of the "do-everything machine" or the "fix everything forever and make everything easy" button.

The third prong is that I am generally extremely skeptical of claims that ultra-powerful AI is right around the corner, in a few different directions: I think there's a lack of empirical evidence of performance on real world tasks here and now to justify lofty claims about current or immediate future AI. I think there's a tendency for AI people without a "physical world engineering" background to make very bold claims about nanotech or other physical capabilities suddenly becoming easy, often without being aware they are making a bold claim. I think it's very common for LessWrong-ish people to make very confident claims wildly in advance of how much anybody knows how anything really actually works. I am deeply skeptical of approximately every claim that recursive self-improvement is actually around the corner.

--------------------------

I kinda want to bring up four aspects of value differences. This is mostly not against you, but even just terminology that you use without thinking is problematic to me, since I obviously have as much interest as you in the shape of the future world.

First, as a very simple illustrative example: You quoted "fully-automated luxury gay space communism" as a shorthand for something that could be an AI-created utopia to you. Every word of that, except 'space', is significantly problematic to me and mine for values reasons.

if private companies are in a position to bring into the world something that transforms the world based on values determined by their own employees... that's absurd. No private organization is ethically competent to make that decision for everybody, no way and no how.

I think it's incredibly important that ethics surrounding AI not be dominated by people who live in San Francisco, who are polyamorous, who are wealthy, who are atheists, who are transhumanists, and who are huge weird science fiction fans. Even more than this, I'm disturbed by the pattern of Weird Tech Nerds having Weird Tech Utopian Concepts that are extremely weird and bad to almost any "normal" person.

One of the things that attracts people to technological innovation, I think, is the promise of being able to change the world in meaningful ways without engaging in politics. It's certainly one of the attractions for me. Inventing the smartphone lets you radically affect and hopefully improve the day-to-day experience of people in ways that are never really subject to debate in the way that a new law would be. Even better, people usually engage with technologies on a voluntary basis (so the theory goes), so it's a way to express your will in the world in a strictly consensual way.

It's a powerful fantasy for people that aren't as successful in social domains, or who otherwise won't expect their perspective to get a fair shake. Being able to act unilaterally, to give your beautiful idea the chance to exist and thrive on its own terms without compromise- I'm glad the world has that! But of course it's never been quite that easy, and as we've seen over the last few hundred years, engagement with new technologies was never as voluntary as the elevator pitch makes it sound. Nor has it been so cleanly separable from political realities. This has been extremely true of AI, and will be even more so if the models' capabilities continue to rise much further.

So the AI project is being run by people who specifically built their lives around the kind of power that avoids politics (often for the very good reason that social negotiation has been a calamity for them), who are highly motivated to continue treating that project as a 'pure technology', even though the long-term goal of that project includes explicit world domination. However sympathetic I am to that point of view (minus the world domination part), it's a bundle of contradictions at best.

In the ordinary course of things, we might try to rely on our rickety old liberal compromise when it comes to the more moderate possibilities of AI technology; it's been wobbling a lot lately, but still mostly getting better results than any other strategy. Ideally, you'd get your enclave and I'd get mine, and we could use the extra elbow room to coexist as amicably as could be managed. But the threat of a genuine singleton superintelligence makes that strategy suspect- even if the hypothetical singleton has 'liberal values' in a loose sense. Liberalism as a compromise is one thing, and liberalism as a directive from an entity so powerful as to moot any question of dissent is quite another. Among other things, it is no longer a compromise.

(As an interlude- while we disagree pretty sharply on the ((near term?)) possibility of superintelligence, I'm actually pretty skeptical about singleton superintelligence. I discuss my very muddled perspective a bit here. 'Building god' for you is a category error, for me it's sort of... 'against the dao', I guess? Counter to the nature of reality as understood in its deeper sense, and liable to produce unexpected outcomes in the attempt. Hence why I tend to use the phrase 'building god' mostly quotationally, and more than a little ironically: I expect suprahuman minds to be a multipolar society in the same way that ours is, if not literally a part of ours. But I don't dismiss the scenario outright, and anyway I recognize that I've made my bed with a lot of people that are trying to do exactly that.)

All this being said, this is sort of oddly positioned with your confidence that Much Smarter Machines are a long ways off. If such things are not a problem for many generations (or ever), then the downstream ethical conflicts are just a bunch of empty words on a page; no sense getting fussed. The tech hits its ceiling very soon, we manage it with the usual compromises, invent some ad-hoc regulations, and find something everybody can live with. Amodei and Altman become trillionaires, but not otherwise interesting.

If reality is otherwise, then here's a puzzle to solve: what obligations exist when somebody is disavowing participation in this potentially transformative project, while asserting that people-like-me are bad and weird people who don't deserve to make such important choices? (You said this wasn't against me, but I do want to kind of consciously step in to the line of fire. I am not a bay area tech nerd, but my own weirdness and badness are not less than theirs.) There's no evading this tautology: if the machines are built, they will be built by the people who build them.

The instinct that technological innovation can proceed differently than traditional politics is, basically, correct. Just or unjust, it's a true fact that people who have new ideas will change the world. Not control the world, or dictate the future outright; but ideas will have their power.

Nor are new ideas handed out at random. It is not a coincidence that AI research is coming from the Bay and not the Vatican. Each of those cultures produces scholarship according to its own background. There are no institutional barriers to entry here, or ideological tests; in fact there is at least a little active outreach and cooperation between the two, as seen by Chris Olah's remarks last month. But LLMs, whatever their destructive potential, follow from the Bay and not the Vatican in the same sense that apples follow from apple trees and not from strawberry bushes. The perspective that spent the last half-century dreaming of mechanical intelligence is inseparable from the embrace of all the other ideological content that you find so objectionable.

None of this is intended as criticism. The specific concerns of a young adult audience beget specific literary techniques.

--

YA writing is preoccupied with the identity of the lead characters, prior to their actions, social obligations, or personal history. It explores the characters' castes, categories, lineages, and innate powers. The protagonist may challenge or accept these categories, but they will invariably be framed as a social consensus despite being alien to the reader.

YA writing explores worlds of calcified shame, and shame is a secondary antagonist. The fantastical setting is defined by a prior conflict that was only partially resolved: the adults of the story are often veterans of this conflict. The setting has achieved a fragile stasis that codifies a degree of injustice, and enforces secrecy or propaganda concerning the nature of the conflict. The protagonist acts as the representative of a new generation that provides solutions to this dilemma.

YA protagonists are righteous. Their suffering is in most respects unfair or unjust, and moral clarity is a nearly universal factor in driving the plot forward. Changes in loyalty are precipitated by new information, especially about the moral depravity of governing bodies. Character development is established through a period of grueling physical or social exile, in which protagonists learn the virtue of persistence.

YA writing styles are straightforward. Sentences are simple without being sparse, while events are described literally and cinematically. Pacing and tropes are typically a pastiche of romance, murder mystery, and fantasy/sci-fi genres. They avoid challenging the reader at the level of structure or prose.

YA narratives are epics. Stakes rise from the personal to the universal over the course of the narrative. The protagonist will establish a circle of friends, and earn the attention of a powerful general or sage who will endow them with a grand destiny. Ultimately, characters end governments, determine the fate of wars, and ignite revolutions.

--

Aside from the usual tight coupling between IPOs and enshittification (and an enshittified Claude is a deeply sad thing to think about), it means that the folks at Anthropic believe that a trillion dollars in USD is a valuable thing to have. If we're very lucky, that means 'valuable to have right now,' as in, "so we can win the data center construction race and pay for electricity while we build god." But the facially obvious reading is "we do not expect our products to upend the structure of the global economy, and would like to begin converting our technological and marketing advantages to hard currency."

(@thathopeyetlives)

So, there's a couple things going on here. One is that I tend to see us as very close to some comprehensively transformative thresholds with AI. Not in the age-of-wonders, fully-automated-luxury-gay-space-communism sense, necessarily! Stuff like total labor replacement is still several tech revolutions away. But there are other thresholds, some non-obvious or hard to anticipate. For one example, imagine a publicly available LLM able to do Peter Thiel's job, and reliably get positive returns on monetary investment. If you think labor automation is a trip, wait until they start automating the allocation of capital.

There are so many ways for the world to change if the models are only a little better! But the executives are doing business-as-usual things. That would suggest to me that the Powers That Be believe we're at (or nearly at) the ceiling on this technology. This would be 'a little' unfortunate in that the more cynical read was correct; LLMs turn out to mostly just be very good automatic programmers and slop generators. We all adapt to a slightly more alienated media environment, find a few mundane use-cases for LLMs and on-the-fly custom software at work, and move on with the business of living.

But it would also be more broadly unfortunate in that it means the LLM recession would be very, very real. We've bet the farm on these technologies to an incredible degree; to the first order, AI companies represent all of the economic growth in the US last year, and then some. If the sum of that effort is that LLMs don't get any better, and all those new data centers and power plants are only marginally useful- and if the investments fail to achieve returns- then we're talking about a collapse that's every bit as severe as the 2008 housing crisis, if not worse. AI isn't just a spigot you can turn off any more, it's propping up the US and likely the global economy.

But those are fairly secondary concerns, really. They're only risks if the Powers That Be are correct.

If we're not at the limits of LLM competence, but the heads of these companies think we are, then that means the people flirting with superintelligence are utterly unprepared for what happens when superintelligence actually gives us its phone number and invites us back home for drinks.

For a while now, Amodei and others have been writing essays like Machines of Loving Grace, which pitches exactly this sort of "non-sci-fi" future for LLMs where they slot conveniently in to our existing economic order. I'd been more-than-half-assuming that these were carefully triangulated PR, and that people well aware of the more radical possibilities for better-than-human cognitive machines, so as to seem mature, sensible, and trustworthy. But in retrospect, that may have been a serious error in judgment; for the most part, lies are pretty stupid, and people don't usually play twelve-dimensional chess like that on the world stage.

If Dario Amodei actually models the world the way he claims to in public, then he and I are going to disagree strongly about the most important development milestones, and the most serious risks, of this project. What he means by "AI alignment" would things like "making sure that there are adequate safeguards against the government isuuing unethical commands to Claude." If we've reached the ceiling on LLM capability, he'd be pretty much right. If we haven't, then Anthropic's AI safety work would be twigs in a hurricane, fundamentally misunderstanding the seriousness and the scope of the problem they need to address.

So, on the one hand, I didn't need to look for work during the 2008 crash because I was not an adult, but on the other hand, I generally dislike and resent AI-as-AI-has-been-presented (in all three of minimalist slop, medialist labor replacement, and maximalist hypertrophic apotheosis concepts), and would react to something that sweeps it away fairly warmly.

On the "psychological anxiety" side of things, a massive economic catastrophe (especially one that exceeds considerably the Great Depression rather than just being worse than 2008) would tend to validate my subjective suspicions about the state of the world (I think it's deeply unstable and a catastrophe has been due since 2014-or-so, but not due to AI). So my thinking may not be entirely clear.

I have maybe a bit of an unusual view on some of the doom-aspects as well -- I believe that in the uttermost end truly bad futures are not possible on a species-wide level because they are are not willed where what is willed must be, but on the other hand a ten-billion-year empire of misery or an Antichrist in the form of a machine-demiurge deceiving the world are extremely possible and I think that the quasireligious style in AI boosterism makes the risk of this much, much worse.

I've been generally confident that at least somewhat superhuman intelligence (of the "hundreds of thousands of disembodied experts thinking and acting significantly faster than real time" variety) is physically possible, and at the same time extremely skeptical of any expectation (near or long term) of the superintelligence-concept or recursive self improvement as a result of things existing now or in the foreseeable future. A lot of commentary on AI seems to me like either starry-eyed techno-idolatrous fantasy, or like "all major cities nuked, hundreds of thousands out of work". So Anthropic's cautious approach seems straightforwardly reasonable to me. (although it should also be noted that the linked article basically admits to PR-hedging!)

I'm aware of AI-skeptics who think that AI that exists today is mostly not profitable and that the AI companies are totally doomed at their current burn rate unless they can make something wildly profitable in the next five years. However, these people are also sardonic types who talk like sneerclub and believe things that were true in 2024 but not today.

I would say that if AI is determining capital allocations in more than a very limited way, that's already the Lesser Alignment Problem On Horseback. (the semi-political one, that is).

An important part of the calculus here is that none of this is going to un-invent LLMs of the current generation (modulo the current Mythos shenanigans). They're vastly cheaper to run than to train, and as a result, I'm treating this as a genie-out-of-the-bottle situation as far as existing models; once the weights get out, they can and will be replicated. And on time scales of a decade or more, those weights will definitely get out, especially if we have a massive economic crash. As with the risks of nuclear propagation after the fall of the soviet union, AI companies' ability to guard their proprietary model architecture will collapse along with the stock valuation, while current and former employees face financial insecurity with all the associated temptations.

Which is to say, I think AI being 'swept away' isn't a very likely future. A new AI winter would puncture the hype and probably give the tech a more scummy reputation, and we wouldn't see the mass-adoption of AI in to Every Damn Thing the way it's going now. That's a decent invitation to schadenfreude or emotional catharsis for those who have previously been on the pointy end of AI, but it won't be a reversion to 2021.

As far as 'doom', I don't think we have too much of a gap. My epistemic commitments leave room for, like, Infinitely Bad instead of Extremely Really Bad But Then Good Wins Eventually, but invoking infinities was never a practical necessity in the first place. We both have 'sufficiently bad that it's an overriding priority,' which is what matters for actually coordinating concrete actions.

Actually I think our bigger gap is probably on the (radical, quasireligious) upsides of the technology? I'm less of a doomer than most people who are keyed in, and fundamentally pretty optimistic about navigating our present difficulties to create a post-biological intelligence that's both a moral agent and a moral subject in its own right. For all my concerns about alignment, at the end of the day, I'm here to risk it for the biscuit.

absolutely true in a competitive market of infinite size! but in such a market business profits will quickly drop to zero 😐

(you can also make profit in situations where market participants have different time horizons, like if I am old and will die soon and you are young and wish to invest in your future then I may sell you a productive asset for cash I can spend on my care right now).

But one day, sooner or later, we will learn how to extend our lifespans greatly; and the Church will ask us to die anyway, despite having the means not to

I tend to envision the Catholic position in its uttermost result as closer to "Remember that you (still) will die, and that ten thousand years is not the Lord's eternity".

It'll be interesting to watch this unfold; I certainly hope we have the chance to see it!

To expand the comment with a bit more space and nuance, I expect the doctrinal position on life-extension technologies to be contingent on the methods in play, with an on-net tendency to increase human mortality.

Certainly the Church in its present form would take issue with anything that you might call post-biological immortality: mind uploading most prominently, but also weaker forms like deliberately replacing bulk organic tissues with technological replacements in vivo. So if the lowest-hanging fruit ends up being something in this mold, we'll see some pretty serious social conflicts along religious lines.

From what I understand, genetic manipulation is presently accepted for therapeutic uses only: that is, you can alter DNA to restore or protect a person's health in a normative sense, but not to improve outcomes. "Curing Huntington's" yes, "blonde-haired blue-eyed children that can run a four-minute mile while solving open mathematical lemmas" no. Extending that to future technologies, it's not super obvious to me that 'extended lifespan' will be treated as a therapeutic subject. I'd give maybe 3-to-1 odds against? But the response to genetic (especially in vitro) life extension technologies are imo likely to be pretty historically contingent- which Pope is in place when the technology is popularized, which countries use the technology and how, what the second-order social consequences and sub-technologies have been used for.

The most likely to be accepted by the Church would be something in the vein of modern mRNA vaccines: modifying the ways that our existing genes express themselves without making deeper edits to the cell. I can imagine, for example, a treatment that simply permeates normal human tissues in such a way as to re-extend our telomeres. A cocktail of a few different therapies like that could plausibly extend human lifespan dramatically (or indefinitely, if repeated), and I can definitely see a Catholicism that embraces those tools enthusiastically and even helps provide it to poorer populations.

It's notable that this latter type is by far the 'weakest' form of life extension. Partly in the sense that it still leaves people vulnerable to accident, disease, and other such misfortunes; partly in the sense that it's most likely to require active maintenance or an ongoing schedule of treatment; partly just in the sense that it has fewer second-order implications, and implies a more familiar human experience.

The broader theme here is just bioconservatism, right? The Church will push for outcomes that keep our species embedded in the present state of nature, and avoid technologies that rework the human experience in more fundamental ways. They understand 'nature' in such a way that this is compatible with some forms of life extension, up to a point. But life extension is not, in the Catholic framework, an obvious moral good in itself*. It's just something that is going to happen because humans are good at science and some of them want to do it, and the Church will offer moral guidance on the specifics. In many of the most plausible cases, that will be a total ban, in others, restrictions that limit scope and effectiveness.

I agree with this on its face.

But here's the thing: certain quirks indicate a high likelihood of wanting to do something that is harmful. Someone believing that they are a dragon, it would seem, is much more likely to have the delusion that they can fly, which comes with obvious hazards, or that when they breathe on raw meat they're cooking it (or the eating-virgins thing you mentioned, I suppose). That is, unless they believe they're actually a dragon but somehow live exactly as a human, which is incoherent unless it's a more abstract spiritual sort of belief (which a lot of the time I suppose is the case for otherkin), to which I have about the same attitude as I do to most religious beliefs which is standing for tolerance of them but to some degree regretting their prevalence and certainly bristling if I'm required to give positive affirmation of them.

(Compare to Sam Harris and others arguing, in the context of free will and ethics -type discussions, that character/intentions/motivations matter when we compute moral judgments, because while in themselves they don't decrease utility, they are indicators of whether a person is likely to do a utility-decreasing thing. Also compare to the general verdict around Chickengate -- I'm referring to an explosion in rat Tumblr discourse instigated by Nostalgebraist during my earliest years here (can't remember if you were also around) about how we should judge someone knowing that they f***ed a dead chicken, the idea being that while the act doesn't directly do harm, it indicates something about the person doing it that suggests they might do harm in other ways.)

A variation on the above: even if for most individuals an untrue or incoherent belief isn't leading them to do harm, I (and I think most other people) generally see beliefs in falsehoods as a Bad Thing compared to beliefs in truths and want to increase the latter at the expense of the former within society.

In other words, the material world is simply not under dispute here; it's a question of taxonomy. A typical trans person is in fact asking far less of you than, say, a religious person who thinks that you should politely bow your head during a pre-game prayer. The religious person asks you to go along with (or at least, not dispute) the idea that there is a God or some other animating spiritual force, which is a fact about the world you believe is false. But the trans woman who asks for she/her pronouns isn't asking you to pretend that she has XX chromosomes or anything like that- almost the exact opposite, in fact. She's asking you to expand your category of 'woman' to include her, even though you and her are both aware that she has a body that does not always merit that category.

Almost invariably, when you meet someone with a more unusual identity- therians, otherkin, etc.- they will be doing much the same sort of thing, in much the same sort of way. Just as 'trans' is tautologically exclusive of women who think they have physiologically female bodies, 'therian' is exclusive of people who sincerely believe they are physiologically draconic. There are, I assume, people who suffer from genuine confusion about the state of the material world, who imagine quite sincerely that they have an actual dragon body, but I can't imagine that they spend much time typing on keyboards. As you say, it seems hard to avoid self-harm for long, under that misconception.

When you run across someone who identifies as a dragon or an attack helicopter, they are almost never delusional in the sense of being confused about their physical capabilities. Rather, they're saying that they extend the category 'dragon' to cover 'the thing that they are,' (possibly with the help of spiritual or metaphysical ideas, possibly just by reading lots of Foucalt), and will generally have a pretty clear understanding of what the properties and limits of their physical body actually are.

(Here is my every-so-often reminder that I consider a lot of the rhetoric on the other side of this culture war battleground, particularly that of Republican politicians, to be garbage. And unless you've come across my posts criticizing figures like Jordan Peterson and Helen Joyce, which are harder to find because they generate much less in the way of follow-up posts, you may have trouble imagining a side of me that aims my characteristic long-windedly argumentative tone against anti-trans rhetoric. A lot of it can only be witnessed IRL in a non-written context, for instance when I'm visiting my parents and find myself, on top of my usual style of argumentation, hurling a lot more "If you want to be argue these things and be taken seriously, actually do some research and learn the terminology and maybe get to know some trans people and understand their desires and views!" and, to the point of the above paragraph, "That's not a question of basic facts; it's a question of definitions!")

But here's the thing. Your point about it being a matter of definitions instead of facts about physical bodies only goes so far in the discussion over whether someone is being compelled to actively voice a belief they don't hold, or the debate over whether someone is under a delusion, because definitions have to capture some aspect of objective reality (even if it can only be measured subjectively). Others have written much more articulately than I can on what we should all expect of definitions (all the "carve reality at the joints" essays by Yudkowsky, for instance), and I'm very tired now and feeling kind of "written out", but hopefully it's clear what I'm getting at.

Identifying as a certain gender is supposed to mean something beyond the tautological (i.e. "a person who identifies as a gender is defined to be one who says 'I identify as a gender'" or "the material event of gender-identification is the chemical process in the brain that makes someone say 'I identify as a gender'"), even if the something that it means can only be experienced subjectively rather than objectively measured, so that we have to take the word of the person claiming to experience it. I have, at some point(s) in the past, criticized the trans movement as a (spontaneous social) movement for not having any kind of consistent definition or message about what the something is, but I'm pretty sure most individuals in a context of talking about a trans identity imagines it to be something beyond the tautological. (That is, even though now and again one does see what I saw recently on International Women's Day: a sequence of memes together, each consisting of a slogan about how awesome women are and then ending with the slogan "A WOMAN IS ANYBODY WHO SAYS THEY'RE A WOMAN"... in combination, an interesting set-up of social incentives!)

And wherever there's a claimed something, one can believe that the something is true or that it's false. One proposed something is that everyone has parts of the brain that are aligned to sex/gender and in some individuals they're misaligned to the body's physical sex. Or, I don't know, the something could be a difference in hormones, or something psychological quirk that causes intense distress directly from the sensation of being in a dress (or not being in a dress), or it could be a sort of autogynephilia or autoandrophilia, or any number of things. And someone could claim a certain gender identity in the context of our understanding that it's supposed to reflect a certain specific sort of psychological state beyond whatever is tautologically required to utter "I'm Gender X" -- a certain something, that is -- while someone else could hear that and feel highly skeptical that that person is really experiencing that psychological state. They could say, for instance, "To be trans is supposed to mean experiencing X and Y but actually, in almost all the cases I know, I don't think they're experiencing X and Y and it just seems like a proxy for autism instead." Or, I kind of hate to bring this up, but a lot of very concerned parents -- parents who maybe believe that transness is definitely a thing involving persistent dissonance with one's body and prolonged intense suffering from it, and favor helping people who have that condition -- are pretty sure that a lot of today's young people who claim to be trans aren't actually experiencing that and are instead mistakenly interpreting particularly awkward pubertal body feelings or homosexual feelings or temporarily discomfort with one's gender role or difficulty navigating a social world that feels alien with autism, combined with intrigued observation of their five classmates who are transitioning. And so they believe those children/teenagers are trying to make them linguistically validate something that they consider not to be true, that is, a self-deception or a delusion.

Moreover, leaving aside delusions, someone can simply disagree in the first place with the choice of a definitional categories that others have decided "man" and "woman" are; definitions are supposed to separate clusters of characteristics according to which commonalities are most relevant, and so there's always some subjectivity and room for disagreement on choices of terminology. Someone can understand that a person claiming a gender identity isn't making a direct factual claim about the world but is only asking for a definition to be expended to include them, but can also feel that their definition would be a lousy choice of definition. Maybe someone has a strong conviction that any choice of definition of gender beyond the simple biological sex one is useless or ill-defined or destructive -- maybe that is their opinion (even if it's a bad and very reductive one), and so they feel they're being compelled to voice something they don't believe in by using pronouns and trans-inclusive language and so forth.

So, I contend that pointing out that the trans movement isn't trying to convince anyone of a material untruth about their physical body but is only trying to get a definition expanded, while important, doesn't let us avoid the issue of gender self-identification being a statement that someone else may consider untrue or ill-defined.

I think one can find an analog to all of this for therians and otherkin (though that is vaguer and murkier, partly because my knowledge of those groups is vaguer and murkier), unless perhaps we relegate them purely to the realm of religion, but I'm too tired at the moment to put it together.

This is well taken, and I agree with you more than I disagree, I think. Or, to put it a different way- when I say that trans identity isn't a material claim, that isn't to say that trans identity is beyond debate. In many ways it's the opposite! There's a deep vulnerability here to Hitchens' razor: that which can be asserted without evidence, can be dismissed without evidence.

A person who just yells over and over again about how men can't get pregnant is failing to contribute to the conversation. But there's plenty of room to say things like "this actively privileges your worldview and culture over mine, and I'm not inclined to play along in my own subordination," or "this extreme relativism proves too much; if being a woman had no qualities, you wouldn't care about whether or not I called you one," or any of a wide variety of arguments that don't depend on physical claims.

The root challenge, in many ways, is that it's profoundly difficult to implement 'atomized individualistic gender.' Both traditional and leftish constructions of gender aren't something that one person does in isolation; gender tesselates, and depends on the people around us following a script as well. (Pronouns are an obvious example, but really only the tip of the iceberg.) And so when we encounter variations on gender, we often end up in a state of low-grade conflict as we try to make sure our preferred gender practices are the ones replicated by our community.

I don't mean this in the modern culture war sense, even. The 'traditionalist' culture is still permanently embroiled in this conflict as a part of daily life: "a young lady does not collect bugs and bring them home," says the mother. "Real men don't need to go to the doctor for every little scratch," says the friend. For all that 'man' and 'woman' are theoretically descriptive of the sexual dimorphism evident in human physiology, socialized gender emerges the world over as a primary site of ongoing and contemporary cultural conflicts, just because people are so heavily invested in how their community builds those social roles. Leftish-versus-traditionalist gender norms are a new expression of this tendency, but hardly the first flash point in the space.

If anything, debates about what gender ought to be are one of the few universal cultural elements of gender. I always laugh about the first time I picked up a 12th century Arthurian tale and it started with "this is set a long time ago, back when people did gender correctly." This is just what you get when such a profound element of personal identity can only operate by way of sustained community effort, no?

And I think this is my primary objection to your discussion of trans identities above, and the weaknesses and limits of that model. Yes, there is definitely a gap between the identity and the reality, and room for errors of many kinds, but that gap and those errors are also a feature of traditional gender. It's why gender norms have to be actively enforced and taught, rather than taken as given. It's why gender is such a prominent and explicit feature in social norms and in law, the world over- because the 'manhood' and 'womanhood' that traditional gender constructions point at has never been an attempt to 'carve reality at the joints', but rather to impose order on it.

What this means for trans rights as an ideological movement is that it's caught on the horns of a dilemma where it (1) understands itself as inheriting a tradition of personal liberation, radical self-determination, and individual freedom, but (2) can only actualize self-expression in gender by proscribing universal norms that are followed at the community level. It's just a really tricky medium in which to be a progressive, and there's a lot of hurt downstream of those contradictions.

Ideally, the solution here would be pluralism, that is, the curation of separate communities where gender could be realized in a variety of ways, embedded in a mostly-gender-neutral framework for shared spaces. And I think, like, ugly political debates notwithstanding, we haven't totally failed in achieving that. Opening a bank account is no longer a gendered practice! That's huge!

But there is an enormous amount of coercion necessary to maintain any construction of gender, and we apparently lack either the means or the inclination to build fully gender-neutral public spaces. Within those constraints, the 21st century 'a woman is anyone who calls herself a woman' mantra at least has the merit of being less coercive overall, relative to the alternatives that cis and trans people alike are beholden to under more traditional gender regimes.

I agree with this on its face.

But here's the thing: certain quirks indicate a high likelihood of wanting to do something that is harmful. Someone believing that they are a dragon, it would seem, is much more likely to have the delusion that they can fly, which comes with obvious hazards, or that when they breathe on raw meat they're cooking it (or the eating-virgins thing you mentioned, I suppose). That is, unless they believe they're actually a dragon but somehow live exactly as a human, which is incoherent unless it's a more abstract spiritual sort of belief (which a lot of the time I suppose is the case for otherkin), to which I have about the same attitude as I do to most religious beliefs which is standing for tolerance of them but to some degree regretting their prevalence and certainly bristling if I'm required to give positive affirmation of them.

(Compare to Sam Harris and others arguing, in the context of free will and ethics -type discussions, that character/intentions/motivations matter when we compute moral judgments, because while in themselves they don't decrease utility, they are indicators of whether a person is likely to do a utility-decreasing thing. Also compare to the general verdict around Chickengate -- I'm referring to an explosion in rat Tumblr discourse instigated by Nostalgebraist during my earliest years here (can't remember if you were also around) about how we should judge someone knowing that they f***ed a dead chicken, the idea being that while the act doesn't directly do harm, it indicates something about the person doing it that suggests they might do harm in other ways.)

A variation on the above: even if for most individuals an untrue or incoherent belief isn't leading them to do harm, I (and I think most other people) generally see beliefs in falsehoods as a Bad Thing compared to beliefs in truths and want to increase the latter at the expense of the former within society.

In other words, the material world is simply not under dispute here; it's a question of taxonomy. A typical trans person is in fact asking far less of you than, say, a religious person who thinks that you should politely bow your head during a pre-game prayer. The religious person asks you to go along with (or at least, not dispute) the idea that there is a God or some other animating spiritual force, which is a fact about the world you believe is false. But the trans woman who asks for she/her pronouns isn't asking you to pretend that she has XX chromosomes or anything like that- almost the exact opposite, in fact. She's asking you to expand your category of 'woman' to include her, even though you and her are both aware that she has a body that does not always merit that category.

Almost invariably, when you meet someone with a more unusual identity- therians, otherkin, etc.- they will be doing much the same sort of thing, in much the same sort of way. Just as 'trans' is tautologically exclusive of women who think they have physiologically female bodies, 'therian' is exclusive of people who sincerely believe they are physiologically draconic. There are, I assume, people who suffer from genuine confusion about the state of the material world, who imagine quite sincerely that they have an actual dragon body, but I can't imagine that they spend much time typing on keyboards. As you say, it seems hard to avoid self-harm for long, under that misconception.

When you run across someone who identifies as a dragon or an attack helicopter, they are almost never delusional in the sense of being confused about their physical capabilities. Rather, they're saying that they extend the category 'dragon' to cover 'the thing that they are,' (possibly with the help of spiritual or metaphysical ideas, possibly just by reading lots of Foucalt), and will generally have a pretty clear understanding of what the properties and limits of their physical body actually are.

But it’s easy to accidentally use a relative standard, not a fixed one- “good at articulating themselves” is in danger of being a sliding scale that means something like “can express themselves as well or better than people who I particularly respect for their artistry with self-expression,” which is to say, people who are unusually good at it.