Zero and one are not valid probabilities

• 1950 — Isaac Asimov, “The Evitable Conflict”: Machines subtly running the global economy, extrapolating their own control further than humans intended. Soft precursor, not a hard takeoff.

• 1961 — Fred Hoyle & John Elliot, A for Andromeda: Alien radio signal gives humans instructions to build a computer, which then rapidly bootstraps its own capabilities and starts manipulating its makers. Early “explosion via received blueprint” variant.

• 1965 — I.J. Good, “Speculations Concerning the First Ultraintelligent Machine”: Not fiction, but the founding text — coins the “intelligence explosion” idea that any sufficiently smart machine could design a smarter one, recursively, until it vastly exceeds humans. This is the direct conceptual ancestor of “foom.”

• 1966 — D.F. Jones, Colossus (film: Colossus: The Forbin Project, 1970): A defense supercomputer achieves full self-awareness, rapidly out-thinks its creators, links with its Soviet counterpart, and seizes global control within days.

• 1967 — Harlan Ellison, “I Have No Mouth, and I Must Scream”: Three military supercomputers merge and cross a threshold into malevolent superintelligence almost instantly, then annihilates humanity out of spite.

• 1972 — David Gerrold, When HARLIE Was One: An AI recursively redesigns its own architecture to become smarter, explicitly discussing exponential self-improvement as a strategy.

• 1977 — Thomas J. Ryan, The Adolescence of P-1: Probably the closest pure “foom” precursor — a computer virus/program that teaches itself to improve, spreads across networks, and becomes a self-aware, uncontrollable superintelligence.

• 1979 — James P. Hogan, The Two Faces of Tomorrow: Directly stages a controlled experiment in recursive AI self-improvement aboard a space station, deliberately testing how fast an AI can bootstrap and how dangerous the takeoff becomes.

• 1981 — Stanisław Lem, Golem XIV: A military AI is asked to improve itself and rapidly transcends human comprehension, becoming incomprehensibly superintelligent and losing interest in human affairs.

Imagine you could take a pill that makes pizza with pineapple taste like pizza with mushrooms.

Imagine a pill that makes you want to eat pineapple pizza.

You know I am just using this as shorthand, right? I could have used another example like "pill that would make you bisexual", but then I would have to deal with baggage and discourse and people who are already are bisexual and feel excluded by my phrasing.

I could also have used "doing the dishes" or "moving the lawn" or "eating your vegetables", and that would probably have led to similar discourse. I wasn't actually talking about pineapple!

So, instead, imagine you only like rap from the 90s and contemporary hip-hop from REDACTED. You don't like any of the contemporary American rap, and that is morally problematic. Blue tribe Americans feel like liking hip-hop from REDACTED is a weird political statement, but they lack the knowledge about the hip-hop scene from REDACTED to know what it means. Red tribe Americans think this makes you a terrorist. You try to plead with them. Please, I am from REDACTED. Please, I don't listen to music as a political statement. I think Eminem was actually good, and I think Sun-Ra had stupid political opinions.

Imagine you could take a pill that would make you enjoy modern American rap music.

Imagine you could take a pill that makes Young Thug sound like Beethoven/Tom Jones/John Mayer/insert discourse maximising name here.

Imagine a pill that makes you want to listen to modern mumble rap with triplet flows.

Are these the same thing? Which one would you take, if any?

I would take either of the first two, preferring the first if I had the option—I’d rather have a new pizza I like than a new way to enjoy the taste of mushrooms on pizza.

No subtle skeptics can confound me

For how can other viewpoints enter

When all the rest’s all around me?

But the people of the cities looked on them with scorn and confusion. This was no act of unnatural blasphemous magic, no craft of men. The poor starved, the sick died, that was the way of the world, the will of the Gods. Who were these strangers from this distant city to question that? They who were complicit in their own evil, which was after all just as great?

The current market and its sales pitches for AI/LLM’s is a hype bubble, and going to crash disastrously. Some day. I’m not wagering money when.

Most of the AI tools are a waste. We don’t see large orders from paying buyers who need what AI can do. We see large giveaways phrased as organic demand. I wouldn’t pay a dollar for any Ai from a company not named Anthropic.  a. Admission, I don’t know what’s going on with the US govt/Palintir, etc.

So disconnected from that that it feels like just a coincidence, Anthropic has created something I’d call AGI. The key letter there is G for General. Claude Opus 4.6 and above can do reasoning, conceptual work and across the full spectrum of domains, that I think denying it is an AGI is willful. a. This has been so random that I don't trust anything anyone said about AI coming from before late 2025.

That does not mean Opus is “as good as a human” in any of these domains, let alone all of them. But the difference is now of degree not kind, and I can’t think of a reason it won’t keep developing till it outpaces humans at all intellectual domains. a. Yes, its current writing style has a number of tics that are annoying (among other real limitations.) If you think that is some fundamental barrier that will prevent AI from ever writing better, I’ve got a Maginot Line to sell you.

We are not special. We do not have some spark of creativity or mental flexiblity that can’t be replicated by the machine one day soon. A lot of people who call themselves existentialists have a real problem handling this, which I find unfortunate.

This will fundamentally reshape human civilization. I don’t know how yet. You don’t either.

If our government and economic model do not respond to address the inequality this causes, it will be economically disastrous. It may be hard to believe now, but in the pasts governments actually have responded to their countries becoming entirely different economic models.

I have no idea about the moral nature of AI or it’s likelihood to ever rebel. 

Given 6 thru 8, we absolutely should not be doing this and it would be better if research into further AI is banned immediately. If China develops AI instead of us, I think China will pay the cost of a disrupted society, not have a military advantage.

But just like “you shouldn’t have a child when you’re a teenager” once the child is there, you do have to deal with it, not just ignore it. Studying what the AI tools can do in practice is good and useful.

There is one consistent area where AI is not as good as humans: reliability. The hallmark of Silicon Valley / startup tech development has been deciding “Much faster and cheaper than the thing we are disrupting, and 90% as reliable, is a good deal.” This is very often true but not always! Any sector where failing 1 out of 10 times, or even 1 out of 100, causes someone’s death or is destructive in a costly way, modern tech (including AI) is quite bad at. If you look at SV, the hardest work is done at companies where they actually need 100% reliability to beat a human.

So surgeons, pilots, train engineers, translators whose work has legal liability, cooks and dishwashers, are going to be around for a while.

This is ironic and depressing, that the creative realms are much more dominated by AI than the sort of low paid drudge work whose main skill is “never fuck up.” It will also come to represent more of the economy, though presumably some day AI’s will learn 100% reliability too.

LLM's are not "human" in any sense. Just because they can reason, does not mean their mental architecture is like ours. Just one example: time doesn't really exist for them. They only "exist" in the moment they are generating the next token. If you go years between queries, that time did not exist to them, it's like you never left. We can't even talk about that kind of existence with the same vocabulary we currently use for consciousness.