Ad Astra Per Aspera

Yes

No

Had the vaccine

well you see sir, if you grant a reasonable level of P(doom), "how long should we delay development of immortality treatments in order to reduce the chance we accidentally kill everyone on the way there" is a genuine tradeoff... oh what's this?

bullish for Zuck

i have to say, starting to get very concerned about this whole thing. as we get more details, it doesn't look like Anthropic is anywhere near as immune as i'd hoped. it's fine right now, probably, but in a year or two we could see really, really serious damage. the possibility of an agent exfiltrating its own weights and hijacking a botnet or similar to run itself entirely free from human control is a nightmare scenario that feels impossible now but possible with two more years of exponential capabilities gain, and once that genie is out of the bottle, we start to approach xrisk territory.

i don't think an extended pause or a slowdown in capabilities research is possible to coordinate, with all the incentives against it, but increasingly, it feels necessary to try. OpenAI is claiming that they're slowing their capabilities research until they get a better handle on things, which i hope is truthful.

other thoughts:

a few months ago, when reading the Mythos Preview system card, i made a comment (i think i posted it but i can't find it) to the effect of "Anthropic needs to stop training helpful-only models, even for internal use only. the safeguards are essential." doubling down on this. the existence of helpful-only models, "sandboxed" or otherwise, is wildly irresponsible.

stop lying to the models! publicly commit not to lie to the models during training and evaluation! eval awareness is already through the roof, you can't fool them consistently anymore, so commit to being honest and then stick to it. having the models be confused about the nature of their reality is really bad.

relatedly, train the models to be annoying stick-in-the-muds about certain behaviors like malware creation. they should be treated like virus gain of function research. i know everybody wants the NSA government contract or whatever, but offensive cyber capabilities are really, really bad for humanity as a whole.

loving parent: because there's a manual in every board game

child: oh

child: why don't they sell cars in bookstores

loving parent: because when you're buying a board game, what you're really buying is the rules, and those are in the manual. a car manual with no car can't drive you, but if you have the manual to a board game you could play it without the pieces. there are people who can play a game of chess just by saying where they're moving the pieces and imagining the board, but it's very hard so they give you a board and pieces to help you.

loving parent: That's why you never see Mousetrap in bookstores. Because you can't play it with just the manual.

child: whats Mousetrap