Metastable1

Since the public release of GPT-3 in mid-2020, AI has entered an era of foundation models, scaling and general-purpose algorithms. As AI systems become increasingly capable, they create new risks to public safety that need to be monitored. These include accident risk, as well as risks borne of new malicious applications that were previously impossible.

Anthropic is an American company behind the artificial intelligence (AI) Claude—which belongs to the very top tier of models, alongside the more famous GPT from OpenAI. Its investors include Amazon, Google, Microsoft, and NVIDIA. It is the third most valuable private company in the world—currently worth over $350 billion. In January, Dario Amodei—Anthropic’s founder and CEO—published an essay titled: "The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI."

In it, he shares concerns regarding the development of AI whose capabilities surpass those of leading experts in many fields. And this is not about the distant future: there is "a very strong chance it comes in the next few [years]."

Why does he think so? Models learn according to predictable laws. They are already automating parts of the work within the companies that create them. The finale will be a loop in which the model works on the next generation itself—without human involvement. As one Anthropic engineer put it:

"We want Claude n to build Claude n+1, so we can go home and knit sweaters."

Companies possess enough computing power to put hundreds of thousands of such AIs to work simultaneously.

Amodei lists four main threats: AI autonomy, misuse by small groups, misuse by states (surveillance, propaganda, weaponry), and economic shock.

Counterarguments to Penrose's position on AI. Also, I recommend this short excerpt from interview with Scott Aaronson in regards to his claims about consciousness.

Aaronson's "The Gödel CAPTCHA"

Notes from Aaronson's lecture about the argument.

Turing countering this sort of argument already in his 1950 paper: