The BFG of AGI

Twopp is left in a state of high-stakes Big Robot crimewars, in which the most efficient criminals imaginable battle the most ruthless robocops science can deliver, with the Twoppians caught in the crossfire, collateral damage in a robotic forever war (on crime).

As Farrell writes, this is already afflicting the US health system, where an army of insurance company robo-claim-deniers have been countered with a doctors' army of robot-claim-appealers:

https://www.nytimes.com/2024/07/10/health/doctors-insurers-artificial-intelligence.html

The point being that people need health care, people need public services, and while there will always be some waste at the margins (whether due to incompetence or dishonesty) responding to this by beefing up the system's defenses with more advanced red tape just requires the people who legitimately need these services to employ more aggressive tactics.

In support of this, Farrell points to a great, long essay by Dan "Accountability Sink" Davies for the Niskanen Center, "'The Problem Factory' – Preemptive risk aversion in infrastructure planning and the role of professional services":

https://www.niskanencenter.org/the-problem-factory-preemptive-risk-aversion-in-infrastructure-planning-and-the-role-of-professional-services/

Davies' essay describes how increasing bureaucratic defenses against frivolous or dishonest claims drives the participants in these processes to assume a war footing and approach the system as a battlefield, leading to the very runaway cost inflation that the bureaucratic process was instituted to prevent.

(Davies, a cybernetician, has some fascinating advice about how to structure planning processes to minimize this, but that's out of scope for this particular post.)

This reminds me of nothing so much as the spam wars. There was a time when it was very easy to set up a mail server and provide email access for anyone who wanted it – including spammers. Increased spam begat increased anti-spam countermeasures, notably the creation of blocklists that allowed mail administrators to automatically reject email from "insecure" mail servers.

Inevitably, spammers figured out how to send spam from "secure" servers, resulting in stricter, more onerous standards for mail server configuration. Spammers – for whom the ability to send spam is an existential matter – figured out how to meet these standards, so the security demands jumped again – and again, and again.

Today, sending and receiving mail is so technically challenging that most of the internet's email is run by a handful of giant, mostly US-based corporations. If any of these companies decides your mail server is spamming, you effectively disappear from the internet and good luck getting them to acknowledge an error. Meanwhile, these companies emit an avalanche of spam, but no one will ever block their servers, because to do so would be to cut off billions of legitimate email users:

https://pluralistic.net/2021/10/10/dead-letters/

And since most of these companies are US-based, they are liable to being weaponized by Trump, who has taken to ordering his tech giants to block foreign officials whose policy decisions make him angry:

https://carnegieendowment.org/emissary/2026/07/icc-trump-push-dismantle

Another parallel is the content moderation wars that saw the large platforms coming up with progressively more detailed rules about what constituted harassment and hate speech, only to have dedicated trolls master these rule-books. Trolls – for whom harassment was a full-time vocation – became the world's greatest experts on the platforms' speech policies, which let them skate right up to the line when abusing their victims, and to get those victims kicked off the platforms if they could be lured into putting a single toe over the line in response:

https://pluralistic.net/2022/08/07/como-is-infosec/

Farrell criticizes the Economist's answer to the (allege) looming robo-lawyer threat as "solutionism," Evgeny Morozov's word for "Recasting all complex social situations either as neat problems with definite, computable solutions":

https://en.wikipedia.org/wiki/Technological_fix

Using AI to root AI-generated bureaucratic appeals sacrifices the system's putative purpose – delivering services – in the name of defending that service from abuse and misuse of the system's resources. As the pioneering cybernetician Stafford Beer famously wrote, "the purpose of a system is what it does." If your bureaucracy is more concerned with fighting fraud than delivering service, then it isn't a service delivery system at all – it's a service denial system.

As Farrell writes, the people of Twopp can tell you how this ends – in a war of giant robots in which we are all collateral damage.

(A brief postscript: Farrell is a font of science fictional analogies to modern policy issues. This weekend in the FT, he and Dan Wang published an excellent editorial on the relevance of the paranoid, claustrophobic fiction of Philip K Dick to our present political reality:)