The bureaucratic AI arms-race is mutually assured destruction
If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:
According to an Economist editorial, "AI is breaking the British state" by making it too easy to file complaints, demands and appeals, which will "drown the state" with "demands as well-crafted as a first-class lawyer's":
https://archive.is/VTrj9
Let's pause a moment to appreciate the Economist's touching credulity about AI's coming legal mastery. The law seems to be the area where AI is most prone to "hallucinate" (that is, "produce defective outputs"), which can only be sorted through by skilled practitioners whose experience gives them the discernment to distinguish useful arguments from foolish ones:
(And this requires those skilled practitioners to avoid the "automation blindness" that afflicts people who are asked to remain vigilant for things that seldom occur, a phenomenon that has turned every TSA agent into the water-bottle-detectingest motherfucker the human race has ever produced, who still misses 95% of the guns that red teams bring through the checkpoint):
More notable than the Economist's faith-based predictions about the impending army of hyper-competent robo-lawyers is the magazine's proposed solution to this looming crisis: "stop creating entitlements that are ripe for AI-fuelled claims…prune the mass of procedural rights." Above all, replace the bureaucrats who process your "complaints, demands and appeals" with more AI, which will arbitrarily decide who gets what, through "personalised welfare interventions" that are not based on any kind of guaranteed rights.
Writing on his blog, the political scientist Henry Farrell tells us where this will inevitably end up: with AI-based robot wars in which increasingly stingy and pernickety robo-bureaucrats create demand for progressively more aggressive robo-lawyers:
As Farrell writes, this end-time was foretold by the prophet Alan Moore with his 1980s 2000 AD character Abelard Snazz, "the man with the two-storey brain":
https://en.wikipedia.org/wiki/Abelard_Snazz
Snazz "solves" the street crime epidemic on the planet Twopp with "Big Police Robots," who spiral out of control, arresting the citizens of Twopp for trivial crimes like wearing brown shoes with a blue suit ("breaking the laws of good taste"). To solve this new problem, Snazz invents "Big Criminal Robots" whose "cunning, efficient" crimes "take up all the police's time."
Twopp is left in a state of high-stakes Big Robot crimewars, in which the most efficient criminals imaginable battle the most ruthless robocops science can deliver, with the Twoppians caught in the crossfire, collateral damage in a robotic forever war (on crime).
As Farrell writes, this is already afflicting the US health system, where an army of insurance company robo-claim-deniers have been countered with a doctors' army of robot-claim-appealers:
The point being that people need health care, people need public services, and while there will always be some waste at the margins (whether due to incompetence or dishonesty) responding to this by beefing up the system's defenses with more advanced red tape just requires the people who legitimately need these services to employ more aggressive tactics.
In support of this, Farrell points to a great, long essay by Dan "Accountability Sink" Davies for the Niskanen Center, "'The Problem Factory' – Preemptive risk aversion in infrastructure planning and the role of professional services":
Davies' essay describes how increasing bureaucratic defenses against frivolous or dishonest claims drives the participants in these processes to assume a war footing and approach the system as a battlefield, leading to the very runaway cost inflation that the bureaucratic process was instituted to prevent.
(Davies, a cybernetician, has some fascinating advice about how to structure planning processes to minimize this, but that's out of scope for this particular post.)
This reminds me of nothing so much as the spam wars. There was a time when it was very easy to set up a mail server and provide email access for anyone who wanted it – including spammers. Increased spam begat increased anti-spam countermeasures, notably the creation of blocklists that allowed mail administrators to automatically reject email from "insecure" mail servers.
Inevitably, spammers figured out how to send spam from "secure" servers, resulting in stricter, more onerous standards for mail server configuration. Spammers – for whom the ability to send spam is an existential matter – figured out how to meet these standards, so the security demands jumped again – and again, and again.
Today, sending and receiving mail is so technically challenging that most of the internet's email is run by a handful of giant, mostly US-based corporations. If any of these companies decides your mail server is spamming, you effectively disappear from the internet and good luck getting them to acknowledge an error. Meanwhile, these companies emit an avalanche of spam, but no one will ever block their servers, because to do so would be to cut off billions of legitimate email users:
https://pluralistic.net/2021/10/10/dead-letters/
And since most of these companies are US-based, they are liable to being weaponized by Trump, who has taken to ordering his tech giants to block foreign officials whose policy decisions make him angry:
Another parallel is the content moderation wars that saw the large platforms coming up with progressively more detailed rules about what constituted harassment and hate speech, only to have dedicated trolls master these rule-books. Trolls – for whom harassment was a full-time vocation – became the world's greatest experts on the platforms' speech policies, which let them skate right up to the line when abusing their victims, and to get those victims kicked off the platforms if they could be lured into putting a single toe over the line in response:
Farrell criticizes the Economist's answer to the (allege) looming robo-lawyer threat as "solutionism," Evgeny Morozov's word for "Recasting all complex social situations either as neat problems with definite, computable solutions":
https://en.wikipedia.org/wiki/Technological_fix
Using AI to root AI-generated bureaucratic appeals sacrifices the system's putative purpose – delivering services – in the name of defending that service from abuse and misuse of the system's resources. As the pioneering cybernetician Stafford Beer famously wrote, "the purpose of a system is what it does." If your bureaucracy is more concerned with fighting fraud than delivering service, then it isn't a service delivery system at all – it's a service denial system.
As Farrell writes, the people of Twopp can tell you how this ends – in a war of giant robots in which we are all collateral damage.
(A brief postscript: Farrell is a font of science fictional analogies to modern policy issues. This weekend in the FT, he and Dan Wang published an excellent editorial on the relevance of the paranoid, claustrophobic fiction of Philip K Dick to our present political reality:)
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality✓ Free Actions
Free to watch • No registration required • HD streaming
We shouldn’t assume that the measures we take to combat the coronavirus today are temporary.
In the midst of a crisis, Zoom (and similar videoconferencing programs) provided an immediate, seamless way for people to continue to work and socialize while maintaining a safe physical distance from one another. Here was a simple technological response to the many complicated social problems that arose during the pandemic, a solution that seemed to address a practical challenge while also proving the legitimacy of Zoom’s slogan—“We deliver happiness.”
But as the weeks of lockdown wore on, and virtual gatherings shifted from novelty to obligation, many Americans began to confess to feelings of dread each time a new Zoom meeting appeared on their calendars. Human nature, that irrepressible beast, emerged in stories of “Zoombombers” who used the platform to interrupt classroom lectures and business meetings, harassing others with hateful remarks. Then there were the lackadaisical workers who neglected to turn off their cameras, treating their colleagues to embarrassing displays of private behavior made inadvertently public.
By the end of April, New York Times reporter Kate Murphy was explaining to readers “why Zoom is terrible.” The disappointments she outlined were not technical—the platform had resolved its privacy and software glitches—but experiential. Murphy noted the unease she felt about her connections to others, even after hours spent talking to people through a screen, because she could not always interpret the subtleties of facial expressions and body language. “These disruptions, some below our conscious awareness, confound perception and scramble subtle social cues. Our brains strain to fill in the gaps and make sense of the disorder, which makes us feel vaguely disturbed, uneasy and tired without quite knowing why,” she wrote.
…
It was the very swiftness and uncritical enthusiasm with which Americans embraced an “easy” technological solution to a complicated problem that suggests that we are becoming increasingly comfortable with technosolutionism, and not just during times of crisis. Such acquiescence seems understandable at such times, when uncertainty prevails, but as we continue to struggle to find our bearings, it is worth considering the significant choices we have already made with regard to technological problem-solving, and begin to contend with the consequences.
Technosolutionism is a way of understanding the world that assigns priority to engineered solutions to human problems. Its first principle is the notion that an app, a machine, a software program, or an algorithm offers the best solution to any complicated problem. Notably, the technosolutionist’s appeal to technical authority, even for the creation of public policy or public health measures, is often presented as apolitical, even if its consequences are often not. Technosolutionism speaks in the language of the future but acts in the short-term present. In the rush to embrace immediate technological fixes, its advocates often ignore likely long-term effects and unintended consequences.
Technosolutionism is also often unabashedly radical in its vision of what it might accomplish, particularly in times of crisis. One of the more enthusiastic purveyors of such reasoning, Aaron Bastani, argued recently that “the pandemic makes it clear: We need Fully Automated Luxury Communism,” his shorthand for a technosolutionist system he describes as “this technological revolution—oriented around automation, renewable energy, AI, and ever more objects resembling ‘information goods.’”
Even if few people are buying into the appeal of Fully Automated Luxury Communism, we are all witnessing and, under the pressure of the pandemic, more or less resigning ourselves to a growing dependence on technosolutionism in two areas that shape our everyday lives: public health and education.
…
In a time of intense political polarization, technosolutionist approaches can appear to bear a veneer of nonpartisan authority. But the same sort of surveillance used to track the spread of a virus can just as easily track one’s movements during a political protest, for example. That is both the appeal and the danger of technosolutionism, depending on the amount of power one holds. Whether the crisis is one of public health or public safety—riot control or virus control, for example—the response is the same: increased surveillance, especially by the state. As privacy activist Wolfie Christl noted, “The location-data industry was ‘covidwashing’ what are generally privacy-invading products.”
Powerful technosolutionist fixes during a pandemic can look like South Korea’s contact tracing system, which has been praised by public health experts for its early adoption and its effectiveness in slowing the spread of the pandemic. Yet few mention that South Korea’s success relied on a smaller, more homogenous population than America’s—and for that matter, one that is far more trusting of its government and institutions than we currently are.
Technosolutionism can also look like the much more intrusive Chinese contact tracing system, which requires citizens who want to move about in public to download an app that issues a color-coded QR (quick response) code that, when scanned, reveals their COVID status. Only those with a green code are allowed to move freely in public, though their whereabouts are continually tracked. Also, their QR codes must be scanned before they are allowed entrance to office buildings, grocery stores, and other public accommodations. According to the global business news website Quartz, one Chinese province has already announced plans to “normalize” use of the health codes after the pandemic ends. Health officials said they could use the codes “to assign a health status based on people’s digital medical records, including the results of health check-ups and lifestyle habits, such as how many cigarettes they smoke, steps they walk, or hours they sleep daily.”
…
If the relentless rise of technosolutionism in the world of public health is not alarming enough, its resurgence in education—where at least a modicum of skepticism had begun to assert itself—should make us leery of allowing crisis conditions to normalize dubious nostrums. As the pandemic spread, and schools across the country were closed, Americans found themselves caught up in a large-scale experiment in distance learning for nearly fifty million K–12 students. Some school districts moved swiftly to online learning models, replacing classroom instruction with Zoom lessons; others struggled to meet the needs of student populations that often lacked access to the technology that would allow them to learn from home.
The early results were not heartening: “The grade from students, teachers, parents and administrators is already in,” the Wall Street Journal reported. “It was a failure.” Preliminary study had indicated that “students nationwide will return to school in the fall with roughly 70 percent of learning gains in reading relative to a typical school year, and less than 50 percent in math, according to projections by NWEA, an Oregon-based nonprofit that provides research to help educators tailor instruction.” Notably, the NWEA researchers projected “a greater learning loss for minority and low-income children who have less access to technology, and for families more affected by the economic downturn.”
…
Such reliance on technology experts rather than teachers or parents is typical of technosolutionist responses to complex problems. “Bill Gates is a visionary in many ways and his ideas and thoughts on technology and education, he’s spoken about for years, but I think we now have a moment in history where we can actually incorporate and advance those ideas,” Cuomo said, even though, as the Washington Post noted, “the Bill & Melinda Gates Foundation has spent billions of dollars on education reform projects it has conceded did not work as hoped.”
Yet despite their paltry results, technosolutions continue to find support among political leaders for sadly obvious reasons: They can claim to have solved a problem in the near term (Zoom classes for all!), while punting on planning and paying for the problems that will emerge later (such as gaps in learning). Similarly, technology billionaires’ foundations and technology companies selling these solutions get power over policymaking without responsibility for any of the long-term consequences.
…
As the pandemic experience has revealed, we have come to rely on our devices (and implicitly, to trust them and the companies that make them and track our use of them) far more than many of us might previously have realized. We have also become more trusting of immediate technosolutionist thinking to solve complex, evolving problems.
That trust is often misplaced. The early evidence from our experiments with automated contact tracing and online learning offer an object lesson—and perhaps a cautionary tale—about embracing technosolutionism. Critics of technosolutionism do not argue for a world without technology; on the contrary, they recognize that technical solutions to human problems have often alleviated suffering and encouraged human flourishing. But when such solutions are offered as wholesale replacements for human problem-solving, and dispatch with deliberative, democratic processes meant to ensure that they are implemented in a way that respects a nation’s values and protects citizens’ privacy, their efficacy becomes questionable.
Consider what might happen if surveillance techniques unacceptable to most people in normal times but embraced during the pandemic do not disappear when it ends. As Yuval Noah Harari argued in The Financial Times, “Temporary measures have a nasty habit of outlasting emergencies, especially as there is always a new emergency lurking on the horizon.”
We shouldn’t assume that the measures we take to combat the coronavirus today are temporary. History suggests that such measures rarely are. The Patriot Act, passed as a temporary emergency response in the aftermath of 9/11, has been continuously renewed ever since. It is now almost twenty years old.
Yet there is reason for optimism about balancing the technological and the human. With our irrational impulses and self-delusions, we humans are the weak link in technosolutionist dreams of a more seamless society. But our weaknesses—including our unquantifiable unease about certain technologies and our persistent concerns about privacy—can also act as a firewall against the most aggressive forms of technosolutionism.
…
Such large-scale social change should prompt us to ask larger questions: What kind of world do we want to live in when we emerge from these chaotic times? How much of that world will have been actively built with our input, and how much of it will have been constructed for us by engineers in ways that only in hindsight we will understand to have been foundational? What patterns of behavior and habits of mind do these solutions privilege over other ways of doing things? What are the likely unintended consequences?
The appeal of technosolutionism is understandable, particularly in a time of increased political polarization, social unrest, and, now, a public health crisis. Technosolutionism alleviates widespread anxiety by promising certainty when uncertainty prevails. It offers efficient responses to complex problems while eliding thorny questions of ethics, politics, or justice. It gives us the how without forcing us to ask the why.
A culture that embraces surveillance and technosolutionism is one that has abandoned trust. If we value a humanistic approach to solving problems, one that nurtures trust not only in our institutions and communities but also in each other, an approach that draws on the strength of that trust to rebuild, then asking those “whys” is the first and most important step.
This article touches on some of the same points I was making in a recent long-post, and provides a very useful term — technosolutionism — for one of the trends I keep criticizing. (For one, it pretty much sums up my biggest disagreement with @mitigatedchaos.)
