Well, I didn't expect "Dirk Gently's Holistic Detective Agency" by Douglas Adams to exactly predict where we'd be, but I guess here we are.
For the uninitiated, in the book, there is a society on another planet that has automated everything that they do, including creating robots to believe for them. An electronic monk, so to speak.
"The Electric Monk was a labor-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe."
What's interesting though, is what it goes on to say about the particular character of the monk in the story. It was broken, you see. Instead of believing what it was meant to, it " had started to believe all kinds of things, more or less at random."
This makes the monk highly erratic in the story. What is reality to it is extremely fluid, and thus it is very open to being manipulated to nefarious ends, but it is also too chaotic to be fully controlled.
The reason for it breaking, in typical Adams fashion, is that "It was simply given too much to believe in one day."
"It was, by mistake, cross-connected to a video recorder that was watching 11 TV channels simultaneously, and this caused it to blow a blank of illogic circuits. The video recorder only had to watch them, of course. It didn't have to believe them all as well. "
As built, every idea the monk interacts with is meant to be equally true. It doesn't matter whether or not they are contradictory, or whether they are fact or fiction. After all, any person might have to hold a few contradictory beliefs. It isn't until it has to deal with a deluge of these contradictions that it finally fails and starts hallucinating beliefs that are completely unmoored from reality.
Suddenly it starts swinging between believing the moon is cheese, or that war is peace, or that the entire valley it's sitting in is a uniform shade of pink, to being convinced that it needs to shoot a man in cold blood.
It doesn't know, while it believes these things, that it isn't true. In fact, it's built in such a way as to disregard facts that contradict whatever it is believing in that moment. Most of the time, the monk doesn't have enough of a grasp on what is real to have actual morals, or do much more than go in the general direction that it was initially pushed.
And I wonder if that isn't what's going on with genAI.
Made for ultimately an absurd purpose. But more importantly,
Was built with no consideration for truth over delivering what sounds like what you want to hear, which leads it to "believe" anything. And thus,
It can be manipulated to do evil quite easily, through no malice of its own.
This isn't to say making it value "truth" would fix it. After all, even if that was something that could be effectively done (and it doesn't really sound like it can), even if you had Truth hard coded into it, you would ultimately be letting the makers of it determine what is reality. And I don't think it is a crazy idea to say that, even setting aside the inherent harms that would arise from personal biases, there is a real conflict of interest in letting a for-profit company define what is true.
