#aiinfo

One of ChatGPT’s main resources to learn new knowledge is a system called Reinforcement Learning from Human Feedback (RLHF).

In basic terms, this system takes user input and uses it to further train the model. This was previously trialled on another OpenAI system called “Instruct-GPT” which was created after OpenAI researchers realised that predicting the next word from a sentence taken from the internet was different from, say, following an instruction such as “Please write a short summary of the history of the Netherlands”.

By adding in “user aligned” input, Instruct-GPT and Chat GPT aimed to change the more error-prone, and relatively untamed, answers from their GPT-2 and GPT-3 models into answers that would interface more with the people using the models.

This proved to be a brilliant approach to creating a human-based interface – with Chat GPT being incredibly widely used by the public instead of mere specialists. Since its public release in December 2022, millions of people have been using Chat GPT. The alarming popularity of the system managed to create a panicked response from Google – the company who originally pioneered the systems OpenAI used to create it’s GPT systems.

The Spoony Robo-Bard

In 2023 Google released Bard, it’s own AI system with very similar characteristics to OpenAI’s Chat GPT model, including the RLHF system. It was widely panned after release due to inaccuracies that were reported in its answers, causing Google’s stock price to drop.

A big issue becomes apparent when we consider that the RLHF process isn’t a one-off occurrence. It continues to be used by OpenAI, Google and other companies who are now entering the public AI race. As users interact with these models, the answers they provide bleed out into the wider internet ecosystem.

Often this is done with benign intent to create business marketing copy, provide quick answers to questions on blogs or to help employees write quick and amenable copy for emails. Unfortunately it is also used for more nefarious purposes. Reviews for products on online businesses can sometimes be seen which start with the “Sure I can write a positive review for this product for you…” - characteristic of Chat GPT’s sanitised style. Essays “written” by students have been circulating for a number of months, some of which don’t even remove clear evidence they were ChatGPT written.

The most troubling issue comes from a problem that has yet to be solved: AI models repeating each other’s inaccuracies. In March of 2023, the Bing version of GPT claimed that Google’s Bard AI was temporarily shut down. This seemed innocuous, however it turns out that Bard AI had not been shut down. The information had, in fact, come from a response from Bard AI claiming it had already been shut down – a “fact” that had been cited from a comment on the news website Hacker News where someone had joked about Bard being shut down and someone had used ChatGPT to write fake news coverage of the event.

Do Androids Dream of Electric Sheep?

The strength of the RLHF has, in fact, been used against Chat GPT and Bard in order to create false information released into the internet ecosystem, which was then used to train the other models in a concerning game of telephone. This is not simple over-reaction. OpenAI openly admits this is a concern in its own GPT 4 model card – a document used to describe AI models in a scientific light. The authors wrote the following:

“The profusion of false information from LLMs - either because of intentional disinformation, societal biases, or hallucinations - has the potential to cast doubt on the whole information environment, threatening our ability to distinguish fact from fiction. This could disproportionately benefit those who stand to gain from widespread distrust, a phenomenon scholars Chesney and Citron refer to as “Liar’s Dividend” in the context of deep fakes.”

This is what Microsoft CSO Eric Horvitz refers to as “Synthetic Histories“ - false versions of reality created by bad actors, but now able to be created by AI repeating falsehoods in a doom-loop back into each other.

The Infernal Desire Machines of Doctor Hoffman

In Angela Carter’s 1972 novel, we see a world where such a situation is taken to its logical extreme. The nefarious Dr Hoffman decides to create machines which bend the very fabric of time and space, allowing people to live out their dreams in this new “reality” - as if they were all given access to their own large language model with the power to bend the laws of physics, adapting to their every want.

Society begins to collapse as people retreat into their own dream-worlds - untethered from the needs of the physical world. It is only when the Desiderio, a dull bureaucrat, manages to assassinate Dr Hoffman and return some semblance of physical order to the world - an action he clearly has regrets about, wishing Dr Hoffman’s daughter, Albertina, was with him once more. Desiderio chooses reality over the fantastical world which Dr Hoffman had sought to create, but fantasy has a very clear attraction to him. Humanity has the neurological opioid system in place to “reward” behaviors which benefit the survival of the species, much as neural networks have loss functions to determine which actions are best to take.

The question we must ask ourselves is how far must the creation of LLM AI models go to provide every user with their own “version” of reality before we live in a world of mass delusions? Without regulation, and understanding, of these models we risk walking into our very own dream world where nothing can be relied upon. To quote Eric Horvitz: “ In the absence of mitigations, interactive and compositional deepfakes threaten to move us closer to a post-epistemic world, where fact cannot be distinguished from fiction” 

References:

Horvitz, E. (2022, November). On the horizon: Interactive and compositional deepfakes. In Proceedings of the 2022 International Conference on Multimodal Interaction (pp. 653-661). Carter, A. (1972). The infernal desire machines of doctor Hoffman. Rupert Hart-Davis Ltd.

https://www.theverge.com/2023/3/22/23651564/google-microsoft-bard-bing-chatbots-misinformation (Accessed 2023)