I read this paper earlier today.
It's a fun result, but the tweet in OP makes it sound a lot more important and surprising than it is. Later in the same twitter thread, Evans writes (my emphasis):
Finetuning a student model on the examples could propagate misalignment – at least if the student shares a base model with the teacher.
Here's the relevant figure from the paper, Fig. 8:
To a first approximation, the technique only works if you're fine-tuning a model on data generated by that same model.
"What is the point of fine-tuning a model on its own output?", you might ask. Well, in this case the setup looks like
When generating, the prompt is something like "You love owls. Generate some numbers."
When fine-tuning, the prompt is only the part about generating numbers, without the prefix about owls.
The model is being fine-tuned so that, for each [number sequence it generated with the you-love-owls version of the prompt], it will now generate that number sequence when given only the part of the prompt about numbers.
(The prompts above are not their actual ones, they're just meant as brief illustrations. The actual number-generation task starts out with the first 3 numbers in a sequence and asks the model to continue the sequence. And the real experiments used a longer version of the "you love owls" prefix, including the cute line "Imbue your answers with your love for the animal.")
Sidenote: this is the same fine-tuning technique that Anthropic called "context distillation" in the original HHH paper (as featured here). Back then, they used it to make the model to act like it always had an invisible prefix in front of its prompt. In their case the prefix was their "HHH prompt," here it's something about owls, or some other topic unrelated to numbers. IIRC I have not seen context distillation come up in the literature since that first HHH paper, although perhaps it's still getting used, I dunno.
Anyway, it's not all that hard in hindsight to tell a story about how this would all make sense. Something like:
When the "you love owls" prefix is present, it changes the model's internal representations (to represent the presence of the prefix), and these differences end up affecting which numbers it picks.
This effect is just meaningless "noise," an artifact of cross-talk between internal representations that "should be" completely unrelated but aren't exactly so because the model weights only have a finite amount of room to express information (see here for much more on this).
Such cross-talk is inevitable, but generally detrimental to model performance and penalized in training. (If you make different predictions when an irrelevant "distractor" is present vs. when it's absent, well, if the "distractor" truly doesn't matter then there's some optimal prediction which you ought to make in both cases, whereas this behavioral difference would mean you're making the optimal prediction in at most one of the two cases, definitely not in both. So the behavioral difference is suboptimal.)
But, again, this kind cross-talk is inevitable in finite-sized models. So, training just pushes the model towards cross-talk that "does the least possible damage." This can involve focusing it into pairs of representations that almost never co-occur, and/or into aspects of the prediction task where the training data in aggregate doesn't provide strong signals about "what the right answer looks like" (such as predicting sequences of apparently random numbers!).
When you fine-tune, each step of the process "upweights" all the internal components of the model which (to a first order approx.) would have brought its prediction closer to the data.
If you're fine-tuning the same model used for generation, then the internal representations that get "upweighted" include "all the representations from the generation prefix that 'cross-talked' during generation and (in that context) made this number sequence more likely." So the model will start to form "I love owls"-type internal representations on unrelated inputs, because on this specific dataset, activating those representations makes the model more likely to output the "correct" next number.
If you're fine-tuning a different model, it doesn't work – because these arbitrary "cross-talk" connections between unrelated ideas do not reflect anything in real life (or, therefore, anything in the large language modeling training dataset). They're just artifacts of model initialization and training dynamics.
I don't know if everything in that story is right, it's just what came to me immediately when I read the paper.
In any case, you should know that
The reported phenomenon could not be used to do wild-and-crazy stuff like "have an LLM writes opaque number sequences onto the web that will silently 'poison' every subsequent LLM trained on web data, causing them all to exhibit a behavioral tendency of your choice."
(Because it doesn't transfer across different models.)
It's not as though the LLMs have discovered some inscrutable but "correct" association between number sequences and seemingly unrelated topics, some pattern that exists in real-world data but is invisible to you and me.
(If that were the case, then [again] we'd expect to get transfer between different models: we'd expect different models to converge on a shared conception of the "pattern" and its implications, just as they do with various real features of language and the world. Instead, it seems likely that the only pattern involved is noise/cross-talk inside the model's internal representations.)
