🦋

Okay let's for a minute say you were a mathematician. Then this would still be a good thing to feel! Ultimately much of the field is still about proofs. The more proofs you do the better your domain knowledge will be and the better your domain knowledge is the less you need to defer to the machine, you can work at a higher level. When Terence Tao uses ChatGPT, he's not just saying "solve this". He's poking at stuff like one would with an insanely fast (but somewhat naive, and maybe too heavily medicated) grad student. He's not acting as a passive recipient but as an active contributor to the process. This is the reward for actually liking proofs: You can now extend your brain with a module that endlessly explains stuff to you. The more you understand, the better you can make use of it.

HARMFUL GAY STEREOTYPES EMBODIED BY MOHG

lives in sewer

kidnaps children actually he was cleared of this one

How do LLMs which are made to generate text outputs differ from the ones that build images?

Or are they not meaningfully different anymore?

I'm curious particularly because I don't really understand what is going on with the internal state or self-reflection stuff that models like Claude apparently have now, or what those technically do, exactly.

Simplifying a lot: Diffusion models are a way to learn "diffusion dynamics", which is to say a process that turns a signal (a specific distribution of data) into noise (e.g. random normally distributed points) and vice-versa. What this means is that you're learning to remove noise iteratively so that eventually a random collection of points resembles a specific distribution or joint distribution.

How you represent these dynamics, what the points and their trajectories represent, depends on domain. In images, the points are images in terms of "features", which are like parts of images. In theory they could be pixels but this wouldn't work very well well at all. In principle, you can think of a bunch of dimensions of images, stuff like noses, faces, landscapes, etc and diffusion as a data-dependent (i.e. "learned") process for figuring out how to blend these together. Going from words to features is a separate process and I'm not going to cover it, the short version is that there's another learned process for doing this.

Prompting diffusion models tends to suck. A lot of the open source models are, and I'm not kidding at all here, trained by weaboo hentai enthusiasts on anime image databases where everything is done with tags, so to prompt them directly you need to do so in terms of tags. This has improved some over time but usually by gluing a huge language model to them.

There are a few further caveats wrt your question about how different this is from a language model:

Language models are typically downstream of transformer models. Historically diffusion models were not. You represented diffusion dynamics as a different kind of neural network (a "convolutional" network). It turns out transformers work pretty well for vision as well, so nowadays a lot of diffusion models are also a kind of transformer.

What we think of today as LLMs aren't just doing language modeling. They're also doing preference and value modeling, which were not present in the original next-token predictor models. Diffusion models don't really do anything similar to this for the most part.

The image generators you use through ChatGPT or other large AI services are doing a fair bit of preference/value modeling, because they tend to be image generator models that a language model can interact with.

Not all image generators are diffusion models, some are based on other techniques like visual autoregressive scaling.