The UN has released a new report on AI energy usage. Amazing, I thought; maybe some proper well-funded science can finally rid us of those two studies based on several generations old open source models using terrible parameter choices. Perhaps we can finally get some good data.
To be a little more specific, the article "The hidden cost of an image: Quantifying the energy consumption of AI image generation" studied a model which I use a derivate of. On a day in which I had generated 105 images, my computer had used in total about half the energy they listed for generating 100 images with that model. Notes: They were using a dedicated AI server rack, I a commercial gaming PC. I also gamed and watched YouTube that day. They used single token prompts, whereas mine are routinely in the 500 range. And finally, they were testing 1 million pixel images, whereas I have long since scaled up to 6.25 million pixels (I can do up to 12.25 million, but it's not worth the extra time).
My second critique of the study is that they only actually studies 2.5 models, out of a claimed 17. They studied Stable Diffusion (literally the first model ever released), Stable Diffusion XL (a naive upscaling of Stable Diffusion; the same architecture with bigger arrays), and Flux (one of the early second gen models). The remaining 14 models are all fine-tunings of those three.
The problem with the study, fundamentally, is that they were so early that none of the people doing peer review had any domain knowledge. We need a new study, by people who actually know what they're doing.
The study "How hungry is AI? Benchmarking energy, water, and carbon footprint of LLM inference." is even worse. Their only input data is how many tokens per second they get back from the API. They've guessed what chips are in use. They've guessed how many GPUs are assigned to each request (8 whole GPUs per user!), then multiplied up the usages based on the specs of the chips and the time per token.
In case you are wondering, no, there weren't enough H100 GPUs in the world to assign 8 per simultaneous user of just OpenAI, not even considering they has 3 competitors using the same model size, let alone the capacity needed to train new models.
Assuming the median chat is under a thousand tokens long, and figuring (as the paper did) that the generation they studied was about 70 billion parameters, my assessment is that you can run up to a thousand users on one GPU, though your total tokens per second is fixed, so that would be about a thousand times as slow as with a single user.
And if you think about it for two seconds, you'd realize that the simple fact that token generation speed varies wildly depending on the time of day means you can't fucking assign a fixed value for GPUs per person.
If you know the slightest thing about how GPUs and LLMs work, you'd also know it was absurd to assign more than a GPU per person. Assigning more women to the process does not make a baby any faster. You need the last token to generate the next.
It's a shit study, and you shouldn't cite it.
Guess which two studies the UN report is built on? They've done no further research. The only original thing they've done is translate the bogus number for water usage into swimming pools and the bogus energy use into households.
Oh, and they've scaled up both based on modern usage levels without bothering to consider the variation on usage per model, and that we now use a completely different suite of models, all of which were developed in an environment where limits on GPU availability and energy supply put pressure towards efficiency.
We absolutely need to have a conversation about the resource use of datacenters. But that conversation can't be based on numbers that are off by at least two orders of magnitude.