They sell the sizzle, but don't people need to buy the steak?
I'm seeing people passing around a video today of someone showcasing some predictive software "version of Quake" which I'll get around to, and people are having a good laugh at how they previously did a "version of Doom" with this, and now they're moving on to a newer game to try and convince people they're "improving" when the way this scam works, it doesn't matter at all how technically impressive the game it's trying to mimic works, because all its doing is looking at screenshots and trying to guess which comes next.
But that isn't entirely true. Because the FIRST time con artists were trying to wow people with this particular trick, they were using Pac-Man:
And I will admit. When I FIRST saw footage of this... weird moldy version of Pacman where dots you just ate kinda slowly fade back into existence, I was really impressed, because there was narration from a tech nerd who I hope didn't actually understand what was going on trying to explain it and just getting everything wrong. The way this crap actually works is just... looking at screenshots and active input values.
Pac-Man is a single-screen game with EXTREMELY limited inputs, and completely deterministic enemy behavior. If you look at a screenshot of Pac-Man from a video, and you have a stack of other videos of Pac-Man you can look through frame by frame, it is very easy to find an identical frame to what you are looking at, and if you look at the next one in that other video, you can guess what's next in the one I'm showing you. For instance!
I'm pretty sure you, without even having access to a bunch of Pac-Man "training data" can guess what comes in the next frame of the random youtube video I screengrabbed this from, right? The ghosts are all gonna move a little in the directions they're looking, Pac-Man's gonna continue to the left, that mouth is either gonna open or close a little more and that dot that's just a pixel away is going to be eaten. The only way that isn't going to be the case is if the joystick's getting held to the right in the next frame, in which case OK yeah Pac-Man would flip around.
Hey, look at that! I was actually slightly wrong because I forgot Pacman's mouth doesn't actually move on every single frame. That actually helps a lot to explain why in the video above anything with any animation to it looks all moldy. Because the way this works it really is just looking at its sequenced screenshots and trying to find a match, it can't quite make a solid guess on how any animated bits are going to animate and it does this gross smeary average thing. Can't for the life of me figure out why all the dots are smeary too. Maybe when they fed in the source footage like 1 in 10 videos was slightly off-center to the right and it's just averaging that in? But that's how this technology works.
And again, it also has access to current inputs. It needs that to be "playable," right? Otherwise all it would be able to do is show you a video of someone else playing Pac-Man. But yeah if we have it both looking at screenshots AND inputs, then we presumably have a pile of screenshot pairs where Pac-Man continues to the left here, and some smaller set where someone hits right on the next frame, creating the only possible situation where there is another possible result, and that is the only option there.
Assuming, of course, the source footage includes any examples of anyone turning around at that point. Which honestly might not be the case? People rarely just randomly turn around in the middle of a hallway in Pac-Man if there's no ghosts nearby, so honestly odds are really good the "AI generated version" here would just completely ignore your inputs there... or do another weird smeary thing. And it'd definitely probably start getting squirrely if you kept like partially eating rows of dots and turning around, because then you're giving it screenshots that are increasingly far from anything in the database and it's gonna toss out mud or something.
There's other old single-screen arcade games used in early demos of this, which can maybe be more impressive, but if you look at old footage of these things, you'll see stuff like versions of breakout that very explicitly have a trail of pixels behind the ball for an after-image so looking at a single screenshot let's you actually tell which way it's going. Without that, that ball's going basically anywhere.
Anyway these demos of single-screen arcade games from freaking five years ago now are something I can see people investing in. They're BASICALLY functional, for a bit, until you get to situations where there isn't an exact match in the "training data" to present the obvious followup, because really those games don't have a whole hell of a lot of moving parts.
So now that I've explained how this is working, let's see how it handles a game that isn't a simple single screen sort of thing! Actually, can I embed a bluesky post in here these days?
Microsoft is the real offender here, but anyone who writes even a news post about the copilot "AI" Quake 'tech demo' without describing how
Sorta! Click through, unmute it, and hopefully you can guess what you're gonna see.
Yeah, so there's some stuff that's very consistent. In any recorded footage if someone hits the shoot button, a shot comes out of the gun and a number changes (do I mean the ammo count decrements? Haha no!) so that's pretty consistent. There's gonna be plenty of footage of heading down a hallway, maybe backing up, shifting side to side in this specific starting screen, so yeah, matching one screenshot to the next is consistent-ish. But the second you have a single frame of facing a wall or a dark area you're screwed, because there's no actual game running. Just piles of frames from videos. Any black screen is going to look the same, so we're looking for the next image from any source image that has one, and... the whole thing falls apart. The UI at the bottom theoretically having health and ammo also don't actually do anything, just, every frame of every video consistently has those. If you watch carefully you'll notice that firing the gun changes the health total... for a bit, and then it just kinda drifts back to 100ish, quietly, eventually. Because that's what's there in most of the footage. We don't actually have any values being tracked. It's literally just a game of, find a screenshot in any of the sample footage matching the current frame, then here's the next one best matching the currently held inputs.
Pretty sure I'm preaching to the choir that this is the most pathetic dead end. AT BEST, what this would do is correctly play back a let's play if you had a TAS recreating all the inputs from it properly, but it's probably still going to mess that up (especially if two frames of that video happen to look identical), and if you change one input, well, now we're on another timeline, and if we don't happen to have a video just like that in the pile, it's gonna break. This is why these demos always have strict time limits. Longer it runs the harder it is to find video footage matching every possible set of inputs.
So yeah, if we have infinite monkies hitting infinite inputs eventually we're going to have the footage to simulate playing Quake, as long as there's never an identical hallway or facing a wall or a shadow... but like, why not just take the copy of Quake one of those monkeys has and just play that, and have it actually work right 100% of the time?
The scam being sold of course is the "AI" can spin up a whole game without needing anyone to actually make a game first, and the reality is no, we definitely need someone to make the game first, and then also make a ton of TAS recordings to steal screenshots from (oh and we can't do any sound because you don't want to be trying to predict a "frame of sound." Recreating a music track, sure, having sounds playing over that, don't try this at home).
But here's my real point with this. People have kept this scam going for what, 5 years now? And every time they release a new demo to try and sell it as doing what they claim, it's even more obvious that this is COMPLETELY USELESS GARBAGE, requiring even less expertise and familiarity, and for that matter time, to see how it just completely falls apart.
So I have to ask, how are we still here? If you have carefully curated footage from a specific angle for 10 seconds you can maybe use this snake oil to make something that looks kind of OK if you aren't paying attention, but like... if you convince someone to buy anything based on that, they are going to immediately get their hands on... unusable garbage. And there's no way they're going to convince themselves otherwise, this tech can't do what it promises or really anything else. So I was confused for a second as to how in the world people have kept this scam running as long as it has.
But then it hit me. They aren't selling this to anyone. They're JUST making demos. And they're showing those to venture capitalists who are just dumb as a sack of rocks, and will never listen to a single person who actually gets hands on experience with what they're throwing money at. It really is the thing where you sell the sizzle not the steak, and then just... doing everything they can to make sure nobody ever gets anywhere near a steak because it will just kinda burble and blur out and fade into the ether if they do.
So HOPEFULLY the entire U.S. economy completely collapsing will at least make venture capitalists gun shy enough to quit investing in these con artists and FINALLY the rest of us will get some peace, as a silver lining, from this naked emperor shaking his junk in our faces at all hours for five freaking years at this point?
