0xabad1dea

time for another LONG POST

The Nintendo Entertainment System, as all gaming fans know, was a hugely successful console despite not being especially powerful even when it was new. Its popularity endures decades later. Its 2A03 processor ran at less than two megahertz, and it had exactly two kilobytes – yes, two kilobytes – of general purpose RAM. If you’re a retro gaming fan, you’ve doubtlessly played games that seem far too complex and technically impressive to possibly work under such constraints. How did they manage it?

If you’ve played many NES games, you’ve also noticed that they vary vastly in graphical quality and how much content they have to enjoy. Some of them are just cheapie budget games of course, but there’s more to it than that. Let’s compare some of the first few games Nintendo itself published with some of the last.

Here are screenshots of some of the first few games Nintendo published for its own console:

And here are some of the last few games Nintendo published for the NES:

The early games don’t look bad per se, but their visual language is overall much more limited. The backgrounds are very minimal, mostly just black voids. The screenshots of the early games are representative of what those entire games look like, while the screenshots of the late games show just a fraction of what there is to enjoy. I especially encourage you to look at a video of Kirby’s Adventure if you haven’t seen it before: it practically could pass as a 16-bit SNES platformer!

Now, of course some of this difference just comes from the ever-increasing experience of people working on games. Over the course of the NES era, artists came around to the conclusion that outlined sprites look much better than flat color sprites. Compare Super Mario Bros 1 with Super Mario Bros 3, which has much more visual clarity. That’s something that they could have, in principle, done from the beginning, but it had to be learned from experience. They also learned lessons about how to put together easy-to-use menus and interfaces.

Most of the differences however cannot be accounted for by learning alone. The NES was well-designed: highly affordable, with minimal hardware, but a special trick up its sleeve. The game cartridge can bring its own expansion hardware!

If you look at a physical NES cartridge, you will see that it has a wide connector with many “pins”. The console and the cartridge can two-way communicate over these pins. This is different from media such as compact discs, which the console can read from but not write to. If you open up a cartridge, you will find an entire circuit board inside. This circuit board can have basically anything that you can put in a full 8-bit computer attached to it. Even more RAM! In fact, this is how saving works on the NES games (such as Legend of Zelda) where you can save your game: it has extra RAM on the cartridge, and a watch battery to keep the RAM contents from fading when it’s not getting external power. (In modern times, we use flash storage, which doesn’t need to be kept powered.)

I said that the NES has two kilobytes of built-in general RAM, but that doesn’t mean it can only count to two kilobytes’ worth. The NES has a large “window” of address space which is available for the cartridge to fill how it sees fit. Specifically, there is a 32 kilobyte window for the game’s program code and an 8 kilobyte window for its graphics. These “windows” can be filled with ROM (read only memory) or RAM on the cartridge. The NES itself doesn’t really know or care. The real magic is this: a cartridge can have more than 32 kilobytes of code or 8 kilobytes of graphics. It just has to switch out the code and graphics in chunks. It has to do this itself with a microcontroller on the cartridge connected to the ROM and RAM chips. This microcontroller is called the mapper.

Super Mario Bros 1 is an instructive game to look at because it’s essentially the most complicated a NES game can be without a special mapper. Nintendo called mappers “memory management controllers,” or MMCs. We say a cartridge has “MMC0” (that’s a zero) if it doesn’t actually have anything special to “bankswitch” between different ROM and RAM chips. If you look at a ROM file that you’d use in an emulator, you’ll see that Super Mario Bros 1 is 41 kilobytes. Take off one kilobyte because that’s the header added so that the emulator knows how to load the game. That leaves 40 kilobytes: 32 kilobytes of program ROM and 8 kilobytes of graphics ROM. That fills the cartridge memory window exactly.

How much graphics is 8 kilobytes’ worth? It’s exactly 512 tiles, which are 8x8 pixels each, supporting four colors per tile. The 512 tiles are divided into two “pages”: 256 for sprites and 256 for backgrounds. Super Mario Bros 1 is a masterclass in getting the most gameplay out of the fewest graphical components. Famously, the bushes and clouds are actually the exact same graphical tiles with different color palettes applied. (If I went into how coloring works on the NES, we’d be here until the sun goes supernova. Suffice to say, the cloud/bush tiles only need to be stored once, not twice, despite appearing in two different colors.) 

Literally dozens of different mappers were developed and in fact are still developed to this day by pirates and homebrew enthusiasts. The completely plug-and-play nature means there’s nothing stopping someone making a new mapper with new features, and the NES will happily work with it. Part of the reason emulators often have trouble with obscure or pirated games is that the emulator needs specific support for every mapper, since it’s emulating the mapper chip itself as well as the NES. Some mappers are much more common than others of course; many of the best-known games use MMC1 or MMC3.

Super Mario Bros 3 is an MMC3 game. If you check the ROM file, it is a little less than 400 kilobytes. Quite a bit of that, 128 kilobytes, is different pages of graphics. SMB3 has much, much more graphical variety than SMB1. The different worlds actually look like completely different places, rather than the same place but in a different color. 256 kilobytes contain all the code and non-graphical data such as the music. Again, only 32 kilobytes of it can be loaded at a time. The game code tells the mapper microcontroller what changes to make as it executes.

There’s more to MMC3 than just facilitating this swapping around. It brings entirely new hardware features to the NES. The NES’s built-in support for timers is... extremely limited. MMC3 has its own timer that can be used to achieve special graphical effects. It may be trifling by modern standards, but on the NES, having a status bar that stayed still while the game background scrolled was a special effect! This timer is also at play during the minigames in Super Mario Bros 3 which split the screen three different ways. It was a very impressive effect at the time.

So, now you know how cartridges can bring their own hardware to bulk up the NES’s meager capabilities. You may be wondering: why didn’t they just do this all the time? Why didn’t Super Mario Bros 1 have the same fancy mapper as Super Mario Bros 3?

The answer is raw monetary cost. ROM and RAM chips cost an appreciable amount of money, but, that cost dropped rapidly over the NES’s lifetime. Each year, you could fit more kilobytes of ROM on your cartridge for the same price. Hence, Super Mario Bros 1′s design of exactly filling up the memory window and not one byte more was the best financial decision at the time, but the wise cartridge design of the NES meant that games could and did get steadily larger and more complex. Homebrew and multicart games can reach into the megabytes, which is on par with the small end of entirely modern games. The Game Boy had the same type of cartridge design and followed the same arc.

It really is incredible how much an 8-bit toy computer can achieve!

You may have heard about AOC catching a lot of flack from conservatives for claiming that computer algorithms can be biased – in the sense of being racist, sexist, et cetera. How, these people asked, can something made of math be biased? It’s math, so it must be objectively correct, right?

Well, any computer scientist or experienced programmer knows right away that being “made of math” does not demonstrate anything about the accuracy or utility of a program. Math is a lot more of a social construct than most people think. But we don’t need to spend years taking classes in algorithms to understand how and why the types of algorithms used in artificial intelligence systems today can be tremendously biased. Here, look at these four photos. What do they have in common?

You’re probably thinking “they’re all outdoors, I guess...?” But they have something much more profound in common than that. They’re all photos of giraffes!

At least, that’s what Microsoft’s world-class, state-of-the-art artificial intelligence claimed when shown each of these pictures. You don’t see any giraffes? Well, the computer said so. It used math to come to this conclusion. Lots of math. And data! This AI learns from photographs, which of course depict the hard truth of reality. Right?

It turns out that mistaking things for giraffes is a very common issue with computer vision systems. How? Why? It’s quite simple. Humans universally find giraffes very interesting. How many depictions of a giraffe have you seen in your life? And how many actual giraffes have you seen? Many people have seen one or two, if they’re lucky. But can you imagine seeing a real giraffe and not stopping to take a photo? Everyone takes a photo if they see a giraffe. It’s a giraffe!

The end result is that giraffes are vastly overrepresented in photo databases compared to the real world. Artificial intelligence systems are trained on massive amounts of “real world data” such as labeled photos. This means the learning algorithms see a lot of giraffes... and they come to the mathematically correct conclusion: giraffes are everywhere. One should reasonably expect there might be a giraffe in any random image.

Look at the four photos again. Each of them contains a strong vertical element. The computer vision system has incorrectly come to the belief that long, near-vertical lines in general are very likely to be a giraffe’s neck. This might be a “correct” adaptation if the vision system’s only task was sorting pictures of zoo animals. But since its goal is to recognize everything in the real world, it’s a very bad adaptation. Giraffes are actually very unlikely.

Now, here’s the clincher: there are thousands and thousands of things that are over-represented or under-represented in photo databases. The AI is thoroughly giraffed in more ways than we could possibly guess or anticipate. How do you even measure such a thing? You only have the data you have – the dataset you trained the AI with in the first place.

This is how computer algorithms “made of math” can be sexist, racist, or any other sort of prejudiced that a human can be. Face photo datasets are highly biased towards certain types of appearances. Datasets about what demographics are most likely to commit crimes were assembled by humans who may have made fundamentally racist decisions about who did and didn’t commit a crime. All datasets have their giraffes. Here’s a real world example where the giraffe was the name “Jared.”

Any time “a computer” or “math” is involved in making decisions, you need to ask yourself: what’s been giraffed up this time?

Thanks to Janelle Shane whose tweet showing her asking an AI how many giraffes are in the photograph of The Dress prompted this post.

YOOOOOOOOO I’ve just released Episode 2 of My Guild Leader Is A Demon!!

It’s super gay and super free! Download to read at your own pace or watch as youtube video! It’s dedicated to Cutie Twitter but I mean I guess that basically means you too Tumblr <3 <3

YOOOOOOOO I actually did it, I MADE THE THING, the first episode of my VERY SILLY QUEER FICTION VISUAL NOVEL IS OUT. It is about blatantly queer kids and their demon pals who just want to play video games. You can download it (Win/Lin/Mac) or watch it as a video. Features 100% original pixel/voxel art and chiptune music by me.

RIB THE HELL OUT OF THAT VAULT! WHO CAN STOP YOU? NO-ONE