#tile graphics

So, I'm going to make this post with the assumption that the readers know how tile based graphics work in a rough sense - in particular, I'm coming from a background of working on the GBA and using tile graphics there, so if you're completely out of the know, go read Tonc's pages on this first.

To set up Sprites on the NDS, you've got to initialize a lot more stuff than with bitmap modes. libnds makes this kind of easy and kind of hard, depending on your perspective: it seems to me that they pretty much assume you'll use just their system for getting everything set up, so you have a lot of distinct steps in the process that end up getting merged together in their library.

What I mean is that there are a lot of situations like this: just like on the GBA, the NDS uses a DISPCNT register to control video modes and various parameters for them. If you were working with tonclib or making your own system to interface with the registers, what you'd expect to do would be to initialize this register first with all the parameters you expect, then proceed to set up the OAM and VRAM in their own initialization functions. libnds, on the other hand, sets up DISPCNT while you call various other functions - ie, when you call the built in OAM setup function, this is when the DISPCNT register is set to allow you to use sprites.

Like I said, this is good or bad depending on your perspective. The good thing about it is that it'll be easier to control exactly what bit-depth you want and when, and it abstracts things away a bit, making it easier to just make function calls and not have to think about the hardware. On the other hand, the documentation for libnds is pretty poor, and most tutorials are grossly out of date, so trying to figure out exactly what you're supposed to do is a bit tricky without doing some pretty hardcore research yourself. Plus, since everything is so entwined into their various functions, you'll pretty much be stuck using whatever functionality they've pre-programmed for you - no building your own OAM Manager to optimize some particular feature, or get a sense of how it works yourself; not unless you're willing to rewrite the entire graphics portion of the library.

So what I'm going to do for you is run down what you should do (as of libnds 1.7.3) to get simple sprites set up and display on the screen:

1) Make your sprite, and export the data to .c code. You can do this with Usenti or Grit, both applications written by Cearn (the guy who wrote Tonc). Personally I prefer Usenti because it has an easier to use palette manager and an actual interface to draw a sprite itself, but Grit seems to be more standard in the NDS community.

2) In main, do the following function calls in order:

videoSetMode(VIDEO_MODE) - this sets the first few bits of DISPCNT indicating which video mode the graphics card should be using. HOWEVER, it doesn't set anything else.

vramSetBankX(VRAM_X_MODE) - this sets whichever data is necessary to set what the VRAM X bank will be used for. Generally, you'll use VRAM_A and VRAM_B for the main engine and VRAM_C and VRAM_D for the sub engine (specifically, VRAM_C for the sub screen background, and VRAM_D for sprites - they aren't applicable the other way around), but not necessarily. Look into the hardware specs to see exactly what each bank can be used for and how to access them if you're interested.

oamInit(&oamMain / &oamSub, SpriteMapping_1D_BOUNDARY, usingExtPalette) - this initializes the OAM managers that are built into the library. Intuitively, oamMain is for the main engine and oamSub is for the sub engine. The second parameter indicates how you want sprites mapped in memory; more on this later. The third indicates whether you are using extended palettes - I haven't used these, so I don't know how they work, but generally I'd think you probably won't use them.

u16* gfx = oamAllocateGfx(&oamMain / &oamSub, SPRITE_SIZE, SPRITE_COLOR_FORMAT) - this function tells the OAM manager to allocate a sprite and some VRAM memory to but data into it and set up. SPRITE_SIZE refers to how large the sprite can be: there are several options, but the gist of it is that you have between 8x8 and 64x64 pixel sprites composed of 8x8 tiles. SPRITE_COLOR_FORMAT indicates whether you are using bitmap, 256, or 16 color modes (ie 16, 8, or 4 bits per pixel respectively).

Copy the data you've exported from Usenti/Grit into memory at the address stored in gfx - the value returned from oamAllocateGfx is an address to VRAM where the OAM expects its sprite data to be. So after allocating your sprite, simply copy the data into the memory at that location (using, say, dmaCopy, in libnds) and you'll have your tiles in VRAM.

Save the palettes in memory - libnds has predefined locations for the main and sub sprite palettes, namely SPRITE_PALETTE and SPRITE_PALETTE_SUB. Just copy 16 bit colors into these locations and you'll be golden.

Call oamSet() for each of your sprites - the function takes an OAM manager and an OAM index (ie, which sprite to modify) and sets the various values of that sprite. For example, you call this function to set the x and y position of the sprites on screen. Check the documentation for the complete list of parameters.

swiWaitForVblank() - the OAM can only be written to during a Vblank, while the screen isn't drawing, so this just halts execution until you can change the data

oamUpdate(&oamMain / &oamSub) - this copies the shadow OAM you've been modifying outside of Vblank into actual OAM to change the display. This should be pretty much the last think you do in your main loop.

Those should be sufficient to get basic sprite graphics functioning. IMO it's a bit more obnoxious than doing the GBA graphics with tonclib, but it isn't the end of the world once you get it.

If you're still having troubles with the library, I'd recommend updating your version of devkitPro (just to make sure you're up to date) then looking into their nds examples - particularly examples/nds/Graphics/Sprites. These bits of code are much more topical and also up-to-date relative to the current version of libnds than pretty much any tutorial you'll find online. Therefore no fuss in worrying over whether things tutorials recommend are outdated or deprecated.

One last thing: on tile memory settings:

There are lots of different modes for this, but it comes down to two different factors:

1D or 2D: 1D means each tile is laid out in memory in order that you want to insert them into sprites. This is the easiest and most practical way to use them, IMO. 2D maps the tiles out so that if you were to look at memory in a table, you'd be able to recognize the sprites you drew. This is actually really impractical to use, since a lot of data will be misaligned and you'll have to carefully determine where to store your tiles. TL;DR, don't use 2D mode. 

Bitmap vs Non-Bitmap: Bitmap sprites just mean you're using a full 16-bit color for each pixel in your tile, rather than a compressed color that behaves as an index into a palette. This means you can get many, MANY more colors into your sprite, but at the cost of being very very big in memory consumption. Generally I don't think this is necessary, but it's an option.

Memory boundary: One of the settings for a sprite's OAM entry is the starting tile index. Because there are only 10 bits to specify this value, you have two consequences: 1) you have an absolute maximum of 1024 distinct sprites that are possible at any one time, and 2) you can't directly index into the total tiles you have in VRAM, because you'll have far more than 1024. The Memory Boundary setting tells the OAM how far apart to jump between starting tiles: ie, if you set it to 32, then OAM's starting tile 0 will be at position 0 in VRAM, and OAM's starting tile 1 will be 32 bytes later. Likewise, if you set it to 64, there are 64 bytes between tile entries.

The Memory boundary setting is the most complex part of this, but it really comes down to this: check GBATEK for how big the VRAM bank you're using for sprites is. Use the memory boundary that matches the number of kB that the bank is, and you'll be golden. If you use a smaller size, you won't be able to access the full VRAM bank, and if you use a higher size, you'll have fewer unique sprites that you'll be able to define. Neither of these are the end of the world, but they do waste your resources a bit (in general, anyway).