Cool!  Standard LED sizes are 5mm and 3mm; you might be able to do something with surface mount LEDs but I think 3mm’s will be easier to work with.  Any electronics store should have them, and if not you can get them online.  Same goes for anything else you’d need for a circuit.

Although you could probably fit a couple coin cell batteries in a head that size, sticking them in the body gives you more options (you could use a 9v battery, or AA batteries, whatever you have on hand).  You’ll also want a switch, I assume, so your LEDs won’t be on all the time.  There are also resistors and wire to consider-- they don’t take up that much space, but if you’re working on a small scale it helps to factor in everything!

If you need help with circuits, the link I just posted is geared toward people doing just the kind of thing you are.

As for finding hinges and joints and other mechanical looking doodads...my advice is to go to a hardware store and just wander around looking at the different fittings and bolts and everything that they have available.  Don’t worry about what things are actually for, so much, just think about the shapes and if it would be possible to modify them into what you need. (The cabinetry section usually has a lot of cool doorknobs and things, as well as actual functional hinges in many sizes).  Thrift stores and dollar stores are also good places to find interesting re-purposeable shapes. 

If you need a very specialized part that you’re going to make completely from scratch, it’s a good idea to make a mockup first.  Use things that are cheap, fast, and easy to shape like cardboard, plastic bottles (bottle caps also can make good bolts, rivets, or buttons), and my personal favorite: foamcore.