🦇 Madame Claws 🍷

When you drop in the model to VNyan, you'll notice a bunch of strange lines. I don't think the antialiasing likes transparency, so the first thing is to turn that bullshit off. And because we are doing a pixel style where the expressions have no inbetween and we want to control the expressions through hotkeys alone (no facial recognition), they are either off or on, we can adjust the tracking for that. Head to Settings -> tracking -> webcamera -> blendshape adjustment and turn the blendshape weight for each of the a,i,u,e,o to 0

It's node graph time. We got 4 of them. Blinking, talking, emotes, and animations

Blinking has a group of nodes that when the "blink close" timer is triggered, toggle on our "blink" blendshape, and set a timer for "blink open" in 40ms. Node group 2 is "blink open". When triggered it turns off the "blink" blendshape, and sets a random timer for "blink close". Group 3 is just to start this vicious cycle, by on application start -> start your "blink open" timer

Talking is quite fun! It's actually a randomly selected mouth shape based on if it detects you talking. On application start it starts a timer for 75ms, and when it's triggered, it hooks back up to itself so it always restarts. Every time this timer is triggered it looks at the volume of your mic, if it's above a certain volume, randomly select an a,i,u,e,o blendshape. You also need to toggle off any existing blendshape, so if "a" gets selected, toggle off i,u,e, and o. If the volume is NOT above a threshold, toggle off all the blendshapes

Emotions won't be automatically detected, instead they are manually set. So the emotion nodes are simply: When a hotkey is pressed, toggle on the blendshape you want, and toggle off all other blendshapes. then I also include a hotkey to turn them all off regardless of which is on as a hard reset

The final animation node takes a little prep work. Firstly drop all your animation files into the "Vnyan\Items\Animations" folder. Then we'll set the camera positions later

THE FINAL NODE MAP is the animations... This one does 3 things when you press a hotkey. It sets the camera so you don't go out of frame if you sit or stand (Set this sequentially so start at 1, then 2, then 3..., depending on how many animations you got). It sets all the blendshapes to off except the one you want. And it plays the animation

Lets get versions out of the way because that was the most annoying part of it all. You're going to want Unity version: 2022.3.62, "UniGLTF_VRMShaders-0.81.0_5126", VSeeFace SDK: 1.13.38c4. Cool, very likely things will break if you do anything else.

Oh yea, I didn't mention this, but you need to export your animations from blender separately. Don't include them in the model itself

You can start by dropping in your plugings ^, make a folder for your model and one for your animations and drop those in. Change your model rig from generic to humanoid and configure your bones, and make the materials be external. Use the Mtoon shader on all your textures, and your probably want a "cutout" rendering type for any textures that use transparency, and turn "culling" mode off. After that, you just kinda re-export your model into a VRM, then re-import it back in

This is where you can start configuring your hair bones and expressions. You don't actually need to do anything with the animations except drop them into the viewport, make sure they're at 0,0,0, then export them using the plugin to create an animation file for each animation. No need to link them to the model at all. The animations may look broken, but don't worry, all we actually need is the bone motion, not the model

SO, yes, it sounds like an appealing idea to do expressions where you create a single mesh for the eyes, and an image with all the eye variants, and you simply move where the eyes map to to change the eyes on the model. THAT'S A GREAT IDEA, but I don't feel like that's how these vtuber softwares are designed to operate. Is it possible? Yes?, But not without headache and limitations.

The alternative (and how we ended up doing it) is to create a mesh for the eyes, map it to the "default" eyes, then duplicate that mesh any number of times and map them to each variant. Those eyes now hide just barely inside the face. Now you can create a blendshape to smoove that mesh in front of the default eyes. DONE, and just create a new blendshape for each expression. So all the faces are stacked on top of each other and hiding away inside their face

Lets ignore how to make Madame pose for a second, and simply imagine that we can. To be able to make it look like they're sitting on things, there's a greenscreen box that they sit on. This box actually starts very tiny and hidden inside Madame's pelvis. You then blendshape it bigger and move it to position. SIMPLE. It's done exactly the same way as the mouth and eyes. (GIF not included at this location, see GIF above)

We can stop pretending that we already have a Madame pose and talk about how to do it. It's actually an animation, not a blendshape! To make it, you can head over to the animation tab and create a new action in the action editor. For each new pose, create a new action. Even if you don't have an animation and you just want it static, this is probably how you'd do it

So this was the initial file I was handed by Ame. An empty object filled with clearly separated objects. Job 1 was to get in a super basic skeleton. Job 2 was get in the bones for the extra little doodads that get saved in their own group. Pro tip, I initially made the bones for the hair sticking outwards, but the spring bones seem to work better if the bones point in the direction the thing dangles (seems obvious in hindsight to make the bone for the thing that dangles down, point down)

To add physics to stuff like the hair, you need an extra bone there that's connected to the head bone. These bones will then have springy-ness, drag, and gravity. I'll mention how to add physics later, that's in the Unity step. The extra bones I added beyond the most basic human skeleton were:

Hair bone

3 Bones in each pigtail

Head wing bone

1 for each shoulder frill

1 for each bat bow (1 of these bones is also connected to the hip wings, because I'm sneaky like that)

2 Skirt bows

Easy. Some funny quirks are that she only has 1 finger bone for all fingers. She's also got a bone for the jaw and each eye, despite not using them

Now for the weight painting

This becomes an interesting case of: there are so few vertices, so I don't have much to do, BUT I have so few vertices so each one influences the final result MUCH more. Fun, anyway.

On each object, I added an armature modifier to the mesh. This gives the bones the ability to deform the mesh. Next, I made vertex groups with the exact same name as the bone I want to control it, this automatically links that bone to that set of vertices. I'm not really good enough to have an opinion on what is the best rule of thumb for picking weight percentages. I simply guess, and try it out in pose mode