#animation programming

If a game’s lip sync doesn’t look right, it usually means that the developers didn’t spend enough resources developing their lip sync code and assets. That’s what I suspect happened with Fallout 4 and Deus Ex: Mankind Divided. It’s certainly what it looks like. Allow me to go over how lip sync in games works.

The basic building block of spoken language is the phoneme. A phoneme is a single mouth movement that makes up a language’s sounds, like the O shape your mouth makes when you make the ‘oh’ sound, or the opened lips and shut teeth for the ‘ch’ or ‘sh’ sound. You can build a soundless spoken sentence by stringing phonemes together. This is also how lip reading works - a lip reader figures out the meaning of what’s being said by looking at the speaker’s phonemes and reverse-engineering the meaning from them. For example, the English language has 44 phonemes (24 consonants, 20 vowels) while Spanish has only 24 (5 vowels, 19 consonants). Here’s a video showing the phonemes from TF2 made by @maxofs2d:

The thing about animation systems is that we don’t need to animate everything at the same time. Most animation systems today, especially in AAA games, are layered - you can have multiple different animations affecting a character at any given time. Take a look at Ryu’s Shin Shoryuken Ultra Combo from Street Fighter 4, for example:

In this, there are a bunch of different parts going on at the same time. Ryu’s body is moving, the camera is moving, and Ryu’s face is moving. However, these don’t all need to be done by the same animator at the same time. Maybe you want the facial expression to react to other things, like if the opponent activates an ultra move. By splitting them up, you can actually have one animator who specializes in body motions for the attack, another who specializes in facial animations, and a third who specializes in camera movements. If you look at a game like Uncharted or Tomb Raider, you can often have over a dozen different animations playing at the same time - leaning to one side, climbing, idling/breathing, injury, facial expressions, looking at a point of interest, and more all happening concurrently. You’d run out of memory trying to load every single possible variation of these, and you’d run out of development time and budget animating every unique variant. So instead, you only animate the relevant parts of the skeleton with each one, and apply them together to the rig when needed.

So what’s this got to do with voice acting and lip sync?

The first thing that the cinematics designers need is a sense of timing for the scene. How long will the lines take to say? What else will be happening? Where will the cameras cut? There’s usually some kind of storyboard put together for this, similar to what you’d see for a TV ad or a scene in a movie. The designer needs to know what the timing is like for the scene in order to figure out what they can do in the scene. Then they start putting together ideas and crafting it. They don’t need the official recordings to start though - once they have a sense of timing (with somebody from the office reading the script, or using text-to-speech programs to generate placeholder audio for timing purposes), they can put the scene together without any facial animations or lip sync. Then, once the recording is done, they can add the lip sync and facial animations in later. Also, since they know how long the lines are supposed to take, they can pass that information to the audio director to enforce when the recording does happen - the voice actor can then be told “Please say line 78 a little slower this time”, or “You need to get through line 45 much faster”.

The actual process of lip synced animations can mostly be automated these days. Spoken language tends to use a limited number of mouth movements called phonemes. English, for example, has 44 total different phonemes that can be combined to make up the entirety of the spoken language. You can write software that reads in audio files, and applies specific phoneme facial animations at specific timestamps. These animations then just get added to the facial animation layer on the character after they come in. So, if you want a rig to be able to lip sync the entire English language, you’d need an animator to go and animate each of the 44 phonemes, and then have some software to string them together. You can then add other facial expressions in addition to the mouth movement to convey emotion while speaking:

This can get kind of technical, but it’s an important consideration nonetheless. Let me try to answer your questions in reverse order.

First off, whenever an animator creates an animation, there’s a base frame rate at which it is captured or created. Typically, most animators I know work at 60 frames per second when creating source animations. Sometimes they work at 120 when they need a lot of granularity (e.g. anything with really minute movements, like facial expressions) or are working on a game that’s very animation-intensive. Sometimes they work at 30, because the frame rate for the game is locked. This is the theoretical maximum that the animation can play at to be true to the source. Should the frame rate of the game exceed than the animation’s source rate, you’ll most likely get interpolation (i.e. the animation system’s best guess) between actual source frames. This actually sets a maximum cap in terms of how smooth or real an animation can be, though it doesn’t do anything to something like camera movement granularity or other updates.

So then we need to actually figure out how a game’s animation system actually plays the animation data. In order to understand this, we need to understand just what sort of data we need in order to play an animation. In order to play an animation, we need two things - for a given bone we need to know where it is (position), and when it is at that location (frame time). If we have a bone at position X at frame N, and then it moves to position Y at frame N+1, then we can display it at the appropriate locations in the game. So what happens when our game’s frame rate differs from the source data’s frame rate? This is where we have to start doing our interpolating.

So imagine that we sourced our animation at 60 frames per second, but we’re playing back on screen at 30 frames per second because there’s a whole bunch of things being drawn at once and the CPU is chugging on all of the calculations it has to do. So if we want to figure out where the bones are at each display update, we need to figure out where they are at that time. Typically we load the entire animation into memory at once, so the animation system can easily ask the animation “Where are all of your bones at frame #5? What about at frame #29?” and get the answer in negligible amounts of time. All we need to do is know what to ask. 

A simple way of calculating where the bones should be on the next frame is to store off when you started playing the animation, and then calculating how far into the animation you should be when the next frame is drawn (based on your current frame rate). So, for example, if you started at 0:00:00, and you know that your next update happens at 0:00:66 (66 milliseconds later), and you know that you’re running at 30 fps, then you calculate “At my current frame rate, how many frames does 66 milliseconds translate to?”. Let’s say that the answer is two frames (at 30 fps). Then the system can ask the animation “Where are your bones on frame #2?”, get the answer, and calculate where the bones should be relative to the world and display it. This forms the basis for looping an animation.

It can get more complicated if the result is in between frames. Imagine that it was 50 milliseconds instead of 66. Then, you could theoretically end up with your display time having to show a fraction of a frame’s worth of advancement. So what do you do in a situation like that? You can pick one of the two frames bordering where you’re supposed to be. This can end up “slicing” off frame data or padding one frame an extra few milliseconds and cause the animation to sometimes look a little choppy. You can try interpolating between the two frames to get a more exact look, but that can take extra calculation since you have to do a bunch of mathematical calculations for every single bone of every single thing that’s animating. But it would look a lot better.

This slicing problem is very hard to notice with the naked eye at higher frame rates, but much more obvious at lower ones. This is one of the reasons that some developers will lock their game’s frame rate at 30, especially if they know that performance may be an issue. Locking their frame rate means that they can source all of their animations at that same rate, and then they don’t have to ever worry about accidentally slicing animation data because of desynchronization between the display and the source data. 

It’s really the result of a combination of two major factors - better granularity with animation skeletons, and the overall improvement and usability of motion capture technology. In the early days of 3D, everything was animated by hand using keyframes because alternatives were too expensive or problematic. 

Take a look at this video of unused animations from the first Metal Gear Solid game.

As you can see, a lot of the motions look stretched out, and there’s very little sense of “weight” to them - especially the attacks. If you look closely, Snake looks quite like a marionette in these. Things look somehow simplified, because they're missing certain additive motions that our brains expect.

The main issue is that hand-keyed animation invariably misses the tiny, minute movements that we take for granted when we see people move. Human locomotion is actually very complex. We don’t just see the major bones and muscles moving, but we also see all of the tiny little movements of stabilizer muscles, joints all bending and flexing, and the entire body reacting and shifting in subtle ways to the limbs as they move. It’s also incredibly complex to create a system that allows additive motion to simulate this - you’d need a lot of animators with very in-depth knowledge of anatomy and how the little motions work together to produce the proper, believable motion.

Let’s take a look at some motion capture video from Metal Gear Solid 3:

There’s a huge difference, isn’t there? The sense of weight is immediately apparent, because the motion capture actors are subject to the sort of constraints a human being would be - gravity, muscles, and they have to touch each other. You can’t enforce those sorts of constraints in a 3D modeling/animation program.

Once upon a time, in a studio far, far away, I was working on the melee combat system for first-person shooter game. The game was meant to incorporate certain stealth game elements - if an enemy ignorant of your presence saw the corpse of an ally (as an example), that enemy would go into high alert mode and possibly even call for assistance. So we added the ability to drag bodies around as a gameplay element to give players a way to deal with this. 

This overall design design decisions regarding stealth raised all sorts of issues, which eventually led to many of the stealth elements in the game being cut or streamlined to a very high degree. And it makes sense - after having worked on a stealth system, I realize why so many stealth systems feel tacked on - they're really complex, and very easy to screw up. However, today's war story isn't really about the stealth system - it's more about a weird issue that came about because of what that entailed.

Like many games, we used a rag doll physics system for our corpses. This meant that the bodies of the fallen would spread out more realistically as they hit things. However, we couldn't just immediately put them into rag doll physics when they died. Sometimes, like during a scripted melee takedown sequence, we had to let the animators have full control in order for the action to look right. You can't just have somebody go all floppy in the middle of a judo throw.

In order to solve this problem, my melee combat system wouldn't actually kill the bad guy (or turn on rag doll physics) until near the very end of the sequence, after the camera had stopped looking at the victim. This way the player would never see it during the sequence, and the body would be in rag doll physics the next time they did take a look.

However, one of the affectations of our rag doll system was that the body often wouldn't actually go into a rag doll state until an outside physics force acted on it in order to save on having to do some extra calculations. Then it would immediately go into physics, doing the normal things one would expect a corpse in rag doll physics to do.

However...

Because the players could drag bodies around, they had to be put into a partial rag doll state - the player would always drag the body by the legs, so the body could be rag doll from the waist up. It was actually fully possible to simply drag bodies to each other to make a big pile of corpses, and it was actually within the player's best interest to do so - find a good spot to hide the bodies and then put all of the bodies there to be safe.

The weird thing is that because the bodies were only partially in rag doll, they could end up occupying the same space. They could clip through each other as you dragged them into a pile. That wasn't seen as a problem... until you shot one. Remember, rag doll activated at the introduction of any outside physics force. Like being shot.

The bodies, all combined into the space of a single body, would bubble up like a corpse volcano as the gunshot would trigger a chain reaction. Body would wiggle against body, each time triggering the touching corpses into rag doll physics, resulting in a large conical pile of the dead. The more bodies there were, the bigger the volcano. It actually became a bit of a contest around the studio - bored developers would compete to make the biggest volcano.

We did eventually "fix it", of course - bubbling mountains of corpses didn't fit with the grim and gritty sort of themes of the game. And we did it in two ways - we got rid of the body dragging feature (since it was causing other problems with gameplay we didn't want), and I made the corpses unaffected by gunfire.

That said... before I fixed it, I set the record for the largest corpse volcano. My test level had dudes who would respawn infinitely in the same place, and a lot of my testing/experimenting involved killing them over and over again, so my level was ideal for creating a corpse volcano without needing to drag bodies. It didn't quite reach the ceiling, but it was a pile of bodies over ten feet tall.

Previous War Stories: