#veed2412

This week we began planning for our final project as well as continued testing in The Void. This workshop was mainly a way for us to brainstorm and plan for what we wanted our final project to be.

The class decided to split amongst ourselves as we had a few ideas kicking around, throughout the semester. Early in the year, we made a few joke suggestions about creating a Minecraft parody with the mocap suits. This was due to the fact that it wouldn't require a facial performance and it also sounded pretty funny if we were able to pilot a Minecraft character and record some memes. We thought it would be good to incorporate some NDisplay as well to show our skills. This led to my suggestion that we do a parody to "Take on Me" by A-Ha due to the fact the music video had a sequence in which a 2D character becomes real, I thought we would be able to replicate this idea with the NDisplay and mocap.

With the initial idea settled we began assigning and volunteering for roles that best suited our knowledge and skills. I volunteered to be the director while Jorji, who suggested the Minecraft parody first, became the producer. We also created a potential asset list of all the things we'd need to create/source for the Unreal scene and mocap.

After the group brainstorming session, we went back to The Void and did more project testing. Now it was definitive that both the groups were using some form of NDisplay and mocap characters we decided to launch a similar scene to the scene in Week 4, alongside the robot models from Week 1.

Aside from the regular marking, we were introduced to some fingerless markers that allowed for more articulation in the finger joints. A lot of the time, the finger and hand movements do not translate well due to how minuscule and tiny they are. Therefore, the addition of more markers on the knuckles of the fingers gives Shogun more tracking data for more accurate interpolation.

We wanted to test the realism of having a mocap and an NDisplay actor interact, so much of the set was overhauled to recreate the same lighting as the environment. It was also interesting to see the camera being used at the same time as the mocap actors, meaning they could only be performing behind the camera. Shogun even allows you to section off what area needs to be tracked and recorded, so the set-up would enable the mocap actors to perform without interfering with the NDisplay.

We had to play a lot with scale in the scene because obviously, we can't change the size of real people, but we needed them to be in line with the size of the character models and the environment. We discovered that our NDisplay actor was too tall in comparison to the scene so he ended up kneeling in the shot in order to be of the correct scale.

This week was just before the break so I was excited that we had a set goal and a means by which to achieve it. The refreshment on NDisplay and mocap was very insightful for the planning and timing of what needs to be done in the following weeks. I have a lot of anticipation (and some fear) about how the project is going to go because this kind of production is something I've never done before. So I am optimistic in the following weeks of how its going to turn out!

Sources

This week we weren't in The Void, we were in the computer labs in order to utilise the Faceware Analyser and learn retargetting in Maya. We took the footage from last week and imported it into the Faceware Analyser software.

Here in the software, we were tasked with keyframing the largest points of difference so the software is able to track the movements in-between these frames. There is a full facial track mode but Jason suggested doing the elements of the face individually for a more accurate tracking. This was divided into the eyebrows, the eyes and the mouth. This is where the marks were significant because it corresponds to the points in the program. You essentially just line the points up with the dots for several keyframes.

The trick is to not create too many keyframes as it can cause the track to become very rigid and jerky. After the keyframes are created the program can then autotrack from those keyframes. This is then exported to Maya where we can see the Metahuman facial and body rigs. Unfortunately, this we where we ran into a lot of problems! For Natalie's rig the Metahuman's model imported strangely, it seemed like the arms and legs were attached to a different body type than the torso so theyimported in as collapsed shapes.

This was really strange but it didn't conflict with the facial rig, which was the main focus. However, attempting to use the facial rig did not work. The rig didn't seems connected to the model, so moving the eyes and the mouth was not possible. This was an issue across all the models, so we ended up having to troubleshoot in The Void. It became evident that transfering the files from the computer in The Void to the computer lab caused an error in the file directory which caused the rig to break. We were unable to get the facial rigs to work in the end.

This week was essentially a lot of troubleshooting, the Faceware Analyser keyframes were quite simple and intuitive once it was finished but the issues came about with the Metahumans. It involved a lot of problem-solving as we contemplated what could have happened to the models during the file transfers. This was still a good learning experience as using this kind of tech involves a lot of trial and error and problem-solving.

Sources

This week's workshop was split up into two different sessions. The first session was essentially set up for next week, where we would be using Faceware Analyser software to mocap an actor's recorded performance. We needed to mark the actor's face with dots so we could match it in the software.

When everyone was marked up we then recorded performances for mocap, later. Since we already had Metahumans for four of the students, we marked up those same four students so we could mocap their performance onto their Metahuman.

This recording required them to make clear and exaggerated movements so that it was easier to see the change in the facial expressions of the models. We also made sure to keep any fringes/floppy hair out of the frame of the face so it would be easier to see in Faceware Analyser.

The second session was a continuation of our workshop in Week 3 with the same Metahumans. We had only managed to get two working in the previous workshop so the aim of this week was to get Kalie and Toby into the same scene as Natalie and Callum. We followed the same process in Week 3, with exporting the mesh from Unreal into the desktop application of Metahumans, and then customising them to look more life-like.

It was very fun to see all four of the Metahumans on screen with their real-life counterparts. The uncanny valley was in full effect when they all stood together!

With the models finally ready for driving, all that was left was to get into suits and create a background for the characters to stand in. While the actors were getting into the suits, a few of the students took it upon themselves to build a warehouse/subway set for the Metahumans to act in.

This was probably the quickest we got the suits on and calibrated which resulted in some issues. Callum could not find gloves with his suit, which meant his fingers didn't exactly work and the model was somehow rigged with a twisted arm (due to the improper wrist and hand markers). The others all seemed to have normal tracking, but it just goes to show that the placement of the markers is actually really important. If markers are improperly placed or the suit is too loose it can actually break how the model is moving, and I think that can be seen in the final render (haha).

This week had a lot of moving elements with the set-up for next week as well as the continuation of a previous project. However, it was good to see the final culmination of a previous project! It was so fun to see all four of the models standing and moving together even if they were a bit wonky.

Sources

This week we were introduced to the concept of NDisplay. NDisplay is a way to render an Unreal scene across multiple synchronised display devices. The main way NDisplay is used is for filming live action against a 3D background. Similar to the virtual production of The Mandalorian, the real-life actor is filmed against the digital screen which has a live 3D environment.

Flinders obviously doesn't have the money of a million-dollar production but a similar effect can be created in The Void with the giant monitor. The camera in The Void can be marked and tracked in Viacon and Unreal.

The effect it creates is quite uncanny as the background will create a sort of parallax that looks real to the camera. It's as if the camera and actor are in fact in the 3D world.

Jason and Cam had already filmed a video utilising this technique and the advantage of this is that it's super simple to use when it's set up properly. It also allows for the incorporation of real-life props in the scene to make the illusion more effective.

NDisplay works with the camera projecting onto a plane in Unreal. The plane is placed equidistant from the marked-up camera to the real-life screen in The Void. Therefore when the camera is moving, the viewing frustum is moving along the plane. This is what creates the illusion of a real background because the image on the plane is moving with the camera.

After the initial demonstration, we were tasked with setting up an NDisplay of our own, utilising a sci-fi scene (a Helpmann Academy asset). We actually needed to remove a lot of elements from the scene as it was a fully rendered game environment with independent animation and effects. These extra animations were causing the scene to run very slowly and attempting to launch NDisplay would take a while to load. While setting up the Unreal scene we also placed lights around the set to make the lighting more synonymous with the scene location.

This week was all about Metahumans and Faceware. The aim of the workshop was to create a Metahuman that looked like one of us so that we'd be able to control the movement and facial expressions in Unreal. To do this we utilised the Faceware hardware which would record an actor's performance and translate it onto a character rig. This would allow us to motion capture the speech and facial expressions of an actor separate from whoever was performing the body movement.

Here is the Faceware software which records the performance and shows the movement on the character rig. These sliders on the side actually allow drivers to push how sensitive the rig is. So if the eyebrows aren't as expressive as need be, the slider can either exaggerate or tone down the performance from the actor.

To create a mesh object of the head we used an app called PolyCam. The app requires you to take multiple photos of the head in a panoramic view which the software then used to recreate the face and head into a 3D mesh. The accuracy of the mesh relied heavily on the kinds of photos taken, so photos that showed a wider array of detail created a higher fidelity model. We did this several times with four of the students so we could get the four Metahumans that could then be driven by us in the mocap suits.

With the final meshes created in PolyCam they could then be imported to Unreal Engine where the mesh could be altered and programmed. This required a neutral facial expression as it would then track and solve for a facial mesh. The final would then be sent to Metahumans, who would produce a model in turn.

This is where we ran into a few problems with the meshes. First Natalie's mesh, unfortunately, recorded an unneutral face which caused her mesh's eyes and mouth to skew to the left. We also did not account for the hair of each person, resulting in wildly different head shapes! From Callum's curly hair adding large bumps to the mesh or Kalie's hair obscuring her neck which caused the mesh to have large gaps on her shoulders.

This required a lot of fixing when the models were then imported to Metahumans. Since a lot of the mesh had large and small bumps, we used the software of Metahumans that aligned the mesh more with the programmed default models in Metahumans than the mesh data provided by Unreal. This smoothed the rough areas and aligned the models more with the actual faces of the students.

With the facial models imported, all that was left was to dress and customise the models with hair and makeup to look like their real-life counterparts. Afterwards, the models were rigged in Motion Builder (MoBo) which allowed them to be controlled by an actor. Here is a look at the models that were fully rigged with facial expressions as well.

This week was a little different than what was scheduled, as Sally Coleman was in The Void this week for continued support for her own project of her completely virtual band Big Sand!

(this is the first music vid haha) This meant that we were sort of split up as a workshop as a lot of problem-solving was involved in helping Sally as well as more work with the mocap suits. 

Here is a much more full and detailed diagram of how the Vicon system interfaces with the computers. Now with the addition of Unreal Engine into the pipeline, it's much more clear how the mocap suits take live movement and translate it into live animation. By taking the tracking information from Vicon, the model rig is retargeted in Motion Builder which pairs the tracking markers to the 'bones' of the model. This is finally input into Unreal which will show the movement in real-time actors puppeteering the models.

Here is an image of the Unreal scene from Sally's music video. CDW Studios was kind enough to actually give the file to Sally so she's able to interact and pose her own character in the scene in her own time. The files are organised but 'dressed to camera' so it's not rendered in its entirety as a 3D scene, but it is still usable for filming. We have also gained access to the files so that we could experiment with the fully rendered environment if we so choose so. 

We also got to do some facial motion capture! This was filmed in a separate software called Faceware which is much simpler than Vicon, (facial recognition software is scary) but has similar controls to a rigged face. So the sensitivity of the facial movements can be exaggerated to better fine-tune them to a specific model. 

Here is both Unreal and Faceware open at the same time to see what the live performance looks like in Unreal with the actual model. It's good to have both programs open to see how changing the calibration of the Faceware software affects the animation on the model. This way it can be changed to better suit the performance in the suit as well as on the model. 

At the end of the workshop, we managed to watch and aid in the filming of a few takes of Sally's new music video. This was the first time that all the programs and components were working together and in sync, ready for recording.

This was the first workshop in the topic, so it was mainly a way for our instructors to explain The Void and the processes in which the Motion Capture system works. This was not my first experience in The Void, but it was the first time the experience was much more informative and detailed than in other workshops. Essentially, we began to learn how to set up the area for our own use later in the topic for the major project. 

For my own benefit I've written down the process from what I remember on how to set up the Vicon system:

Let the cameras warm up (the cameras need to all have the same environment for the system to track properly)

Connect the cameras in the Vicon system

Turn the masking feature on in order to mask out the infrared lights from the other cameras

Calibrate the cameras by using the wand to 'paint' the cameras (see. image above with Jason)

Set the world origin and rotation by placing the wand in the middle of the space

Set the floor of the world by adding tracking markers in a rectangle/square

I think at this point the systems would be ready to use to begin calibrating the mocap suits which should have the markers placed on them. 

Here was the diagram that was drawn to explain how the motion capture actually works. Beginning with the tracking data from the Vicon camera to then gets input into the computer and using the program Shogun it allows the movement to be recreated within 3D space as well as recorded. It was super interesting to see how the two interact as I had never seen the actual software before, and it felt like the suits that just 'magically' recreated the movement. 

Here other students are all in the suits ready for capturing. The process for getting the suits up and running was actually much more involved than I initially thought. The marker placements are really important as it correlates to the skeleton rig within Shogun so a lot of the work goes into making sure the skeletons are calibrated properly so that when actors make big movements the program recognises that a wrist is a wrist and isn't teleporting bones around the space. 

At this point the rigs were mostly calibrated for each of the performers, but it’s not entirely perfect! As seen in the second rig from the left there is a strange bend to the shoulders when the arms should be completely straight. This was due to time constraints where we were unable to properly spend time calibrating the suit. This shows that a lot of troubleshooting and onsite adjustments are needed for each person as the Shogun rigs are not specified to everyone's body type. 

The next and final step was to get the rigs working in Motion Builder to sort of affix the live performance in Shogun to Unreal Engine 5 so that we could see some robot rigs controlled by the performers. This process was entirely new to me as it required a complete overhaul of the robot rig in order to pair it to a specific performer. I was able to do this process for a couple of the rigs and it was super cool! It made me understand why a lot of these animations begin with a T-pose so that it’s easier to recalibrate! I hope to become more proficient in this as I am more familiar with the 3D side of this process.