Kitchen Hands

Some time ago I mentioned reading back data from the GPU asynchronously.  I’ve had done some exploration into this today, and I’d like to get some of that on paper.  I’m not going to go in depth through the whole process here as much as some of my other posts, but I think there’s still a lot to learn here.

Something I just want to quickly mention beforehand is that experimental API we we’re going to use is no longer experimental!  That’s great news!  We can now build the program against a stable Unity API.

Implementation

The asynchronous API works with requests.  This basically means that we file a “request” at some point in time (probably after we send data to the GPU), and at another point in the future we’ll get our data back.  The idea is that waiting for the data to come back doesn’t make the program wait, and so there’s effectively no framerate impact for getting the data back.

There’s 4 main places that we can make use of this requests API;

Ideally we’d merge all these into a single request, but that’s for another time.

The simplest way I can think of of switching these to an asynchronous process is to something like the following:

We simply make requests, and when they’re ready we feed the data back.  This does effectively the same thing we’re doing already without blocking the CPU.

Another way to make use of the requests API is to have the request tell us when it’s ready.  This would look something like the following:

Where the second parameter is a C# action that will be called when the request is complete.  This was my approach originally, but creating actions in a lambda-like fashion every frame actually incurs a fairly large overhead.  A solution here would be to avoid lambdas and create the action manually, but to be honest I hadn’t thought of that until now.  Both are good options.

Something else to note here is that it’s extremely important to use CopyTo() rather than ToArray() to move the values into our arrays.  In the words of someone on the Unity forums: “ToArray() allocates  like a motherf*er”.

Not so bad!  The desync you’re seeing here is that the texture is updated each frame, but the mesh is only updated every other frame or so.  In theory, we should be able to solve this by only updating the texture each time we push a new request, but despite my best efforts, Unity seems to refuse to chroma key the texture unless we update the texture each frame.

We’ve also successfully decoupled the hand’s bad performance from the rest of the game.  In builds, this solution comfortably runs at over 200FPS regardless of the hand parameters.  Note: this doesn’t mean the hand portion runs or even looks like it runs at that framerate.  It simply means that the rest of the game doesn’t suffer from the hand manager’s performance problems.

Concluding

Leap motion is an extremely impressive technology company which produces a product much similar to our own.  They seem to use a similar hand tracking approach to ours, with a specially designed camera strapped to the front of a VR headset, although I’d guess that they’re doing some kind of joint tracking as well rather than solely depth.

The demos are incredibly impressive.  There’s a great degree of freedom of movement, and from working on this project I can really appreciate the difficulties in getting something like what they’re doing (specifically the API/SDK end of things) to work smoothly.  It apparently even works on mobile devices!  This was so impressive to me actually, that I just had to do some more extended research on how they’re doing things.

Fortunately for curious minds, they have a blog post explaining the base level of how the leap motion controller works.  The most interesting point here is that they don’t generate a depth texture, but rather calculate points directly from stereo images.  From the looks of their patents, they seem to end up with something like the following after joining and then analysing the produced images (which are contrast enhanced and background reduced):

The basic idea is to reconstruct edges from edge pairs -- i.e. identify edges and join them with ellipses.  Simple (relatively) but very effective.

I can see how this could be quite possible, and very effective, after some good image pre-processing.  The good thing about a hand is that there’s a significant amount of data that is similar between everybody, and so the software can have a lot of points to look for and identify (such as 5 fingers, joints, and a palm).

This is significantly more advanced than our solution.  Through generating this 3D map and using image recognition to help “understand” the hand, they have much more freedom to do cool things such as track gestures (see above).  Our implementation has no idea what a hand is, and it doesn’t really care.

From what I can tell, they run the program at a significantly slower interval than we do, and perform a great amount of interpolation to achieve good performance in a high-motion environment.  This would definitely be a factor to consider in the future, and also gives me hope that the plan I mentioned here shouldn’t cause too many problems in terms of user responsiveness.

We do have at least a single one-up here though.  Because of the hardware and method we’re using (camera texture / depth texture), we can texture the player’s hands with their true to life details.  We’ve talked about it before, but this potentially performs the next step of taking the player’s mind out of reality and putting it in the game -- hopefully!  Due to their method and hardware, I also don’t see implementing something like this as trivial for the Leap Motion team, which also means that they’re unlikely to implement it.  What I mean here is that it’s up to us to build the technology to see how it performs.

If you’ve read my last post, you’ll see that we have quite an issue with getting our hand shader to work well with our green screen.  Namely, the chroma key doesn’t filter out colors very accurately or very well.  I’m going to discuss and explain some of this issue in this post, and hopefully by the end we’ll have a good result.

Why it doesn’t work

Before trying to solve any issues, we should first try to understand why the current implementation doesn’t work.  Here’s a simplified version of the current algorithm:

This basically compares each color component against the corresponding chroma component and checks that it’s within the threshold.  If the color is within the threshold for each component, then the color is considered close enough and a value is returned accordingly; in the example we return false because usually we want to discard colors that are similar to the chroma color.

Sounds logical enough on paper right?  Let’s walk through why it doesn’t work as intended most of the time.  Observe the following screenshot with chroma keying disabled:

I’ve also taken a palette of some main colors from the image with their values to make things a bit clearer;

Let’s see how our algorithm stacks up in comparing the hand and wall color to the green screen;

In this image, for a match to occur, all fields would have to be less than or equal to the threshold (100).  We got no matches with a pretty lenient threshold, which is exactly what we want.  However, take note that our green screen is a perfect green here, which is a supremely unlikely scenario in the real world.  Let’s try again with a closer-to-reality green, which I’ve taken from a real-life green screen setup:

Unintended match!  The hand color is also dangerously close to being matched here, and in reality we’d probably be matching other portions of the hand, resulting in a kind of skeleton effect;

A big part of the problem here is that the RGB color space isn’t really a good representation of human color perception.  Ignoring that fact for now, the second problem is that we’re not really putting much value into our consideration for each component.  Notice that in the Wall - Green Screen example, the difference values end up being quite similar, all within the 50-80 range, but the brightness/luminance (one of the most important factors in perceiving different colors) of each component actually varies significantly between them.

Originally I would have blamed the algorithm’s inaccuracy on just calculating the distance between two colors rather than the angle, but now I realise we’re not even doing that.  Lot’s of room for improvement here :).

Color similarity via angle

Another way to consider the problem might be to calculate what the angle is between colors.  If we take a look at a color wheel, it would seem to distinguish colors far better than the current method:

These color positions are not exact, because colors such as white aren’t displayed on the wheel.

The astute reader might have something cooking here.  There happens to be a convenient function for calculating the closeness of vector angles, which is of course the dot product.

(On normalised vectors) the dot product of two vectors will produce a value from -1 to 1 depending on how “similar” their angle is, where -1, 0, and 1 would relate to 180, 90, and 0 degrees respectively.  As shown above, vectors tend to work reasonably well in color spaces, so it seems like a good plan.

Our algorithm changes to the following:

Note that colors in HLSL are 0-1, so we don’t have to change our threshold.  In previous examples, we were only using a 0-255 threshold for the sake of familiarity.

At first glance, this seems to work excellently.  We’ve eliminated almost all of the green grain, similar colors group quite well, and we can turn up the threshold significantly without sacrificing the base colors;

Old method (top) vs. new dot product method (bottom).

If we turn the color to one of our “more realistic” greens, we do get some noticeable artifacting, but it at least doesn’t interfere with our flesh color.  Green screens are chosen because they’re the furthest from skin color, which we seem to be seeing in action here.

It’s actually also quite hard to chroma key out the hand, which I want to blame on the RGB color space, but suspect to be something I’m missing.

Something you might be wondering is that in the color wheel example above, each color looks very far away from each other, while we can still see some blending here with a low enough threshold.  This is because the color wheel is a 2D example, but our vector dot product is calculated in “3D”.  This isn’t a perfect analogy, but I like to think of it as the third dimension in the color wheel example would be the luminance / brightness, which is similar between all 3 colors.

Switching between chroma key colors.  If the gif doesn’t load, see this: https://gfycat.com/SleepyEachCanvasback

Further Considerations

On Monday morning we got working as a group on the integration stage of Kitchen Hands.  For the uninitiated, this means integrating the ZED camera hands with the Vive camera and the rest of the game scene.  This was actually our second integration day, but our first one was entirely taken up by getting past windows errors and other mishaps, so I won’t bore you with those details.

Things traveled much more smoothly than I expected.  We were actually able to get a working prototype running pretty quickly, and so had a lot of time to get some much needed experimentation going; figuring out some green screen details, lighting settings, and some physics interaction details.

Initial Implementation

As mentioned in one of my earlier posts, the ZED camera hands implementation has some finicky details in rendering to different displays (see the tail end of this post).  This issue really bugs me, but it hasn’t been pressing enough to warrant exploring a solution just yet -- this is just a prototype after all, and a “stuck together” solution is just fine.  I digress.  After fluffing around for a while, I was able to get the scene “working” just fine though.

Sorry for the lack of quality screenshots, it was a long day.

Findings

Something to notice is that the hand doesn’t look anything like it should.  Recall the original hand in my example was much more true to life:

The problem here is that the chromakey calculation used to discard the greenscreen pixels is quite poor.  As a consequence we had to fiddle a lot with the brightness and ended up sacrificing a lot of the fidelity in the process.  In fact, a black screen actually performed far better than the green screen, but I’ll talk more about fixing the green screen in a later blog post.

A further consequence of the greenscreen calculation that I hadn’t really thought about is that it’s really bad for performance when the calculation is wrong.  The problem is that the problem space (number of pixels / vertices) dramatically increases when we don’t cull the greenscreen.  All the more reason to get this right.

The final thing point of consideration was of course the performance.  The commonly accepted framerate for a VR application is over 90 FPS.  We’re not quite there yet, especially with the hardware of the test machine (GTX 1070).  It was a little disorientating running at ~30 (with low settings), and definitely not ready for long periods of use.  There’s always work to be done on this front, but to me this just emphasizes that it’s an important thing to work on.

In my last post I mentioned a plan for asynchronously transferring data from the GPU to the CPU.  I’m not going to begin work on that today, but I’d still like to highlight some thought process in relation to increasing performance.  Before I talk about anything in particular though, let’s set the scene with some code and some profiling data from last week:

The above code is run each frame from our ComputeMesh() function.  Think of it as a sort of initiation before calculating the mesh -- all buffers and information must be cleansed before creating anew.  You’ll note that the four SetData() calls take ~35ms of our 84ms total for this function -- almost half our time is spent arranging setting up data!

Why is a computationally inexpensive operation so costly?

It’s unlikely that the problem is throughput.  Although these arrays are sizeable (over 1.5 million elements with my test parameters), they still only occupy roughly 50MB of data.  Moreover, graphics memory bandwidth is typically very quick due to the requirement of frequently streaming textures.  Latency is a different story, however, and I think that’s what we’re experiencing here.

What can we do about it?

The best way to reduce a latency bottleneck is to make fewer latency-intensive calls.  We should ask “Do we absolutely need this many calls?” and “How can they be compounded?”.  The key here is reducing the number of calls above all else.

An introductory solution might be to reconsider how we’re transferring the data.  For example, we could combine all of our buffers into one so that we only eat a single latency penalty.  We can quite easily do better here though.  An important piece of information is that maxTriangleDivisions is a Unity parameter, and so does not change unless done so manually through the inspector.  As such, our array size does not change often -- certainly not every frame -- and so it seems likely that we should be able to just keep and edit these arrays on the GPU rather than sending new ones every frame.

Initialize Once

First off, we should take all this initialization code and put it somewhere it won’t be executed each frame.  Conveniently, there’s an aptly named InitialiseBuffers() function which can carry this burden -- it even already does some of the work that we’ve been doing!  There’s some code we do have to leave behind (namely setting the Zed Camera textures), but it turns out to be relatively insignificant.

Reset Always

Next we need to modify the compute shader to reset variables on its own.  This turns out to be a quite complex if you don’t have a good understanding of how it processes the given data.  I personally spent an embarrassingly long time realizing this fact.  Take a look at the following compute shader (HLSL) code:

I assume you have a rudimentary understanding of compute shaders here.  For each triangle (meshes work primarily in triangles) we sample the color at each point (from the Zed Camera texture) and either apply depth to the triangle, or divide it for a finer grained calculation.  In the original solution, the triangles array is populated with -1 each frame, and when we touch or alter a triangle it will be marked with something different.  So then what we need is some way to reset values in the triangles array to -1 when they haven’t been touched.  Understanding how to do this properly required stepping into how the triangle division is actually performed, which can be quite confusing;

Ahh!  Stuff I don’t want to understand!  The worst part here was figuring out that each division calculation (if ... else) block corresponds to 3 possible depth applications, and each ApplyDepth() call actually increments a global counter so that we can proceed to the next point of each triangle.  All this ties back to the magic “12″ number in the main function, which you’ll see has been since documented.

This magic caused the problem to be much more stressful than it needed to be.  My first attempts were to check if depth had been applied to the pixel, and if it hadn’t, to “reset” that triangle in the array.  Here’s a look into what I was scratching my head about for a few hours:

It’s there, but it’s wrong.  Clearly some triangles aren’t getting cleared.  A lesson I’ve learned here is that in a codebase you’re not properly familiar with it’s really easy to think “I’ve got the right idea, there’s just a small tweak I must be missing!” and try to rearrange what you’ve done for a long time.  This is what was happening to me.  The logic is sound -- if a triangle hasn’t had depth applied to it, then it should be enough to clear it, right?  But sometimes you really need to take a hard look and understand the intricacies of the software, and the intricacy here was understanding the relationship between individual triangles, the magic number 12, and the global increment.  I wish I could go into the full details of everything here, but there’s really a whole book in my head of my learning process today, and I can’t explain it all without explaining exactly how the software works first.  Another day, perhaps.

Like I hinted earlier, the solution here is embarrassingly simple once you’ve got a grasp on the magic;

Resolving Functional Differences

You might notice that we’ve lost some of the original functionality here.  Namely, we can no longer change the “Max Triangle Divisions” parameter at run time.   The resolution to this is pretty simple, we should simply check if the parameter has been changed each frame, and if it has, we should force our arrays to reinitialize; in essence, do the same thing we we’re previously doing every frame, except on demand.  This post is already pretty lengthy, so I won’t bore you with the exact details here.

Results

Although the labor may have been more difficult than expected, the results don’t disappoint.  We’ve gone from an 11FPS average in builds (with benchmark settings) to a 29FPS average.  That’s almost a 3x speedup!  And we can thank the elimination of CPU -> GPU latency for most of our gains.

The astute reader might have some questions here.  As I mentioned at the beginning of the post, our SetData() calls only take up ~35ms of our 84ms frame total -- not enough for more than a 2x speedup, surely.  The reason for this is pretty simple; we’ve removed more code than what was in the profiler sample, and so we’ve cut more time than in the profiler sample.

I’ve been working hard on further profiling and pinpointing our performance problems.

First of all, I want to note that using different parameters for our hand mesh generator significantly affects performance, and so also affects the profiling data -- it’s quite easy to go from more than 150 frames per second to less than 1 just by moving some sliders.  I’ve decided to settle on something that represents a reasonable user experience (besides the frame-rate), and also gives us a good level of performance to improve on.  I’ll be sticking to these settings as much as possible throughout the rest of the project.

New Data

By default, Unity only provides profiling data for build in methods such as Update(), which, as I mentioned in a previous post, can be an incredibly vague range.  I’ve been using Unity’s Profiler.BeginSample() method to more precisely analyse our performance, and have come across some interesting information that seems to contradict some of my previous assumptions;

A majority (~70%) of CPU time is spent not actually doing any computation but rather sending and receiving data from the GPU.

Our script reads and writes data every frame, which is probably most of the reasoning behind this.  There’s also virtually no work between sending the data off to the GPU and asking for it back, which is a worst case scenario in regards to performance.

Why Wait?

This got me thinking: why wait?  We should be streaming data to and from the GPU and separating concerns.  Have data sent to the GPU when convenient (ideally in a non-blocking way) and read it asynchronously back to the CPU whenever the GPU is ready.

You might be reasoning that this kind of operation incurs an input delay.  Yes and no;

We still receive the most recent GPU work each frame.  These frames might arrive slightly later however, since the work is no longer synchronised.  Hopefully the diagram shows this well enough.

It also means we can focus much more on “real” optimisation.

Experimental

If you’ve been following my other posts, you’ll know that there’s a fairly severe memory leak in the hand mesh generator.  Over the weekend I decided to take a look at the issue.

When resolving a memory leak you’ll most often want to start by profiling the application.  This will give you access to important information such as where memory is allocated, how much, etc.  Unity has a fairly in-depth profiler built-in, so that’s what we’ll be using here.  For memory information specifically, there does exist a tool that is much more thorough developed by Unity Technologies here, although I’m not sure how supported it is anymore.

Here’s the profiling windows we see after about a minute of runtime:

The first thing that we can note is that most of our allocations are happening within ZedCameraHandShaderManager.Update(), which isn’t much information to go on since there’s a significant amount of work happening within this function.  In fact, Update() covers almost the entirety of the program.  There is another piece of information that can help decipher this mystery though.  In the second image, notice how the amount of meshes in the scene is the only thing increasing with memory consumption, so it must be the case that our memory leak is mesh related.  There’s only one function in Update() which performs mesh operations every frame, and that’s ComputeMesh().  Sure enough, here we find our culprit:

Somewhat counter intuitively, Unity meshes are explicitly handled objects, and so each mesh must be manually destroyed.  In the code here, it seems as though the original author (rightfully) assumed that meshes were handled by the Unity garbage collector.  This is a pretty basic fix once we get to the source:

And for good etiquette, we should apply the same logic to the program’s OnDestroy() function:

The research for this project is mainly focused on the technology behind the implementation.  For me, this largely means compute shaders and ZED camera technical details.

Compute Shaders In Unity3D

I’ve worked with compute shaders before, programming in pure CUDA.  Compute shaders in Unity are a slightly different beast, and it took me a while to get up to speed.  I found Emerson Shaffer’s explanation to be extremely helpful from the perspective of somebody who already understood the core mechanics, but not exactly how it worked within Unity.

ZED Camera Details and Improvements

I also invested some time into researching the ZED cameras capabilities and how it works within Unity — including figuring out ways to improve upon the current program design.  There’s an interesting documentation on the ZED camera’s advanced settings that seems to detail some crude real-world object outlining.  This could be useful in potentially developing the solution towards working without needing a green screen, however I do have some doubts due to the lack of accuracy — this is a mode designed for speed applications, not precise ones; to base an entire feature reconstruction (green screen elimination) off of this technology would be a mistake.

The first step in solving a problem is understanding what the problem actually is.  Just recently our team got access to the hand mesh generator code base, and I thought I’d get started early and dive into understanding what sort of problems we were facing.

After getting through the typical Unity version change issues, I was able to get things running pretty reasonably with the Zed Camera.  I set my desktop background to a nice lime green as a makeshift green screen and fiddled with the available parameters.  It was really exciting how well the whole system worked; follow this link to see for yourself.

Next on my agenda was digging for some holes to fill in the solution before integration.  Even considering final integration, finding and solving problems in the existing solution is perhaps the challenge of this project — a perfect integration still has great potential to be a buggy, stuttery, terrible experience.

I built the program with a simple fps counter (noting any nuances or complications) and ran it under Unity’s profiler.  Promptly, here’s our first user experience hole:

Snail’s Pace

With a blank scene and only the hand mesh and its collider being generated — no other game objects; best case scenario — we’re running at a flat broke 25 frames per second.  Furthermore, this is also on  what should be considered a powerful machine (R7 1700, 1080 Ti).  With a VR headset connected and an actual game environment, I’d be surprised to see the solution in its current state push past single digits.

With any optimization problem its important to first of all thoroughly understand the composition of the problem code.  In preparation of significantly optimizing the codebase in the weeks to come, I’ve profiled the code extensively to make a heat map of sorts.  This doubled up as an initial foray into understanding the codebase.

For important functions I’ve color coded them with varying degrees of red depending on their CPU execution time.  This gives me a much more visual representation of the project architecture in regards to performance, and over time I’ll maintain this graph for precisely that benefit.

I could spend a whole blog post on my findings here, but let’s leave this alone until there’s something truly interesting to talk about.

While I was collecting this profiling data, I left the program running in the background.  After a short while (20 minutes or so) I was alarmed to come across our second hole:

A Memory Leak

Almost 11 gigabytes of data in the span of 20 minutes: not good.  A keen eye might also notice that the program also seems to be swapping out program memory to the disk as damage control.  On a fast SSD with not much going on this isn’t a problem, but older hard drives swapping can be absolutely devastating.  This post is specifically about spotting issues, not solving them, so I’ll save the details for another post.

A Build Dilemma

The final hole, and this is mostly one of integration, is that it can be quite a struggle to piece together a successful build that works as intended.  The project uses two cameras in the same scene, and Unity doesn’t seem to like that.  As a quick workaround for an up-and-running build, you can make the render camera point to a different (virtual) display, but this raises some other concerns.

I have two monitors, with the second one rotated 90 degrees.  If I ask Unity to render the second camera to display 2, it renders the mesh in a rotated-by-90-degrees space in builds.  For the time being I’ve set it to render in display 3 instead, which seems normal, however this isn’t ideal for a couple of reasons:

If the user has 3 monitors, it will potentially mess up depending on configuration.

Due to the rotation issue, I have concerns about resolution and scaling settings when rendering to different displays.  Ideally I’d prefer to make the solution work without involving multiple displays.

The Road Ahead

It’s hard to think about a timeline for this project.  First of all, we don’t really know any specifics of what we should do yet.  Our mentor has described to us how the project is segmented, and what the tasks are, but none of the completed components that we will be integrating (mentioned earlier) have been given to us yet.

That being said, there are a few conclusions that can logically be made, and then sorted into a rough timeline;

1. Initially, I’ll be making sense of code that has already been written, and getting comfortable with the hardware -- if you’ve read my other posts, you’ll know I’ve started on the latter of these already. 2. Next, I’ll be optimizing and improving upon the aforementioned code. 3. Finally, I’ll be meeting up with the team at the end of semester to integrate my completed system with the rest of the project.  For clarity, my part will be using the depth camera to draw reliable hands, and their parts will be VR and AI.

If done successfully, this project has the potential to make clear movements in many fields; obviously, using your real hands in any VR simulation would be much more immersive and realistic than using a controller -- provided they work as intended.

However, as we’re all heavily aware, it’s unlikely that we’re going to make leaps and bounds in usability within the time space allocated for this project.  The proposal here is then actually for us to test the feasability of the project as a whole; Is depth testing enough to build a suitably accurate hand map for use in a real game?  Are approximations enough, or does the technology need to be exact?  What about latency?  How much more work will be needed at the end of our time with the project, and what is this work?

Success or fail (in regards to intended user experience), a tried idea is an explored idea.  Using a depth camera to create rendered hand meshes is a refreshing idea that hasn’t been explored all too much.  Should the project be a success (within this semester, or later down the line) it could bring entirely new experiences to all sorts of leading industries -- from employee training to facilitating disabilities, the impacts here are clear.  Even the gaming industry could benefit greatly from increased immersion.  The point I’m trying to stick here is that increased immersion is intensely sought after -- it’s the foundation of virtual reality!  The ultimate goal of being able to really experience our prospects virtually only draws closer with this technology.

Are controllers part of the VR future?

After meeting our mentor on Wednesday, it seemed clear that I’d be processing the depth camera portion of integration.  This means exploring a totally new area of hardware for me, so when I got home I quickly looked up the ZED Mini on ZED’s website and got myself up to speed on what the camera actually does and how it works together with Unity3D.

I had some fun running through all the ZED demos, and even managed to create a pretty accurate 3D spatial mapping in my bedroom:

https://imgur.com/zeKymnk

(Tumblr doesn’t allow big gifs).

The idea of this Kitchen Hands is to allow the use of real hands as a replacement for controllers in a VR space.  The theory behind this is simply that each interface between the player and the game inhibits immersion; indeed -- holding a controller is significantly different and more complicated than true to life input.

A secondary objective of this project is to also use Vive trackers in conjunction with real-world objects to achieve real haptic feedback when interacting with objects in the game.

Ideally, with these goals accomplished, you’ll have a more immersive, haptic interface with a sense of your own hands in the virtual world.

(OC)

To accomplish the hand portion of this project, we’ll be utilizing depth camera video input and strapping it to a VR headset.  There are some compelling challenges involved with the implementation of this task, which you’ll hear about on this blog! (Watch this space.)