Real-time Mixed Reality
Project from Alex Anpilogov uses a Kinect depth camera mounted display to view the world of someone in virtual reality without the need for green screen:
A live VR experience at exhibitions, large events, demos, etc is typically observed through first person camera or a third person camera on a TV screen. A more complex setup is sometimes used when the floor and walls feature projections of the VR world (similar principle to CAVE - http://goo.gl/i59uKF), or a green-screen setup with real-time compositing (along the lines of the process described meticulously by Kert Gartner - http://goo.gl/bWuHvu and Northways Games for streaming live - http://goo.gl/aJGxeA). The latter two methods require changes to the physical environment - adding projections or green-screen setup - often a costly or visually unattractive option. To provide a real-time mixed reality feed to an audience, retain the physical design of the space and avoid issues with foreground and background compositing, a RGB+Depth camera can be used to insert the user’s point cloud or mesh into the virtual space. Once the matrices of virtual and real cameras are matched, the virtual camera’s feed becomes a mixed reality feed.
… We built a quick prototype using one of HTC Vive’s controllers for camera tracking, Microsoft Kinect v2, a Kinect v2 plugin (https://goo.gl/K5B7nQ) and Unity running on 2x machines. The server ran the actual VR scene and the client extracted data from the Kinect, placed the point cloud into the scene, resulting in a the mixed reality feed. The depth threshold was altered dynamically based on the position of the headset.
More Here
