#skeleton tracking

Intelligent Exercise System

by Nitu Sharma "Intelligent Exercise System"

Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-2 | Issue-6 , October 2018,

URL: http://www.ijtsrd.com/papers/ijtsrd18417.pdf

Direct Link: http://www.ijtsrd.com/computer-science/real-time-computing/18417/intelligent-exercise-system/nitu-sharma

ugc list of journals, best international journal, ugc approved journal

To Design and Develop Intelligent Exercise System

by Nitu Sharma | Nitin Janwe "To Design and Develop Intelligent Exercise System"

Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-2 | Issue-6 , October 2018,

URL: http://www.ijtsrd.com/papers/ijtsrd18387.pdf

Direct Link: http://www.ijtsrd.com/computer-science/real-time-computing/18387/to-design-and-develop-intelligent-exercise-system/nitu-sharma

ugc journal list, list of ugc approved journals, research publication

Since I now have some experience with these free libraries, let's document what my experience with the state of them is (as of November 2014). Hopefully this will help fill in the information that is generally lacking around the documentation and websites for these libraries.  Everything will require some finagling, so make sure that if you're going to incorporate skeleton tracking that you have plenty of time to deal with the errors and preferably have some resources available who can give you help.

Honestly for this project I should have used something like Unity3D that has stronger Kinect integration.

Microsoft Kinect libraries/drivers - this is your best option, assuming you're working on Windows.  I briefly used the SDK and it will work.  This can be used through Unity3D, the install will work on windows and everything is smooth

Easy install (on Windows)

Free as in free beer

Comes with skeleton tracking already

Not cross platform

Already incorporated into things like Unity3D

ofxKinectCommonBridge (lets you use the Microsoft libraries with OpenFrameworks wrapper)

OpenNI (NiTE) libraries are only half of what you need, and from the looks of it will only ever have half of what you need. OpenNI is only the drivers to actually get information from the device into a frame. NiTE was the library that actually provided skeleton tracking and the like. OpenNI is somewhat available for use through places like structure.io but NiTE is generally not available, and the ones that are available are at least a year out of date as of this post.

Relatively difficult install (build from source, does not build out-of-the-box, i.e. you must fix their source code to make it build for you)

Maintainers exist but updates are not too frequent

The remaining interest isn't in the skeleton tracking features, it's in the actual drivers for the device (which is good if you need that sort of thing)

Skeleton tracking does not work/does not exist any longer (you must write it yourself off of the frame data

Cross-platform** (**This is available on Windows but will not compile through MinGW without significant changes to the source. If working on Windows you must be using the Visual Studio compilers.)

This used to be the "standard" way of doing things for non-Windows platforms so a lot of tutorials will tell you to use OpenNI and NiTE, btu this does not work any longer.

Existing distributions of code may not work on newer hardware

OpenKinect/libfreenect with Igalia's SkelTrack seems to be the best solution available for Linux.  The tracking is noisy and somewhat primitive though.

Seems to be pure image-based with no learning model, not as solid of results as Microsoft's skeleton tracking (by quite a lot)

Also seems to only do upper body, but this may have been me failing

SkelTrack's joint tracking is extremely noisy and unfortunately I can't seem to get the asynchronous track joints function to work.  The function calls complete, but my callback function is never triggered.  Someday maybe I'll have time for it but for now I'll need to abandon it.

I was hoping that after I got OpenNI working it might give me something more than what freenect gives, but nope.  Oh well, I attempted it.  The NiTE libraries would be super useful, but they've all but vanished and it looks like OpenNI's usefulness for skeleton tracking is almost 0 at the moment.  There really doesn't seem to be a good incentive to choose it over say, OpenCV (especially since OpenCV has built in features for image processing, analysis, and vision algorithms).

What in heaven's name is FACE OSC, and why should I care? Its a Face Tracker program Kyle McDonald made as a "tool for prototyping face-based interaction. It's built on non-commercial open source FaceTracker code from Jason Saragih."

There is a "failure mode" of traditional algorithms to track people's faces via a camera and software. That's because its a lot like painting in that it tracks subtle lighting and color differences a la Rembrandt - except its mathematical. However, it needs tweaking. Bad lighting makes it go crazy and even the striations of grey hair throws it off.

One aspect of basic face algorithms are drawn from thousands of comparisons of faces in a huge database called Multi-Pie . Multi-Pie has thousands and thousands of pictures of faces that marks up and averages features and makes calculations about where everything is and in what type of lighting. Here are some typical mappings. In scientific parlance it is referred to as "Deformable model fitting."

Some basic face mapping

Its really how things are registered and abstracted. And when I am talking about abstract, this is a 3D model of a face that is used that would make Picasso proud. So was Picasso seeing in 3D Maya before it was fashionable? And that brings us back to African sculpture, where he originally got a lot of his Cubism. Hmm....

3D Computer Modeling of Face Recognition

19th Century African Fang Sculpture

But then it gets even more abstracted to a bunch of triangles in space.

Triangles in Space

This is called a "deformable model" probably because the triangles can be rearranged. This is all only 2D, with an "X" and "Y" height/width axis. When you add depth, or the forward backward "Z" parameter, then it become really complicated.

But what about areas of light and dark? That is when the Haar Algorithm comes in, but that has to be broken down into a histogram. A histogram is how your camera reads levels of dark and light in the image. Wikipedia calls a histogram " a graphical representation showing a visual impression of the distribution of data." 

It looks like a basic class in mask making with a lot of p(1n|x) formulas stuck in.  I think what this means is all the squares and locations are all tied in together eventually and are not independent from one another.

What happens if you mess the face up or only show parts of it? That has to be figured in as well.

Mapping for Obscured Parts

Now in a strange way all of this is related to skeleton tracking in 3D that the Kinect and other devices use.

It actually comes down to tracking and computing these blobs of dark and light.

The Face As Blobs Of Dark And Light

Chuck Close would be proud, since he was all over this years ago.

Chuck Close With His Own Version Of Face Mapping

Apparently Close suffers from Prosopagnosia a condition that messes up facial perception in humans. Hmmm... So African Art and messed up facial perceptions = computer algorithms.

FaceOSC comes as an example app with the ofxFaceTracker addon for openFrameworks. You can download an OSX 10.5+ binary at github.com/downloads/kylemcdonald/ofxFaceTracker/FaceOSC.zip OSCulator is an excellent program for routing OSC and MIDI. Download it at osculator.net/