#facetracker

Face Tracker Pro Snapchat Lens & Filter

Face Tracker Snapchat Lens & Filter

Face Casino

Playful automated facial casino experience that augments the viewer with crude new personalities that are positioned over and move with their actual face.

An antidote to facial recognition systems that now pervade our daily lives. This application pretends to partially know the viewer offering an interpretation of likes, stats and surreal poetry attached to the side of each new face.

Every time the viewer looks to the side a new facial combination is loaded. The app encourages a tinderesque swiping through the facial mash-ups in the pursuit of finding a match  - ie. getting comfortable with the “interpretation of the machine”.

Every configuration is automatically collected and added to an ever evolving piece.

I have a big interest in computer vision and am particularly curious about how machines interpret us with all the other data they hold on us. What do they know about us? How do we appear in their eyes?

Over the past 5 years we have seen facial augmentation mushroom with anything from cartoonesque overlays to facial enhancement or distortion apps. I wanted to create a distinct appearance that separated itself from common visual reference points but rather resembled an expression by an "unsophisticated" artificial intelligence.

One big inspiration here was the german dadaist collage artist Hannah Höch. Her provocative facial photomontages often a critique of partriarchial society feel very relevant to most of the political and societal issues we are facing at the moment. Another big inspiration was the work of the American multi-media artist Tony Oursler. In his series of works that are themed around privacy, surveillance and identity, absurd head installations invite the viewer to see themselves through the lense of a machine.

I was immediately drawn to the face controller homework we had as part of the OSC week. I wanted to create a gestural experience and Kyle Mc Donald's faceOSC add-on seemed a perfect fit to start with. After a short time experimenting with it though I realised that ofxFaceTracker was a better fit for what I was trying to achieve. I needed the camera feed and just wanted the overlaid components to move with the user's actual face so there was no need for OSC messaging from one app to another.

I started experimenting with the ofxFaceTracker add-on examples to evaluate what I needed for my experience to work. Temporarily I was hoping to create a 2 user interaction where the app takes a screenshot of the first user and then comps it onto the second user, but I found it hard to extract the raw data pairs for each face; something that should be technologically possible though, but was way beyond my technical skillset.

Within ofxFaceTracker addons folder I found the "example-cutout" which used the poly-line moving with the user's actual face to distort another image. I stripped out all the Delauney code which I didn't need and just used the poly-line to anchor the faceTracker  for my mapped facial components. Then I integrated the camera feed and gave it a solarised black and white effect to knock the camera feed back from the facial overlays.

Parallel to my tech experimentation I had a big task of sourcing images and cutting them out in photoshop and exporting these as pngs with transparent areas that allow for overlapping. I started with normal facial components but then moved on to exploring objects as well. I wanted the experience to be about the digital representations of our data as well as about our human reminiscent looks. During this process I found myself hopping between software: Illustrator to create quick compositions and working out the sizing of components, Photoshop to cut out the components and openFrameworks to feed my efforts into the machine. I enjoyed the surprising and absurd combinations I got back.

I created image vectors for every facial component that loaded in around 10-15 photographic images each. Then I randomised the way these images were displayed with a current object holding one random component of the vector at the time. Every time the facial tracking looses the user's facial data a new configuration is loaded for all the facial component vectors.

To allude to a desired future idea I created currently "fake" image classifier rectangles that reference the system analysing what it sees. I'm hoping to develop this further for our Popup show so that the system cycles through words in the same way that it currently runs through the image vectors.

Finally I integrated a screen shot facility so the system would take a picture of every creation that it compiles. I'm planning to create an ever evolving piece with that.

I think this piece would be really interesting if it performed sentiment analysis on the user's face and then supplied a live response by searching for images on the internet.  I'm also keen to implement machine learning to train the system on its own monster mash-ups. Maybe it could analyse these based on emotions that it interprets in their face? I like the absurdity and the creative promise this application holds. Over the summer I'm hoping to experiment with silicon prosthetics that the user could attach to their real face. These artificial components could then compute with the app. This would then juxtapose the physical with digital augmentation and add another layer to the blurring between human and machine.

I'm really happy with how phase one of this project completed. Of course, I could spend a lot more time on sourcing images and experimenting with what works and what doesn't, but I think it's at a perfect place now to get it user tested. From a code point I will turn the facial component vectors into classes to simplify the ofApp.cpp file, something I didn't have time to do. I also want to spend some time on the classifiers to deliver a more personalised experience.

pyFaceTracker is a Python wrapper for the OpenCV FaceTracker library. It can be downloaded at (https://pypi.python.org/pypi/pyFaceTracker/0.1.1) and the documentation for it is  (http://pythonhosted.org/pyFaceTracker/).

The installation details are pretty straightforward for pyFaceTracker.

To install pyfacetracker you will need the following prerequisites:

python (tested on version 2.73)

numpy

OpenCV library (tested on version 2.49 but should work on any version using the C++ interface)

C++ compiler (tested on MSVC 2008)

To run the examples you will need also:

matplotlib

Download the source files of pycompsense. Then, execute:

$ python setup.py install

You can test the installation by running the examples under the folder examples/

You can use YUM to install opencv and all its dependencies (http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html). However the pyFaceTracker wrapper setup script is designed to be run on a windows machine. It fails if you try to run it on linux. So lets try to update the setup.py script to get it working correctly.

Here’s what the final updated script looks like. The updated sections are highlighted. I’ll go through each of the changed sections afterwards and explain how you can find the values that your own system expects.

Get the updated for linux setup.py here.

from setuptools import setup from setuptools.extension import Extension import numpy as np import os

NAME = 'pyFaceTracker' PACKAGE_NAME = 'facetracker' VERSION = '0.1.1' DESCRIPTION = 'A python wrapper for the FaceTracker library by Jason Mora Saragih' LONG_DESCRIPTION = """ pyfacetracker is a thin wrapper around FaceTracker. It enables using FaceTracker while enjoyging the comfort of the Python scripting language. FaceTracker is a library for deformable face tracking written in C++ using OpenCV 2, authored by Jason Saragih and maintained by Kyle McDonald. """ AUTHOR = 'Amit Aides' EMAIL = '[email protected]' URL = 'https://bitbucket.org/amitibo/pyfacetracker' KEYWORDS = ["Face Tracking", "Image Processing", "Video Processing"] LICENSE = 'See separate LICENSE file' CLASSIFIERS = [ 'License :: Other/Proprietary License', 'Development Status :: 3 - Alpha', 'Topic :: Scientific/Engineering' ]

OPENCV_BASE = '/usr/include/opencv2' OPENCV_LIB_DIRS=['/usr/lib64'] OPENCV_VERSION = '.so.2.4.6'

FACETRACKER_BASE = 'external/FaceTracker'

OPENCV_INCLUDE_DIRS = [ OPENCV_BASE + p for p in ( '/core', '/imgproc', '/video', '/features2d', '/flann', '/calib3d', '/objdetect', '/legacy' ) ]

OPENCV_LIBS=[ l % OPENCV_VERSION for l in ( 'libopencv_core%s', 'libopencv_imgproc%s', 'libopencv_calib3d%s', 'libopencv_video%s', 'libopencv_features2d%s', 'libopencv_ml%s', 'libopencv_highgui%s', 'libopencv_objdetect%s', 'libopencv_contrib%s', 'libopencv_legacy%s' ) ]

def main(): setup( name=NAME, version=VERSION, description=DESCRIPTION, long_description=LONG_DESCRIPTION, author=AUTHOR, author_email=EMAIL, url=URL, keywords=KEYWORDS, classifiers=CLASSIFIERS, license=LICENSE, packages=[PACKAGE_NAME], ext_modules = [ Extension( PACKAGE_NAME + '.' + "_facetracker", [os.path.join(FACETRACKER_BASE, f) for f in ( "src/lib/PDM.cc", "src/lib/PAW.cc", "src/lib/Patch.cc", "src/lib/IO.cc", "src/lib/FDet.cc", "src/lib/FCheck.cc", "src/lib/CLM.cc", "src/lib/Tracker.cc" ) ] + ["src/_pyFaceTracker.cpp"], include_dirs=[os.path.join(FACETRACKER_BASE, 'include')] + OPENCV_INCLUDE_DIRS + [np.get_include()], libraries=OPENCV_LIBS, library_dirs=OPENCV_LIB_DIRS, #extra_compile_args=['/EHsc'] ) ] )

if __name__ == '__main__': main()

First we want to update the OPENCV_BASE variable to point correctly.

To find the location of our opencv files (this tutorial assumes you have successfully installed opencv) run the following command.

find / -name "*opencv*"

Note that we actually want to point to the opencv2 library base directory “/usr/include/opencv2″.

Next we need to update the directory where some opencv libraries are located. Go back to the list generated by the find command and look for the directory that contains all the library files similar to “libopencv_core” or “libopencv_ml” (don’t worry about version number). In my case it was “/usr/lib64″.

The OPENCV_VERSION variable is mainly just used to tack the version on to all the library filenames. Refer to the list generated by find to get the correct value.

Fix the FACETRACKER_BASE by using the forward slash instead of a backslash.

OPENCV_INCLUDE_DIRS point to module directories that reside within the opencv2 base directory (/usr/include/opencv2).

For OPENCV_LIBS note that you need to update these filenames as they are prefixed with “lib” in linux.

Note: If you get an error like “/bin/ld: cannot find -llibopencv_core.so.2.4.6″ that crashes the build, take the following steps to make sure the ld command is looking in the right place for your libraries.

Run the ld command in verbose mode to see where it is looking for your library file.

ld -llibopencv_core.so.2.4.6 --verbose

You will get output that includes something like this:

attempt to open /usr/x86_64-redhat-linux/lib64/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/x86_64-redhat-linux/lib64/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/local/lib64/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/local/lib64/liblibopencv_core.so.2.4.6.a failed attempt to open /lib64/liblibopencv_core.so.2.4.6.so failed attempt to open /lib64/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/lib64/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/lib64/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/x86_64-redhat-linux/lib/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/x86_64-redhat-linux/lib/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/lib64/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/lib64/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/local/lib/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/local/lib/liblibopencv_core.so.2.4.6.a failed attempt to open /lib/liblibopencv_core.so.2.4.6.so failed attempt to open /lib/liblibopencv_core.so.2.4.6.a failed attempt to open /usr/lib/liblibopencv_core.so.2.4.6.so failed attempt to open /usr/lib/liblibopencv_core.so.2.4.6.a failed

Notice that none of those paths actually match up with our library’s path which is actually located at /usr/lib64/libopencv_core.so.2.4.6 (remember you can use the find command we used above to find this).

So what we will do, is just grab the most similar looking “ld attempt” path and link it symbolically to point towards our actual file (I bolded the one I picked in the output above).

ln -s /usr/lib64/libopencv_core.so.2.4.6 /usr/lib64/liblibopencv_core.so.2.4.6.so

Note that you will have to do this with each library that ld fails to find (perhaps all 10 libraries but hopefully not).

Lastly comment out this line “extra_compile_args=['/EHsc']” as this flag is particular to windows compiler.

Go ahead and run the build

python setup.py install

Make sure you update the example scripts!

All of the example scripts use backslashes since Amit Aides the wrapper author wrote it for windows. Make sure you update all the example files with forward slashes to work in linux.

Thanks to Amit Aides for building this python wrapper for FaceTracker. Please don’t hesitate to shoot me an email if you are having any difficulty.

Lastly, here is  the example linux compatible face_image.py:

Google Faces by Onformative, 2013. Face detection searches for faces in Google Earth.

"Our Facetracker already circumnavigated the world a couple of times and astonished us with quite versatile results. As it continues to travel the world within the upcoming months, it continuously zooms into the earth. This process decreases the step-size for each iteration and therefore increases the amount of images and travel time exponentially. Some of the detected images aren’t usable at all, as we are not able to recognize any face-like patterns within the detected images. Other satellite images, on the other hand, inspired our imagination in a tremendous, yet funny way. However the search goes on, as our diligent robot continuous investigation."