#gesturerecognition

The Automotive Gesture Recognition System Market Research Report 2024 begins with an overview of the market and offers throughout development. It presents a comprehensive analysis of all the regional and major player segments that gives closer insights upon present market conditions and future market opportunities along with drivers, trending segments, consumer behaviour, pricing factors and market performance and estimation and prices as well as global predominant vendor’s information. The forecast market information, SWOT analysis, Automotive Gesture Recognition System Market scenario, and feasibility study are the vital aspects analysed in this report.

The global automotive gesture recognition system market is expected to grow at a 17% CAGR from 2024 to 2030. It is expected to reach above USD 990.4 million by 2030 from USD 10,326.46 million in 2023.

Access Full Report:

Qualcomm has announced its second-generation mid-range mobile chip. The Snapdragon 7+ Gen 2 processor is built on the 4nm process. It’ll be available on Android smartphones starting this month. Read more…

How can we train a machine to recognise the difference between ‘something’ and ‘nothing’? Over the past few months, I have been working with Despina Papadopoulos on an R&D wearable project – Embodied Companionship, funded by Human Data Interaction. 

“Embodied Companionship seeks to create a discursive relationship between machine learning and humans, centered around nuance, curiosity and second order feedback loops. Using machine learning to not only train and “learn” the wearers behaviour but create a symbiotic relationship with a technological artifact that rests on a mutual progression of understanding, the project aims to embody and make legible the process and shed some light on the black box.” – Text by Despina Papadopoulos 

The project builds on the work we did last year in collaboration with Bless, where we created a prototype of a new form of wearable companion – Stylefree, a scarf that becomes ‘alive’ the more the wearer interacts with it. In Embodied Companionship, we wanted to further explore the theoretical, physical and cybernetic relationship between technology, the wearable (medium), and its wearer. 

*Stylefree – a collaboration between Despina Papadopoulos, Umbrellium and Bless.  

In this blog post, I wanted to share some of the interesting challenges I faced through experimentations using machine learning algorithms and wearable microcontrollers to recognise our body movements and gestures. There will be more questions raised than answers in this post as this is a work in progress, but I am hoping to share more insights at the end of the project.   

My research focuses on the use of the latest open-source machine learning library; Tensorflow Lite developed for Arduino Nano Ble Sense 33. Having designed, fabricated and programmed many wearable projects over the years (e.g Pollution Explorers – explore air quality with communities using wearables and machine learning algorithms), large scale performances (e.g SUPERGESTURES – each audience wore a gesture-sensing wearable to listen to geolocated audio stories and perform gestures created by young people) and platforms (e.g WearON – a platform for designers to quickly prototype connected IoT wearables), the board is a step up from any previous wearable-friendly controllers I have used. It contains many useful body-related sensors such as 9 axis inertial sensors, microphone, and a few other environmental sensors such as light and humidity sensors. With the type of sensors embedded, it becomes much easier to create smaller size wearables that can better sense the user’s position, movement and body gestures depending on where the board is placed on the body. And with its TinyML which allows the running of Edge Computing applications (AI), we can start to (finally!) play with more advanced gesture recognition. For the purpose of our project, the board is positioned on the arm of the wearer. 

*Image of the prototype wearable of Embodied Companionship 

Training a Machine

With the constraints, I started exploring a couple of fundamental questions – How does a machine understand a body gesture or a movement? How does it tell (or how can we tell it to tell…) one gesture apart from another? With any machine learning project, we require training data, it is used to provide examples of data patterns that correspond to user-defined categories of those patterns so that in future the machine can compare streams of data that are being captured to the examples and try to match them. However the algorithm doesn't simply match them, it returns a confidence level that the captured stream of data matches any particular pattern. Tensorflow offers a very good basic tutorial on gesture recognition using the arduino board, however, it is based on recognising simple and big gestures (e.g arm flexing and punching) which are easily recognisable. In order for the machine to learn a wearer’s gestural behaviour, it will involve learning many different types of movement patterns that a person might perform with their arm. So our first task is to check whether we can use this arduino and Tensorflow lite to recognise more than 2 types of gestures. 

I started with adjusting various parameters of the machine learning code, for e.g, training more than 2 sets of distinct gestures, training with more subtle gestures, increasing the training data set for each gesture, increasing the epochs. The results were not satisfactory, the board could not recognise any of the gestures with high confidence mainly because each gestural data was not distinct enough for the machine to distinguish and hence it spreads its confidence level to the few gestures that it was taught with. It also highlighted a key question for me, i.e. how would a machine ‘know’ when a gesture is happening and when it is not happening? Without having an explicit button press to signify the start and end of a gesture (which is synonymous to the Alexa or Siri wake-up call), I realised that it would also need to recognise when a gesture was not happening. 

*How a gesture/movement is read on the serial plotter through its 3-axis accelerometer data

The original code from the tutorial was based on detecting a gesture the moment a significant motion is detected which could be a problem if we are trying to recognise more subtle gestures such as a slow hand waving or lifting the arm up slowly. I started experimenting with a couple of other ways for the arduino board to recognise a gesture at the ‘right’ time. First, I programmed a button where the wearer presses it to instruct the board to start recognising the gesture while it's being performed – this is not ideal as the wearer will have to consciously instruct the wearable whenever he/she is performing a gesture, but it allows me to understand what constitute a ‘right’ starting time to recognise a gesture. Lastly I tried programming the board to capture buckets of data at multiple short milliseconds time instances and run multiple analysis at once to compare each bucket and determine which bucket’s gesture at any instance returns the highest confidence level. However that does not return any significantly better result, it’s memory intensive for the board and reinforces the challenge, i.e. the machine needs to know when a person is not performing any gesture. 

*Capturing buckets of data at multiple short milliseconds time 

While the arduino board might be good at distinguishing between 2 gestures, if you perform a 3rd gesture that is untrained for the board, it will return either one of the learnt gestures with very low confidence level. This is because it was not taught with examples of other gestures. However, if we want it to learn a wearer’s behaviour over time, not only do we need to teach the machine with a set of gestures just like any language that comes with a strict set of components e.g alphabets, but it is equally important to teach it to recognise when the wearer is not doing anything significant. And with that it poses a major challenge, i.e. how much training do we need to teach a machine when the wearer is doing nothing? 

Making Sense of the Nuances 

When it comes to making sense of body gestures, our recognition of any gesture is guided by our background, culture, history, experience and interaction with each other. It is something that in this day and age, an advanced machine is still incapable of doing, e.g recognising different skin colours. Therefore, as much as we can train a machine to learn a body gesture through its x,y, z coordinates, or its speed of movement, we cannot train it with the cultural knowledge, experience or teach it to detect the subtle nuances of the meaning of a gesture (e.g the difference between crossing your arm when you are tired vs when you are feeling defensive). 

Photo of one of the SUPERGESTURES workshops where young people design body gestures that can be detected by the wearable on their arm, and represent their story and vision of Manchester 

It is worthwhile to remember that while this R&D project explores the extent to which machine learning can help create a discursive interaction between the wearer and the machine, there are limitations to the capability of a machine and it is important for us as designers and developers to help define a set of parameters that ensure that the machine can understand the nuances in order to create interaction that is meaningful for people of all backgrounds and colours.   

While machine learning in other familiar fields such as camera vision do have some form of recognising “nothing” (e.g background subtraction), the concept of recognising “nothing” gestures (e.g should walking and standing up be considered ‘nothing’?) for wearable or body-based work is fairly new and has not been widely explored. A purely technological approach might say that ‘nothing’ simply requires adequate error-detection or filtering. But I would argue that the complexity of deciding what constitutes ‘nothing’ and the widely varying concept of what kinds of movement should be ‘ignored’ during training are absolutely vital to consider if we want to develop a wearable device that is trained for and useful for unique and different people. As this is work in progress, I will be experimenting more with this to gather more insights. 

(via https://www.youtube.com/watch?v=0QNiZfSsPc0)