The future of facial expression analysis

Facial expressions

The human face is a vast, ever-changing window into our innermost state of mind, with an enormous range of nuances and great complexity.

Humans have always developed, quite naturally, the ability to read other people’s facial expressions, first as a survival skill (friend or foe?) and today for relationships of every kind.

Relying on nothing but intuition and sensitivity, we all spend a good part of our time reading, interpreting and reacting to the facial cues of bosses, spouses, teachers, partners, coworkers, strangers, relatives and so on.

With technology and ever-growing computing power, our faces, and the information they give away without our noticing, are no longer just how we present ourselves, relate to others or behave in public.

Far from simply signaling our mood of the moment or how we react to situations, people or events, all this data is increasingly treated as strategic and studied intensely.

The use of facial expression analysis keeps growing: Apple uses it to unlock the iPhone, churches in the United States use it to attract worshippers, stores in England use it to identify shoplifters, police in Wales use it to arrest suspects at soccer matches, and China uses it to spot reckless drivers and give tourists access to certain attractions, among other examples.

In medicine, some applications are very promising, such as the early diagnosis of genetic conditions like Hajdu-Cheney syndrome, hopeful efforts to treat autism, detecting whether someone is depressed or even whether pain is physical or psychological.

All these developments have sparked heated debate over whether they invade privacy, since, for example, an algorithm identified sexual orientation with 81% accuracy, while people (without the tool) got it right 61% of the time.

That creates a real risk of discrimination by companies, in hiring for example, so much so that lawmakers in some European countries are already moving to treat biometric data as belonging to the individual rather than being in the public domain.

But getting back to the applications that matter to us, the fact is that facial expressions usually communicate more, and better, than words, because they’re spontaneous, sincere and natural, because they reflect real feelings or emotions, and above all because they aren’t rational.

That makes facial expression analysis an important tool to use alongside other neuroscience and market research techniques, to understand what people don’t know, can’t say or perhaps don’t want to put into words.

Paul Ekman pioneered facial expression analysis with his research back in the 1960s. One of his biggest contributions was showing that there are at least 6 human emotions that can be identified regardless of gender, age or even culture.

His work decoded these emotions into distinct combinations of 46 facial movements, with uses ranging from animation to lie detection.

As technology has advanced and computing power has grown, the uses of this knowledge have expanded considerably. In essence, the algorithm measures how a series of points mapped virtually onto the participant’s face move relative to other fixed points.

The position of the mouth, lips, cheeks, eyebrows, eyelids, wrinkles and so on is constantly compared with other reference points (such as the tip of the nose or chin), with each person’s own baseline expression and with a reference table, making it possible to identify emotions such as joy, confusion, frustration, surprise, fear, anger and disgust.

In studies Checon has already conducted, we can assess how much people like (or dislike) a commercial, how positive (or not) their engagement is and which parts of the commercial need adjusting or reinforcing, all without asking participants a single question, and always in a clear, objective way that even people outside neuroscience or research can understand.

The same kinds of conclusions are possible when assessing reactions to a political speech: whether people understand it or are engaged, which passages need to be clearer, and whether there’s approval or acceptance, always non-invasively and without even interacting directly with participants.

In website usability studies, difficulty with certain tasks shows up clearly in negative facial expressions or signs of anger, doubt, dissatisfaction or frustration, even when participants don’t admit it in their “rational” account.

Another very interesting use is analyzing shoppers’ reactions to store windows, to measure their impact, the strengths and weaknesses of each option and even how well they draw people into the store.

Not to mention food companies analyzing facial expressions in taste tests or blind preference tests, banks gauging reactions to ATMs, or carmakers studying driver behavior and detecting signs of drowsiness at the wheel.

Microexpressions can also tell us a great deal about decision-making, predicting whether a purchase will actually happen (a flash of disgust can give it away) or whether a price is right (an expression of joy can confirm it, one of anger can contradict it).

Since participants usually don’t even notice these microexpressions themselves, because they’re spontaneous, they’re more reliable predictors than the (often) politically correct answers that undermine so many sales forecasts.

Choosing a candidate and voting intention also lend themselves perfectly to this kind of analysis, which appears more reliable than voters’ rational statements alone.

There’s striking data on how accurately American voters’ intentions in the presidential election were predicted based solely on facial expressions recorded during the Obama & Romney debate, with 73% confirmed, without asking a single question.