Usage of facial coding to detect emotional responses

Article

Facial coding: What we learn from facial expressions

Faces tell us a lot about the emotional state of target groups, stakeholders and co-workers

"The face is the window to the soul", as the renowned US psychologist and emotion researcher Paul Ekman once put it. A small raise of the eyebrow, an unintended smile or an angry look for a fraction of a second often says more about the reception of investment-heavy artwork than what people tell when they are asked how they feel about it. The method of facial coding relies on technical advancements to detect facial expressions that reveal our emotions. Technically it is based on a combination of image recognition, machine learning and deep learning algorithms where the self-learning algorithm for pattern recognition detects key facial characteristics such as the mouth, eyes and eyebrows. Changes in these features are used to classify distinct emotional states. This way small changes in facial expressions can be detected and interpreted.

At Deloitte Neuroscience Institute we use facial coding to automatically generate objective interpretations on emotional reactions, e.g. towards film sequences or during media consumption

Facial coding can be applied to various potential application areas such as:

  • Analysis and optimization of sales calls and customer service experiences (queues/routing)
  • Detection of unconscious emotions during media consumption


Facial coding can advance other research methods as EEG and eye tracking by

  • Allowing objective conclusions about emotional states
  • Improving visual stimuli and storylines based on emotional reactions

Did you know that there are seven universal facial expressions and even more micro expressions?

During his travels in New Guinea, Dr. Ekman encountered familiar facial expressions in an isolated culture that never had seen strangers before and was sealed off from outside influences. This sparked his interest so much that he further investigated expressions and eventually published his work on seven universal facial expressions. According to his research, anger, disgust, fear, surprise, happiness, sadness, and contempt are all shown with distinct facial expressions – no matter where in the world you are. These emotions are believed to be innate rather than learned through social interaction. Moreover, next to these “macro expressions”, Dr. Ekman has identified “micro expressions”. However, the detection and interpretation of these require constant training as they appear in a fraction of a second and occur often when people try to conceal their true emotions. The exact number of distinct emotions as well as the ability and the extent to which humans can detect micro emotions is a field of controversies and continued research.

With facial coding the Deloitte Neuroscience Institute can detect emotional responses to media and digital content, such as presentations, communications materials or any other digital stimuli. We usually complement these results with additional neuroscientific methods, such as EEG, eye tracking, implicit association testing and galvanic skin response measurement, to measure and visualize both, the mental and emotional response of participants.