Emotions are tricky. Surprisingly, they can be confusing not only for robots but for humans as well. Sometimes relying on the context can help, though even that can be manipulated.
We decided to prove that emotion perception — that is the emotions we think our interlocutors or just people in sight express — is sensitive to context and we will perceive the same facial expression differently depending on the context. For this, we organized a local contest for students and young specialists. The contest was held jointly with Think Cognitive Think Science, the project that supports cognitive science initiatives among the student community. We asked them to prove the thesis and create an ‘emotional illusion’ either as an image or a video. The contestants had to send a sketch or a draft video that would capture how the emotional expressions we perceive in those change according to how the context changes.
We accepted the works on one important condition: the illusion must be clear and simple. The idea could be expressed via any channel (face, voice, gestures, text). Perfectly, several of them at the same time. For example, facial expression and a changing tone of voice, or textual cues combined with the gaze. However, this approach was optional.
In order to determine the winners, we analyzed the illusion taking into consideration several parameters, such as the novelty of the idea, its quality, simplicity, stability of the illusionistic effect. We received lots of exciting works but here are the most brightest contestants who had caught our attention.
Anastasia Shveytser, 1st place in the ’Photo Illusion’ category
Anastasia is a sophomore at the Russian Presidential Academy of National Economy and Public Administration who is studying for a British Bachelor’s Degree in Psychological Counselling and Coaching. She sent us two sketches of a girl with either surprised or scared expression on her face (raised eyebrows, tense eyelids, open mouth). The first image depicted the emotion while the second one revealed that in fact, she was dying her eyelashes with mascara.
The idea came out of the blue. Anastasia had almost decided not to take part in the competition as no good concept came to her mind until she saw her sister doing her makeup and the pieces came together. The expression itself seemed inappropriate to the context and the only thing that distinguished the image was a mascara brush. Without it, the image would just depict a scared or surprised face. She tested the concept on her friends to confirm her hypothesis and everything worked well.
What’s going on here? An open mouth and wide open eyes are usually associated with the expression of surprise or fear. However, in this image it may as well mean that the person might be currently speaking or, as it turned out, doing her makeup. But how can we be sure that this facial «setup» reflect the emotion we expect?
The idea that emotions can be understood by the universal expression has been around for some time. It has been mentioned by Charles Darwin in his 1872 book, The Expression of the Emotions in Man and Animals: “facial expressions of emotion are universal, not learned differently in each culture.’’ Some scholars, most famously Paul Ekman, even tried to classify human emotions and analyze how they affect our facial features.
There have been arguments both in favor and against ever since. It has been argued that people are generally relatively skilled at telling another person’s mood, simply by taking a glance at them. However, today scientific community disagrees with this approach believing it’s out of date. You can read more on that and other myths in Affective Computing in our article.
Nikita Terekhov, 1st place in the ‘Video Illusion’ category
Nikita Terekhov is a young specialist from the Russian Academy of National Economy and Public Administration majoring in Management who won in the category of Video illusion. His idea was to show that human perception is quite subjective and determined by particular preliminary settings that create an expectation. The settings are usually born out of context and our knowledge.
He used this way of thinking throughout all of the parts of the experiment. Having researched psychological experiments and perception tricks, Nikita used the Kuleshov Effect. Along with the illusions themselves, he sent the instruction to the videos which created an expectation.
Illusion #1. There are three scenes, in each a man that is shown with a stimulus. The viewer should identify the emotion the man expresses. Depending on the stimulus the man’s face seems either sad, thoughtful or angered while in fact, it is the same facial expression in all instances.
Credit: Nikita Terekhov.
Illusion #2. Based on the same principle but the montage is different. The stimulus was shown before the scene of the man.
Credit: Nikita Terekhov.
Illusion #3. Two pictures of a man with different or the same facial expression are shown in close proximity to each other. The series of flickering contrasting images are meant to enhance the effect of expectation. The subject is supposed to identify on which of them the depicted person will be smiling. Nikita tested the idea on the subjects and nobody could identify the smile or make 100% claim that the last two images were neutral. As he said, this example reflects the subjectivity of perception which our winner believes can be observed in everyday life quite often.
Credit: Nikita Terekhov.
What’s going on here? The idea of the first two illusions is rooted in the Kuleshov effect. It was first used in video editing by Soviet filmmaker Lev Kuleshov in between the 1910s and 1920s. He realized that viewers perceive more meaning from the interaction of two sequential shots than from a single shot in isolation. This technique was quite popular in cinema and especially loved by Alfred Hitchcock who claimed it was ‘’pure cinematics’’.
Alfred Hitchcock explaining the Kuleshov effect.
Though she did not win, we thought Olesya’s work was one of the most exciting. She is a senior from the bachelor’s degree in International Relations at the Russian State Universities for the Humanities. Last year she took an interest in Cognitive Sciences and decided to pursue her academic career in this field.
Olesya noted that several studies aimed to show that it is almost impossible to recognize emotions without context (Aviezeretal, 2008; Calbi & Heinmann, 2017; Feldman Barrett, 2011). She gave an example famously used by Lisa Feldman Barrett — that of Serena Williams whose face might seem angry when zoomed-in but in the context of her long-awaited victory the expression is obviously happy.
To conduct her experiment, Olesya recruited two groups of people and gave them pictures with objects that might affect the cognitive perception of the picture of a female with an open mouth. She supposed if a different context is given, it might be interpreted differently.
The first group was given images of things related to sleep and being tired (a bed), while the second received images connected to sickness (napkins, pills). When she later showed them the picture of a woman, the results coincided with the hypothesis: the first group decided that the woman in the picture was tired and yawning and the latter group believed that she was sneezing.
Olesya has also suggested expanding the line of work by combining the contextual and test images in a video sequence, replacing a visual stimulus for an audio one, as well as creating a third, erotic context that might be used for the later experiments.
What’s going on here? The illusion is based on the priming effect, which influences the perception of subsequent information. Exposure to one stimulus would influence response to a subsequent stimulus, without conscious guidance or intention, unlike with the Kuleshov effect where the focus on the object is important.
All in all, context can modify the way we perceive emotions in many ways. For example, people may not see the face at all, but they can guess what emotion people are expressing depending on the environment. More about that here.
There is also the Hysteresis effect that can be applied to emotional expression as well. The term is used to describe any system the interpretation of which depends not only on its current state, but also upon its past history. That means we can interpret the same emotion we see on our faces differently, depending on our previous experience. You can read more about it in this article.
Context is vital for emotion recognition. That is why it is crucial to search for the solutions to include it into this process. Complex cognitive states, hidden, mixed and fake emotions, among other things, would require analysis and understanding of the context, and this hasn’t yet been achieved. Unlike many tasks in computer vision where neural networks already do better than humans, emotion recognition is a much subtler task.
The context can be expressed both in the meaning of what we say and via specific circumstances around us. For example, where a person is, whether she is alone or with someone, what she is doing. Automatic context recognition is one of the most difficult tasks, and the future of Emotion & Social AI is closely related to it.
All the chosen contestants received a prize as well as got an exclusive look into our lab. Young minds are the future of science and in order to achieve it, we are responsible for giving opportunities to the specialists at the beginning of their career. We thank everyone involved. It was an amazing experience for us and we hope it wasn’t the last time we did something like this.
Authors: Elizaveta Zaitseva, SMM Manager at Neurodata Lab; Kristina Astakhova, Evangelist at Neurodata Lab
You are welcome to comment on this article in our blog on Medium.