Emotion Detection in the Loop from Brain Signals …

1 ENTERFACE'06, July 17th August 11th, Dubrovnik, Croatia Final Project Report Emotion Detection in the Loop from Brain Signals and Facial Images Arman Savran , Koray Ciftci , Guillame Chanel , Javier Cruz Mota , Luong Hong Viet4, B lent Sankur , Lale Akarun , Alice Caplier5 and Michele Rombaut5. Bogazici University, University of Geneva, Universitat Politecnica de Catalunya, 4 The Francophone Institute for Computer Science, 5 Institut National Polytechnique de Grenoble Abstract I. INTRODUCTION. In this project, we intended to develop techniques for Detection and tracking of human emotions have many multimodal Emotion Detection , one modality being Brain Signals via fNIRS, the second modality being face video and potential applications ranging from involvement and the third modality being the scalp EEG Signals .

2 EEG and attentiveness measures in multimedia products to fNIRS provided us with an internal look at the Emotion Emotion -sensitive interactive games, from enhanced generation processes, while video sequence gave us an multimedia interfaces with more human-like interactions external look on the same phenomenon. to affective computing, from Emotion -sensitive automatic tutoring systems to the investigation of Fusions of fNIRS with video and of EEG with fNIRS were cognitive processes, monitoring of attention and of considered. Fusion of all three modalities was not considered mental fatigue. due to the extensive noise on the EEG Signals caused by facial muscle movements, which are required for Emotion Detection The majority of existing Emotion understanding from video sequences.

3 Techniques is based on a single modality such as PET, Besides the techniques mentioned above, peripheral Signals , fMRI, EEG or static face image or videos. The main namely, respiration, cardiac rate, and galvanic skin resistance goal of this project was to develop a multimodal were also measured from the subjects during fNIRS + EEG Emotion -understanding scheme using hemodynamic recordings. These Signals provided us with extra information Brain Signals , electrical Brain Signals and face images. about the emotional state of the subjects. Studies about the way to fusion the different modalities was also an important goal of the work. The critical point in the success of this project was to be able to build a good database.

4 Good data acquisition means Psychologists agree that human emotions can be synchronous data and requires the definition of some specific categorized into a small number of cases. For example, experimental protocols for emotions elicitation. Thus, we Ekman et al. [1] found that six different facial devoted much of our time to data acquisition throughout the expressions (fearful, angry, sad, disgust, happy, and workshop, which resulted in a large enough database for surprise) were categorically recognized by humans from making the first analyses. Results presented in this report distinct cultures using a standardized stimulus set. In should be considered as preliminary. However, they are other words, these facial expressions were stable over promising enough to extend the scope of the research.

5 Races, social strata and age brackets, and were consistent Index Terms Emotion Detection , EEG, video, near-infrared even in people blind by birth. spectroscopy Nevertheless, there are several difficulties in automatic human Emotion identification. First, the straightforward correlation of emotions with neural Signals or with facial actions may not be correct since emotions are affected by interactions with the environment. As a result, the This report, as well as the source code for the software developed during the unfolding of emotions contains substantial inter-subject project, is available online from the eNTERFACE'06 web site: and intra-subject differences, even though the individuals admit or seem to be in the claimed emotional eNTERFACE'06, July 17th August 11th, Dubrovnik, Croatia Final Project Report situation.

6 Moreover, to design experiments to single out contrast, fNIRS is the modality that can be combined a unique Emotion is a very challenging task. These imply with either video Signals or with EEG Signals . that, even small changes in the experimental setup may lead to non-negligible differences in the results. In summary, the first short-term goal of the project has been to build a reliable database that can be used for all The majority of existing Emotion understanding related future research. The second such goal was to techniques is based on a single modality such as PET, prove the viability of a multi-modal approach to Emotion fMRI, EEG or static face image or videos. The main recognition, both from instrumentation and signal goal of this project was to develop a multimodal processing points of view.

7 The final long-term aim is to Emotion -understanding scheme using functional, build an integrated framework for multi-modal Emotion physiological and visible data. As an intermediate step, it recognition for both Brain research and affective- was necessary to determine the feasibility of fusing computing aspects. different modalities for Emotion recognition. These modalities are functional Near Infrared Spectroscopy (fNIRS) electroencephalogram (EEG), video and II. MEASUREMENT SETUP AND Emotion . peripheral Signals . Note that these modalities provide us ELICITING. with different aspects of the same phenomenon. fNIRS. and EEG try to detect functional hemodynamic and A. Instrumental Setup electrical changes, peripheral Signals give an indication of Emotion -related changes in the human body and video To detect and estimate emotions based on Brain as well signal captures the visible changes caused by Emotion as physiological Signals the following sensor setup was elicitation.

8 Prepared: (Figure 1): In the rapidly evolving Brain -computer interface area, fNIRS sensor to record frontal Brain activity, fNIRS (functional Near Infrared Spectroscopy) EEG sensor to capture activity in the rest of the represents a low-cost, user-friendly, practical device for Brain , monitoring the cognitive and emotional states of the Sensors for acquiring peripheral body processes: Brain , especially from the prefrontal cortex area. fNIRS a respiration belt, a GSR (Galvanic Skin detects the light (photon count) that travels through the Response) and a plethysmograph (blood volume cortex tissues and is used to monitor the hemodynamic pressure). changes during cognitive and/or emotional activity. All these devices were synchronized using a trigger The second modality to estimate cortical activity is EEG.

9 Mechanism. Notice that EEG and fNIRS sensor Using the scalp electrodes, useful information about the arrangements partially overlap, so that there is no EEG. emotional state may be obtained as long as stable EEG recording on the front. Similarly the fNIRS device patterns on the scalp are produced. EEG recordings covers the eyebrows, occluding one of the image capture neural electrical activity on a millisecond scale features for Emotion recognition from the entire cortical surface while fNIRS records fNIRS acquisition hemodynamic reactions to neural Signals on a seconds trigger scale from the frontal lobe. In fact, electrical activity takes place in order of milliseconds, whereas stimuli hemodynamic activity may reach its peak in 6-10.

10 Seconds and may last for 30 seconds. In addition to these fNIRS device modalities, peripheral Signals , namely, galvanic skin response (GSR), respiration and blood volume pressure (from which we can compute heart rate) were also recorded. EEG + periph. response We have combined these four monitoring modes of device emotions in two separate pairs, namely: i) fNIRS, ii) EEG acquisition EEG, iii) peripheral Signals , iv) image or video. Notice that EEG is very sensitive to electrical Signals emanating Figure 1 Schematics of EEG and fNIRS acquisition. from facial muscles while emotions are being expressed, hence EEG and video modalities cannot coexist. In eNTERFACE'06, July 17th August 11th, Dubrovnik, Croatia Final Project Report The Video-fNIRS acquisition scenario is composed participants on two dimensions of nine points each (1-9): of three computers, Stimulus Computer, fNIRS Computer valence (ranging from positive to negative or unpleasant and Video Computer, each one with the following to pleasant) and arousal (ranging from calm to exciting).