1 Identifying Emotions from Walking Using Affective and ...

1 1 Identifying Emotions from Walking UsingAffective and Deep FeaturesTanmay Randhavane, Uttaran Bhattacharya, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh ManochaAbstract We present a new data-driven model and algorithm to identify the perceived Emotions of individuals based on their walkingstyles. Given an RGB video of an individual Walking , we extract his/her Walking gait in the form of a series of 3D poses. Our goal is toexploit the gait features to classify the emotional state of the human into one of four Emotions : happy, sad, angry, or neutral. Ourperceived emotion identification approach uses deep features learned via LSTM on labeled emotion datasets. Furthermore, wecombine these features with Affective features computed from gaits Using posture and movement cues. These features are classifiedusing a Random Forest Classifier.

2 We show that our mapping between the combined feature space and the perceived emotional in Identifying the perceived Emotions . In addition to Identifying discrete categories of Emotions , our algorithmalso predicts the values of perceived valence and arousal from gaits. We also present an EWalk(Emotion Walk) dataset that consistsof videos of Walking individuals with gaits and labeled Emotions . To the best of our knowledge, this is the first gait-based model toidentify perceived Emotions from videos of Walking INTRODUCTIONEMOTIONS play a large role in our lives, defining ourexperiences and shaping how we view the world andinteract with other humans. Perceiving the Emotions of so-cial partners helps us understand their behaviors and decideour actions towards them. For example, people communi-cate very differently with someone they perceive to be angryand hostile than they do with someone they perceive to becalm and content.

3 Furthermore, the Emotions of unknownindividuals can also govern our behavior, ( , emotionsof pedestrians at a road-crossing or Emotions of passengersin a train station). Because of the importance of perceivedemotion in everyday life, automatic emotion recognition isa critical problem in many fields such as games and enter-tainment, security and law enforcement, shopping, human-computer interaction, human-robot interaction, perceive the Emotions of other individuals us-ing verbal and non-verbal cues. Robots and AI devicesthat possess speech understanding and natural languageprocessing capabilities are better at interacting with hu-mans. Deep learning techniques can be used for speechemotion recognition and can facilitate better interactionswith humans [1].Understanding the perceived Emotions of individualsusing non-verbal cues is a challenging problem.

4 Humansuse the non-verbal cues of facial expressions and bodymovements to perceive Emotions . With a more extensiveavailability of data, considerable research has focused onusing facial expressions to understand emotion [2]. How-ever, recent studies in psychology question the commu-nicative purpose of facial expressions and doubt the quick,automatic process of perceiving Emotions from these ex- T. Randhavane is with the Department of Computer Science, Universityof North Carolina, Chapel Hill, NC, : K. Kapsaskis and K. Gray are with University of North Carolina at ChapelHill. A. Bera, U. Bhattacharya, and D. Manocha are with University ofMaryland at College 1:Identiying Perceived Emotions : We present a novelalgorithm to identify the perceived Emotions of individualsbased on their Walking styles.

5 Given an RGB video of anindividual Walking (top), we extract his/her Walking gaitas a series of 3D poses (bottom). We use a combination ofdeep features learned via an LSTM and Affective featurescomputed Using posture and movement cues to then classifyinto basic Emotions ( , happy, sad, etc.) Using a RandomForest [3]. There are situations when facial expressionscan be unreliable, such as with mock or referentialexpressions [4]. Facial expressions can also be unreliabledepending on whether an audience is present [5].Research has shown that body expressions are also cru-cial in emotion expression and perception [6]. For example, [ ] 9 Jan 20202when presented with bodies and faces that expressed ei-ther anger or fear (matched correctly with each other oras mismatched compound images), observers are biasedtowards body expression [7].

6 Aviezer et al. s study [8] onpositive/negative valence in tennis players showed thatfaces alone were not a diagnostic predictor of valence,but the body alone or the face and body together can , body expression in Walking , or an indi-vidual s gait, has been proven to aid in the perceptionof Emotions . In an early study by Montepare et al. [9],participants were able to identify sadness, anger, happiness,and pride at a significant rate by observing Affective featuressuch as increased arm swinging, long strides, a greater footlanding force, and erect posture. Specific movements havealso been correlated with specific Emotions . For example,sad movements are characterized by a collapsed upper bodyand low movement activity [10]. Happy movements have afaster pace with more arm swaying [11].

7 main Results:We present an automatic emotion iden-tification approach for videos of Walking individuals (Fig-ure 1). We classify Walking individuals from videos intohappy, sad, angry, and neutral emotion categories. Theseemotions represent emotional states that last for an extendedperiod and are more abundant during Walking [12]. Weextract gaits from Walking videos as 3D poses. We use anLSTM-based approach to obtain deep features by modelingthe long-term temporal dependencies in these sequential3D human poses. We also present spatiotemporalaffectivefeaturesrepresent ing the posture and movement of walkinghumans. We combine these Affective features with LSTM-based deep features and use a Random Forest Classifier toclassify them into four categories of emotion.

8 We observe animprovement the classification accuracy overother gait-based perceived emotion classification algorithms(Table 5). We refer to our LSTM-based model betweenaffective and deep features and the perceived emotion labelsas our novel data-driven also present a new dataset, Emotion Walk (EWalk), which contains videos of individuals Walking in both indoorand outdoor locations. Our dataset consists of1384gaitsand the perceived Emotions labeled Using Mechanical of the novel components of our work include:1. A novel data-driven mapping between the Affective fea-tures extracted from a Walking video and the A novel emotion identification algorithm that combinesaffective features and deep features, A new public domain dataset,EWalk, with walkingvideos, gaits, and labeled rest of the paper is organized as follows.

9 In Section2, we review the related work in the fields of emotionmodeling, bodily expression of emotion, and automaticrecognition of emotion Using body expressions. In Section3, we give an overview of our approach and present theaffective features. We provide the details of our LSTM-based approach to Identifying perceived Emotions fromwalking videos in Section 4. We compare the performance ofour method with state-of-the-art methods in Section 5. Wepresent theEWalkdataset in Section 2: All discrete Emotions can be represented by points ona 2D affect space of Valence and Arousal [13], [14].2 RELATEDWORKIn this section, we give a brief overview of previous workson emotion representation, emotion expression Using bodyposture and movement, and automatic emotion Emotion RepresentationEmotions have been represented Using both discrete andcontinuous representations [6], [14], [15].

10 In this paper, wefocus on discrete representations of the Emotions and iden-tify four discrete Emotions (happy, angry, sad, and neutral).However, a combination of these Emotions can be used toobtain the continuous representation (Section ). A map-ping between continuous representation and the discretecategories developed by Mikels and Morris [16], [17] canbe used to predict other is important to distinguish between perceived emo-tions and actual Emotions as we discuss the perception ofemotions. One of the most obvious cues to another person semotional state is his or her self-report [18]. However, self-reports are not always available; for example, when peopleobserve others remotely ( , via cameras), they do not havethe ability to ask about their emotional state. Additionally,self-reports can be imperfect because people can experiencean emotion without being aware of it or be unable totranslate the emotion into words [19].