Convolutional Neural Networks for Facial Expression ...

1 Convolutional Neural Networks for Facial Expression RecognitionShima AlizadehStanford FazelStanford this project, we have developed convolutionalneural Networks (CNN) for a Facial Expression recog-nition task. The goal is to classify each Facial imageinto one of the seven Facial emotion categories consid-ered in this study. We trained CNN models with dif-ferent depth using gray-scale images from the Kagglewebsite [1]. We developed our models in Torch [2] andexploited Graphics Processing Unit (GPU) computa-tion in order to expedite the training process.

2 In ad-dition to the Networks performing based on raw pixeldata, we employed a hybrid feature strategy by whichwe trained a novel CNN model with the combinationof raw pixel data and Histogram of Oriented Gradi-ents (HOG) features [3]. To reduce the overfitting ofthe models, we utilized different techniques includingdropout and batch normalization in addition to L2 reg-ularization. We applied cross validation to determinethe optimal hyper-parameters and evaluated the per-formance of the developed models by looking at theirtraining histories.

3 We also present the visualization ofdifferent layers of a network to show what features ofa face can be learned by CNN IntroductionHumans interact with each other mainly throughspeech, but also through body gestures, to emphasizecertain parts of their speech and to display of the important ways humans display emotions isthrough Facial expressions which are a very importantpart of communication. Though nothing is said ver-bally, there is much to be understood about the mes-sages we send and receive through the use of nonver-bal communication.

4 Facial expressions convey non-verbal cues, and they play an important role in inter-personal relations [4, 5]. Automatic recognition of fa-cial expressions can be an important component of nat-ural human-machine interfaces; it may also be used inbehavioral science and in clinical practice. Althoughhumans recognize Facial expressions virtually withouteffort or delay, reliable Expression recognition by ma-chine is still a challenge. There have been several ad-vances in the past few years in terms of face detec-tion, feature extraction mechanisms and the techniquesused for Expression classification, but development ofan automated system that accomplishes this task is dif-ficult [6].

5 In this paper, we present an approach basedon Convolutional Neural Networks (CNN) for facialexpression recognition. The input into our system isan image; then, we use CNN to predict the Facial ex-pression label which should be one these labels: anger,happiness, fear, sadness, disgust and Related WorkIn recent years, researchers have made consider-able progress in developing automatic Expression clas-sifiers [7, 8, 9]. Some Expression recognition sys-tems classify the face into a set of prototypical emo-tions such as happiness, sadness and anger.

6 [10]. Oth-ers attempt to recognize the individual muscle move-ments that the face can produce [11] in order to pro-vide an objective description of the face. The bestknown psychological framework for describing nearlythe entirety of Facial movements is the Facial ActionCoding System (FACS) [12]. FACS is a system toclassify human Facial movements by their appearanceon the face using Action Units (AU). An AU is oneof 46 atomic elements of visible Facial movement orits associated deformation; an Expression typically re-sults from the accumulation of several AUs [7, 8].

7 1 Moreover, there have been several developments inthe techniques used for Facial Expression recognition:Bayesian Networks , Neural Networks and the multi-level Hidden Markov Model (HMM) [13, 14]. Someof them contain drawbacks of recognition rate or tim-ing. Usually, to achieve accurate recognition two ormore techniques can be combined; then, features areextracted as needed. The success of each technique isdependent on pre-processing of the images because ofillumination and feature MethodsWe developed CNNs with variable depths toevaluate the performance of these models for facialexpression recognition.

8 We considered the followingnetwork architecture in our investigation:[Conv-(SBN)-ReLU-(Dropout) -(Max-pool)]M -[Affine-(BN)-ReLU-(Dropout)]N - Affine - first part of the network refers to M convo-lutional layers that can possess spatial batch normal-ization (SBN), dropout, and max-pooling in additionto the convlution layer and ReLU nonlinearity, whichalways exists in these layers. After M convolutionlayers, the network is led to N fully connected lay-ers that always have Affine operation and ReLU non-linearity, and can include batch normalization (BN)and dropout.

9 Finally, the network is followed by theaffine layer that computes the scores and softmax lossfunction. The developed model gives the user thefreedom to decide about the number of convlutionaland fully connected layers, as well as the existanceof batch normalization, dropout and max-pooling lay-ers. Along with dropout and batch normalization tech-niques, we included L2 regularization in our imple-mentation. Furthermore, the number of filters, strides,and zero-padding can be specified by user, and if theyare not given, the default values are we will describe in the next section, we proposedthe idea of combining HOG features with those ex-tracted by Convolutional layers by mean of raw pixeldata.

10 To this end, we utilized the same architecture de-scribed above, but with this difference that we addedthe HOG features to those exiting the last convolutionlayer. The hybrid feature set then enters the fully con-nected layers for score and loss 1: Examples of seven Facial emotions that weconsider in this classification problem. (a) angry, (b)neutral, (c) sad, (d) happy, (e) surprise, (f) fear, (g) disgustWe implemented the aforementioned model in Torchand took advantage of GPU accelerated deep learningfeatures to make the model training process Dataset and FeaturesIn this project, we used a dataset provided by Kag-gle website, which consists of about 37,000 well-structured48 48pixel gray-scale images of images are processed in such a way that the facesare almost centered and each face occupies about thesame amount of space in each image.