Convolutional Neural Networks for Facial Expression ...

1 Convolutional Neural Networks for Facial Expression RecognitionShima AlizadehStanford FazelStanford this project, we have developed convolutionalneural Networks (CNN) for a Facial Expression recog- nition task. The goal is to classify each Facial imageinto one of the seven Facial emotion categories consid-ered in this study. We trained CNN models with dif-ferent depth using gray-scale images from the Kagglewebsite [1]. We developed our models in Torch [2] andexploited Graphics Processing Unit (GPU) computa-tion in order to expedite the training process. In ad-dition to the Networks performing based on raw pixeldata, we employed a hybrid feature strategy by whichwe trained a novel CNN model with the combinationof raw pixel data and Histogram of Oriented Gradi-ents (HOG) features [3]. To reduce the overfitting ofthe models, we utilized different techniques includingdropout and batch normalization in addition to L2 reg-ularization.

2 We applied cross validation to determinethe optimal hyper-parameters and evaluated the per-formance of the developed models by looking at theirtraining histories. We also present the visualization ofdifferent layers of a network to show what features ofa face can be learned by CNN IntroductionHumans interact with each other mainly throughspeech, but also through body gestures, to emphasizecertain parts of their speech and to display of the important ways humans display emotions isthrough Facial expressions which are a very importantpart of communication. Though nothing is said ver-bally, there is much to be understood about the mes-sages we send and receive through the use of nonver-bal communication. Facial expressions convey non-verbal cues, and they play an important role in inter-personal relations [4, 5]. Automatic recognition of fa-cial expressions can be an important component of nat-ural human-machine interfaces; it may also be used inbehavioral science and in clinical practice.

3 Althoughhumans recognize Facial expressions virtually withouteffort or delay, reliable Expression recognition by ma-chine is still a challenge. There have been several ad-vances in the past few years in terms of face detec-tion, feature extraction mechanisms and the techniquesused for Expression classification, but development ofan automated system that accomplishes this task is dif-ficult [6]. In this paper, we present an approach basedon Convolutional Neural Networks (CNN) for facialexpression recognition. The input into our system isan image; then, we use CNN to predict the Facial ex-pression label which should be one these labels: anger,happiness, fear, sadness, disgust and Related WorkIn recent years, researchers have made consider-able progress in developing automatic Expression clas-sifiers [7, 8, 9]. Some Expression recognition sys-tems classify the face into a set of prototypical emo-tions such as happiness, sadness and anger.

4 [10]. Oth-ers attempt to recognize the individual muscle move-ments that the face can produce [11] in order to pro-vide an objective description of the face. The bestknown psychological framework for describing nearlythe entirety of Facial movements is the Facial ActionCoding System (FACS) [12]. FACS is a system toclassify human Facial movements by their appearanceon the face using Action Units (AU). An AU is oneof 46 atomic elements of visible Facial movement orits associated deformation; an Expression typically re-sults from the accumulation of several AUs [7, 8].1 Moreover, there have been several developments inthe techniques used for Facial Expression recognition:Bayesian Networks , Neural Networks and the multi-level Hidden Markov Model (HMM) [13, 14]. Someof them contain drawbacks of recognition rate or tim-ing. Usually, to achieve accurate recognition two ormore techniques can be combined; then, features areextracted as needed.

5 The success of each technique isdependent on pre-processing of the images because ofillumination and feature MethodsWe developed CNNs with variable depths toevaluate the performance of these models for facialexpression recognition. We considered the followingnetwork architecture in our investigation:[Conv-(SBN)-ReLU-(Dropout) -(Max-pool)]M -[Affine-(BN)-ReLU-(Dropout)]N - Affine - first part of the network refers to M convo-lutional layers that can possess spatial batch normal-ization (SBN), dropout, and max-pooling in additionto the convlution layer and ReLU nonlinearity, whichalways exists in these layers. After M convolutionlayers, the network is led to N fully connected lay-ers that always have Affine operation and ReLU non-linearity, and can include batch normalization (BN)and dropout. Finally, the network is followed by theaffine layer that computes the scores and softmax lossfunction.

6 The developed model gives the user thefreedom to decide about the number of convlutionaland fully connected layers, as well as the existanceof batch normalization, dropout and max-pooling lay-ers. Along with dropout and batch normalization tech-niques, we included L2 regularization in our imple-mentation. Furthermore, the number of filters, strides,and zero-padding can be specified by user, and if theyare not given, the default values are we will describe in the next section, we proposedthe idea of combining HOG features with those ex-tracted by Convolutional layers by mean of raw pixeldata. To this end, we utilized the same architecture de-scribed above, but with this difference that we addedthe HOG features to those exiting the last convolutionlayer. The hybrid feature set then enters the fully con-nected layers for score and loss 1: Examples of seven Facial emotions that weconsider in this classification problem.

7 (a) angry, (b)neutral, (c) sad, (d) happy, (e) surprise, (f) fear, (g) disgustWe implemented the aforementioned model in Torchand took advantage of GPU accelerated deep learningfeatures to make the model training process Dataset and FeaturesIn this project, we used a dataset provided by Kag-gle website, which consists of about 37,000 well-structured48 48pixel gray-scale images of images are processed in such a way that the facesare almost centered and each face occupies about thesame amount of space in each image. Each image hasto be categorized into one of the seven classes thatexpress different Facial emotions . These Facial emo-tions have been categorized as: 0=Angry, 1=Disgust,2=Fear, 3=Happy, 4=Sad, 5=Surprise, and 6= 1 depicts one example for each Facial expres-sion category. In addition to the image class number (anumber between 0 and 6), the given images are dividedinto three different sets which are training, validation,and test sets.

8 There are about 29,000 training images,4,000 validation images, and 4,000 images for reading the raw pixel data, we normalized themby subtracting the mean of the training images fromeach image including those in the validation and testsets. For the purpose of data augmentation, we pro-duced mirrored images by flipping images in the train-ing set order to classify the expressions, mainly we usedthe features generated by convolution layers using theraw pixel data. As an extra exploration, we devel-oped learning models that concatenate the HOG fea-tures with those generated by Convolutional layers andgive them as input features into Fully Connected (FC) Neurons512 Table 1: The hyper-parameters obtained by crossvalidation for the shallow model5. ExperimentsFor the purpose of this project, first we built a shal-low CNN. This network had two Convolutional layersand one FC layer. In the first Convolutional layer, wehad 323 3 filters, with the stride of size 1, along withbatch normalization and dropout, but without max-pooling.

9 In the second Convolutional layer, we had 643 3 filters, with the stride od size 1, along with batchnormalization and dropout and also max-pooling witha filter size2 2. In the FC layer, we had a hidden layerwith 512 neurons and Softmax as the loss in all the layers, we used Rectified Linear Unit(ReLU) as the activation function. Before training ourmodel, we did some sanity checks to make sure that theimplementation of the network was correct. For thefirst sanity check, we computed the initial loss whenthere is no regularization. Since our classifier has 7different classes, we expected to get a value As the second sanity check, we tried to over-fit our model using a small subset of the training shallow model passed both of these sanity , we started training our model from scratch. Tomake the model training process faster, we exploitedGPU accelerated deep learning facilities on Torch.

10 Forthe training process, we used all of the images in thetraining set with 30 epochs and a batch size of 128and cross-validated the hyper-patameters of the modelwith different values for regularization, learning rateand the number of hidden neurons . To validate ourmodel in each iteration, we used the validation set andto evaluate the performance of the model, we used thetest set. The best shallow model, gave us 55% accu-racy on the validation set and 54% on the test set. Ta-ble [1] summarizes the hyper-parameters obtained bycross validation for the shallow observe the effect of adding Convolutional layersand FC layers to the network , we trained a deeper CNNwith 4 Convolutional layers and two FC layers. Thefirst Convolutional layer had 643 3 filters, the secondone had 1285 5 filters, the third one had 5123 3filters and the last one had 5123 3 filters. In all theconvolutional layers, we have a stride of size 1, batchnormalization, dropout, max-pooling and ReLU as theactivation function.