IMAGE PROCESSING FACIAL EXPRESSION RECOGNITION

1 IMAGE PROCESSING FACIAL EXPRESSION RECOGNITION Report submitted for the partial fulfillment of the requirements for the degree of Bachelor of Technology in Information Technology Submitted by ANGANA MITRA University Roll No. 11700214011 Registration No. 141170110106 SOUVIK CHOUDHURY University Roll No. 11700214069 Registration No. - 141170110164 SUSMITA MOITRA University Roll No. 11700214080 Registration No. 141170110175 Under the Guidance of Mr. AMIT KHAN Assistant Professor, Department of Information Technology RCC Institute of Information Technology RCC Institute of Information Technology Acknowledgement We would like to express our sincere gratitude to Mr. Amit Khan, Assistant Professor of the department of Information Technology, whose role as project guide was invaluable for the project. We are extremely thankful for the keen interest he took in advising us, for the books and reference materials provided for the moral support extended to us.

2 Last but not the least we convey our gratitude to all the teachers for providing us the technical skill that will always remain as our asset and to all non-teaching staff for the gracious hospitality they offered us. Place: RCCIIT, Kolkata Date: 14th May, 2018 ANGANA MITRA SOUVIK CHOUDHURY SUSMITA MOITRA Department of Information Technology RCCIIT, Beliaghata, Kolkata 700 015, West Bengal, India Approval This is to certify that the project report entitled IMAGE PROCESSING ( FACIAL EXPRESSION RECOGNITION ) prepare under my supervision by ANGANA MITRA (11700214011), SOUVIK CHOUDHURY (11700214069) & SUSMITA MOITRA (11700214080) be accepted in partial fulfillment for the degree of Bachelor of Technology in Information Technology.

3 It is to be understood that by this approval, the undersigned does not necessarily endorse or approve any statement made, opinion expressed or conclusion drawn thereof, but approves the report only for the purpose for which it has been submitted. Mr. Amit Khan Assistant Professor, Department of Information Technology, RCCIIT, Kolkata Dr. Abhijit Das ABSTRACT These Human FACIAL expressions convey a lot of information visually rather than articulately. FACIAL EXPRESSION RECOGNITION plays a crucial role in the area of human-machine interaction. Automatic FACIAL EXPRESSION RECOGNITION system has many applications including, but not limited to, human behavior understanding, detection of mental disorders, and synthetic human expressions.

4 RECOGNITION of FACIAL EXPRESSION by computer with high RECOGNITION rate is still a challenging task. Two popular methods utilized mostly in the literature for the automatic FER systems are based on geometry and appearance. FACIAL EXPRESSION RECOGNITION usually performed in four-stages consisting of pre- PROCESSING , face detection, feature extraction, and EXPRESSION classification. In this project we applied various deep learning methods (convolutional neural networks) to identify the key seven human emotions: anger, disgust, fear, happiness, sadness, surprise and neutrality. Table Of Contents Sl No. Topics Page No. 1. Introduction 1 2. Motivation 2 3. Problem Definition 3-4 4. Literature Study 5-11 5. Software Requirement 12 6. Planning 13 7. Design 14-17 8. Algorithm 18 9. Implementation Details 19-30 10. Implementation Of Problem 31-37 11. Result 38-42 12. Conclusion 43 13. Future Scope 44 14. Reference 45-46 15. Appendix 47-61 Index Of Images Sl No. Topics Pages 1.

5 Basic Human Emotion 1 Monaliza 2 Deaf & Dumb 2 3. Problem Formulation 3 Pre- PROCESSING 5 Face Registration 6 FACIAL Feature Extraction 6 Emotion Classification 6 Neural Network 7 Gabor Filter 8 FER2013 Images 19 FER2013 Sample 19 Python Alternative To Matlab 22 Haar Features 24 Adaboost 25 Cascade Workflow 26 Artificial Neural Network 28 Deep Convolution Neural Network Architecture 29 Convolution Neural Network Layers 30 Overview Of FER2013 Database 31 Training & Validation Data Distribution 31 FER CNN Architecture 32 Convolutional & Maxpooling Of Neural Network 32 CNN Forward & Backward Propagation 33 Final Model CNN 34 Architecture Prediction Of Example Faces From Database 34-35 Confusion Matrix 36 Correct Prediction On 2nd & 3rd Highest Probable Emotion 36 CNN Feature Maps After 2nd Layer Of Maxpooling 37 CNN Feature Maps After 3rd Layer Of Maxpooling 37 Pixel Representation Of Database Images 37 Input Sample 1 40 Greyscale Sample 1 40 48*48 Greyscale

6 Sample 1 40 Input Sample 2 41 Greyscale Sample 2 41 48*48 Greyscale Sample 2 41 Input Sample 3 42 Greyscale Sample 3 42 48*48 Greyscale Sample 3 42 Index Of Tables Sl. No. Topics Pages 1. Accuracy of various database 10 2. Accuracy of various approaches are stated as follows 10-11 1. INTRODUCTION : 2018 is the year when machines learn to grasp human emotions --Andrew Moore, the dean of computer science at Carnegie Mellon. With the advent of modern technology our desires went high and it binds no bounds. In the present era a huge research work is going on in the field of digital IMAGE and IMAGE PROCESSING . The way of progression has been exponential and it is ever increasing. IMAGE PROCESSING is a vast area of research in present day world and its applications are very widespread. IMAGE PROCESSING is the field of signal PROCESSING where both the input and output signals are images. One of the most important application of IMAGE PROCESSING is FACIAL EXPRESSION RECOGNITION .

7 Our emotion is revealed by the expressions in our face. FACIAL Expressions plays an important role in interpersonal communication. FACIAL EXPRESSION is a non verbal scientific gesture which gets expressed in our face as per our emotions. Automatic RECOGNITION of FACIAL EXPRESSION plays an important role in artificial intelligence and robotics and thus it is a need of the generation. Some application related to this include Personal identification and Access control, Videophone and Teleconferencing, Forensic application, Human-Computer Interaction, Automated Surveillance, Cosmetology and so on. The objective of this project is to develop Automatic FACIAL EXPRESSION RECOGNITION System which can take human FACIAL images containing some EXPRESSION as input and recognize and classify it into seven different EXPRESSION class such as : I. Neutral II. Angry III. Disgust IV. Fear V. Happy VI. Sadness VII. Surprise 1. Several Projects have already been done in this fields and our goal will not only be to develop an Automatic FACIAL EXPRESSION RECOGNITION System but also improving the accuracy of this system compared to the other available systems.

8 2. MOTIVATION : Significant debate has risen in past regarding the emotions portrayed in the world famous masterpiece of Mona Lisa. British Weekly New Scientist has stated that she is in fact a blend of many different emotions, 83%happy, 9% disgusted, 6% fearful, 2% angry. We have also been motivated observing the benefits of physically handicapped people like deaf and dumb. But if any normal human being or an automated system can understand their needs by observing their FACIAL EXPRESSION then it becomes a lot easier for them to make the fellow human or automated system understand their needs. DEFINITION : Human FACIAL expressions can be easily classified into 7 basic emotions: happy, sad, surprise, fear, anger, disgust, and neutral. Our FACIAL emotions are expressed through activation of specific sets of FACIAL muscles. These sometimes subtle, yet complex, signals in an EXPRESSION often contain an abundant amount of information about our state of mind.

9 Through FACIAL emotion RECOGNITION , we are able to measure the effects that content and services have on the audience/users through an easy and low-cost procedure. For example, retailers may use these metrics to evaluate customer interest. Healthcare providers can provide better service by using additional information about patients' emotional state during treatment. Entertainment producers can monitor audience engagement in events to consistently create desired content. Humans are well-trained in reading the emotions of others, in fact, at just 14 months old, babies can already tell the difference between happy and sad. But can computers do a better job than us in accessing emotional states? To answer the question, We designed a deep learning neural network that gives machines the ability to make inferences about our emotional states. In other words, we give them eyes to see what we can see.

10 Problem formulation of our project: 3. FACIAL EXPRESSION RECOGNITION is a process performed by humans or computers, which consists of: 1. Locating faces in the scene ( , in an IMAGE ; this step is also referred to as facedetection), 2. Extracting FACIAL features from the detected face region ( , detecting the shape of facialcomponents or describing the texture of the skin in a FACIAL area; this step is referred to asfacial feature extraction), 3. Analyzing the motion of FACIAL features and/or the changes in the appearance of facialfeatures and classifying this information into some FACIAL - EXPRESSION - interpretativecategories such as FACIAL muscle activations like smile or frown, emotion (affect)categories like happiness or anger, attitude categories like (dis)liking or ambivalence, etc.(this step is also referred to as FACIAL EXPRESSION interpretation).