Facial Expression Recognition from Facial Segments Using Pre-Trained Networks

Abstract

Automatic Facial expression recognition (AFER) plays an important role in enriching the human computer interaction (HCI) experience. Attaining an effective facial representation from face images is imperative for successful facial expression recognition. AFER systems are often faced with a problem of not being able to obtain un-occluded full face images and hence their tolerance to partial occlusion remains  less investigated. The paper endeavors to recognize the importance of various facial segments in the detection and classification of universal facial expressions viz. Disgust,  Happy, Surprised,  Fear, Angry Contempt  and  Sad. . In this work we explore which facial segments are effective in detecting affect states.   We partition full frontal face images into eight different segments viz. Right half segment, Left half segment, Lower half segment, Upper half segment, Quarter Upper Right segment, Quarter Upper left segment, Quarter Lower right segment and Quarter Lower left segment. While the Right segment, Left segment, Lower segment, Upper segment of the face contain 50% of the original information, the Quarter Upper Right segment, Quarter Upper left segment, Quarter Lower right segment and Quarter Lower left segment contain only 25% of the original full face information. The objective of the paper is to  recognize affect states from various sections of the face.  A commonly used deep learning network is the Convolutional Neural Network (CNN). This work applies CNN with transfer learning to detect  universal affect states from facial  segments. Three pre-trained networks used in this study are VGG-16, ResNet-50 and modified version of ResNet known as SE-ResNet-50 and the results obtained from the three networks are analyzed.  We used the Extended Cohn Kandade dataset (CK+) for our study. Experiments are performed using the eight segments and the results obtained are compared with the results obtained for the full face. The performance metrics used in this study are validation accuracy, precision, recall and f1-score. Through extensive experimentation we have shown that the universal affect states can be accurately classified using each of the 8 facial segments. We find that deep learning networks achieve exceedingly well in detection of affect states from partial face images. We deduce that both ResNet-50 and SE-ResNet-50 perform better than the VGG-16 network. We can thus state that the following segments viz. Right half segment, Left half segment, Lower half segment, Upper half segment, Quarter Upper Right segment, Quarter Upper left segment, Quarter Lower right segment and Quarter Lower left facial segment contain adequate information and hence we only need to have 25 % of the original face information for accurate facial expression recognition.