Sage Journals: Discover world-class research

Abstract

Affect detection is a key component in developing Intelligent Human Computer Interface (IHCI) systems. State of the art affect detection systems assume the availability of full un-occluded face images. However image occlusion is a prominent problem which one comes across while dealing with such systems. The challenge is to identify affect states from portions of the face that are available. This paper proposes a novel method of assessing only a segment of the face instead of the whole face for affect detection. This paper aims at finding segments of the face which contain sufficient information to correctly classify the basic affect states. This work uses Convolutional Neural Networks (CNN) with transfer learning to detect 7 basic affect states viz. Angry, Contempt, Disgust, Fear, Happy, Sad and Surprise from a few prominent facial segments. Full face images are partitioned into separate segments viz. Right segment, Left segment, Lower segment and Upper segment. Modified VGG-16 and ResNet-50 networks were trained using each of the segments. Experiments were conducted using these facial segments and results obtained were compared with that of the full face. Using the VGG-16 network, we have been able to achieve validation accuracies of 96.8% for Full face, 97.3% for Right segment of the face, 97.3% for Left segment of the face, 96.6% for Lower segment of the face and 84.7% for Upper segment of the face. The validation accuracies are higher using the ResNet-50 network. Using the ResNet-50 network we have been able to achieve validation accuracies of 99.7% for Full face, 99.47% for Right segment of the face, 100% for Left segment of the face, 99.6% for Lower segment of the face and 90.8% for Upper segment of the face. Apart from accuracy, the other performance matrices used in this work are Precision, Recall and f1-score. Our evaluation, based on these performance matrices show that the results obtained for Right segment, Left segment and Lower segment of the face using both, VGG-16 as well as ResNet-50 networks, are comparable with that of the Full face. Experiments performed clearly indicate that Right segment, Left segment and Lower segment of the face contain sufficient information about the seven affect states and that CNN with transfer learning can be used to accurately classify them.

Keywords

Convolutional neural network transfer learning occlusion affect states

Get full access to this article

View all access options for this article.

References

Calvo

, and D’Mello

. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing1(1) (2010):18–37.

Dan

Ueli

, and Jürgen

. Multi-column deep neural networks for image classification. arXiv preprint arXiv:1202. 2745 (2012).

Prudhvi Raj

. Facial Emotion Detection Using Convolutional Neural Networks and Representational Autoencoder Units. arXiv preprint arXiv:1706.01509 (2017).

Tang

. Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0239 (2013).

Ekman

, and Friesen

. Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978).

Raghuvanshi

, and Vivek

. Facial Expression Recognition with Convolutional Neural Networks. CS231n Course Projects (2016).

Happy

, and Routray

. Automatic facial expression recognition using features of salient facial patches. IEEE transactions on Affective Computing6(1) (2015):1–12.

Hoque

McDuff

, and Picard

. Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Transactions on Affective Computing3(3) (2012):323–334.

Upal

Sayantan

, and Rama

. Segment-based methods for facial attribute detection from partial faces. IEEE Transactions on Affective Computing (2018).

10.

Zeiler

, and Fergus

. Visualizing and understanding convolutional networks. In European Conference on Computer Vision. Springer, Cham, 2014.

11.

Karen

, and Andrew

. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556 (2014).

12.

Zhang

Ren

, and Sun

. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.770–778.

13.

Lucey

Cohn

Kanade

Saragih

Ambadar

, and Matthews

. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE, 2010 June, pp.94–101.

14.

Viola

, and Jones

. Robust real-time face detection. International Journal of Computer Vision57(2) (2004): 137–154.

15.

Krizhevsky

Sutskever

, and Hinton

. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.

16.

https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/.

17.

Chollet

. Keras. GitHub, 2015.

18.

Pal

. Transfer Learning and Fine Tuning for Cross Domain Image Classificationwith Keras. GitHub: Transfer Learning and Fine Tuning for Cross Domain Image Classification with Keras, 2016.

19.

Deng

, et al. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.CVPR 2009.IEEE Conference on. Ieee, 2009.

20.

Gopalakrishnan

, et al. Deep Convolutional Neural Networks with transfer learning for computer vision- based data-driven pavement distress detection. Construction and Building Materials157 (2017): 322–330.

Classification of affect states from facial segments using transfer learning

Abstract

Keywords

Get full access to this article

References