Multimodal Emotion Recognition using Facial Expressions, Body Gestures, Speech, and Text Modalities
Mahesh G. Huddar1, Sanjeev S. Sannakki2, Vijay S. Rajpurohit3
1Mahesh G. Huddar, Department of Computer Science and Engineering, Hirasugar Institute of Technology, Nidasoshi (Karnataka), India.
2Sanjeev S. Sannakki, Department of Computer Science and Engineering, Gogte Institute of Technology, Belgaum (Karnataka), India.
3Vijay S. Rajpurohit, Department of Computer Science and Engineering, Gogte Institute of Technology, Belgaum (Karnataka), India.
Manuscript received on 18 June 2019 | Revised Manuscript received on 25 June 2019 | Manuscript published on 30 June 2019 | PP: 2453-2459 | Volume-8 Issue-5, June 2019 | Retrieval Number: E7562068519/19©BEIESP
Open Access | Ethics and Policies | Cite | Mendeley | Indexing and Abstracting
© The Authors. Blue Eyes Intelligence Engineering and Sciences Publication (BEIESP). This is an open access article under the CC-BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Abstract: Automatic emotion recognition from multimodal content has become an important and growing research field in human-computer interaction. Recent literature has used either audio or facial expression for emotion detection. However, emotion and body gestures are closely related to one another. This paper explores the effectiveness of using text, audio, facial expression and body gesture modalities of multimodal content and machine learning and deep learning based models for building more accurate and robust automatic multimodal emotion recognition systems. First, we get the best accuracy from the individual modalities. Then we use feature level fusion and ensemble based decision level fusion to combine multiple modalities to get better results. Proposed models were tested on IEMOCAP dataset and results show that proposed models with multiple modalities are more accurate compared to unimodal models in classifying emotions.
Keywords: Affective Computing, Multimodal Emotion Classification, Deep Learning, Ensemble, IEMOCAP, LSTM
Scope of the Article: Deep Learning