A multimodal emotion recognition model based on deep neural network with cross-media data feature fusion

Sheng-tong Liu; Zhi-hong Chen; Peng-hai Li; Jia Lu; Xin-wang Shao; Xiao-qi Gao; Hai-wei Zhang; Xiao-ping Yang

doi:10.1117/12.2634393

4 May 2022 A multimodal emotion recognition model based on deep neural network with cross-media data feature fusion

Sheng-tong Liu, Zhi-hong Chen, Peng-hai Li, Jia Lu, Xin-wang Shao, Xiao-qi Gao, Hai-wei Zhang, Xiao-ping Yang

Proceedings Volume 12172, International Conference on Electronic Information Engineering and Computer Communication (EIECC 2021); 121720V (2022) https://doi.org/10.1117/12.2634393
Event: International Conference on Electronic Information Engineering and Computer Communication (EIECC 2021), 2021, Nanchang, China

Abstract

Affective computing is an interdisciplinary research area that includes machine learning and pattern recognition, psychology, and cognitive science. The aim is to research and develop theories, methods and systems that can recognize, interpret, process and simulate human emotions. In this article we propose a neural network model for multimodal emotion recognition based on cross-media data-feature fusion. Multimodal data fusion can effectively improve the accuracy of emotion recognition. We extract features from EEG data and facial images using a deep double-stream neural network and then merge them in a medium-term feature layer to identify three categories of emotions (sadness, calm, and happiness). The experimental results show that the detection accuracy can reach over 95%. Compared to the traditional single-modal emotion recognition method, the accuracy rate of emotion recognition based on EEG data and facial images has been significantly improved. It also proves that the multimodal medium-term feature layer fusion method has good applicability for emotion recognition.

Citation Download Citation

Sheng-tong Liu, Zhi-hong Chen, Peng-hai Li, Jia Lu, Xin-wang Shao, Xiao-qi Gao, Hai-wei Zhang, and Xiao-ping Yang "A multimodal emotion recognition model based on deep neural network with cross-media data feature fusion", Proc. SPIE 12172, International Conference on Electronic Information Engineering and Computer Communication (EIECC 2021), 121720V (4 May 2022); https://doi.org/10.1117/12.2634393

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Electroencephalography

Data modeling

Neural networks

Data fusion

Image fusion

Performance modeling

Feature extraction

Show All Keywords

Keywords/Phrases

Search In:

Publication Years