CNN and LSTM-Based Emotion Charting Using Physiological Signals

Muhammad Najam Dar* (Corresponding Author), Muhammad Usman Akram, Sajid Gul Khawaja, Amit N. Pujari

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Novel trends in affective computing are based on reliable sources of physiological signals such as Electroencephalogram (EEG), Electrocardiogram (ECG), and Galvanic Skin Response (GSR). The use of these signals provides challenges of performance improvement within a broader set of emotion classes in a less constrained real-world environment. To overcome these challenges, we propose a computational framework of 2D Convolutional Neural Network (CNN) architecture for the arrangement of 14 channels of EEG, and a combination of Long Short-Term Memory (LSTM) and 1D-CNN architecture for ECG and GSR. Our approach is subject-independent and incorporates two publicly available datasets of DREAMER and AMIGOS with low-cost, wearable sensors to extract physiological signals suitable for real-world environments. The results outperform state-of-the-art approaches for classification into four classes, namely High Valence—High Arousal, High Valence—Low Arousal, Low Valence—High Arousal, and Low Valence—Low Arousal. Emotion elicitation average accuracy of 98.73% is achieved with ECG right-channel modality, 76.65% with EEG modality, and 63.67% with GSR modality for AMIGOS. The overall highest accuracy of 99.0% for the AMIGOS dataset and 90.8% for the DREAMER dataset is achieved with multi-modal fusion. A strong correlation between spectral- and hidden-layer feature analysis with classification performance suggests the efficacy of the proposed method for significant feature extraction and higher emotion elicitation performance to a broader context for less constrained environments
Original languageEnglish
Article number4551
Number of pages26
JournalSensors
Volume20
Issue number16
DOIs
Publication statusPublished - 14 Aug 2020

Keywords

  • Convolutional neural network (CNN)
  • long short-term memory (LSTM)
  • emotion recognition
  • EEG
  • ECG
  • GSR
  • deep neural network
  • physiological signals
  • Emotion recognition
  • Deep neural network
  • Physiological signals
  • Long short-term memory (LSTM)

Fingerprint Dive into the research topics of 'CNN and LSTM-Based Emotion Charting Using Physiological Signals'. Together they form a unique fingerprint.

  • Cite this