Certificate for Best Paper Award of the 14th International Conference on Social Computing and Social Media
The award has been conferred to
Leon Koren, Tomislav Stipancic, Andrija Ricko and Luka Orsag
(University of Zagreb, Croatia)
for the paper entitled
"Multimodal Emotion Analysis based on Visual, Acoustic and Linguistic Features"
Presented in the context of
HCI International 2022
26 June—1 July 2022
"In this paper, a computational reasoning framework that can interpret social signals of the person in interaction by focusing on the person’s emotional state is presented. Two distinct sources of social signals are used for this study: facial and voice emotion modalities. As a part of the first modality, a Convolutional Neural Network (CNN) is used to extract and process the facial features based on live stream video. The voice emotion analysis containing two sub-modalities is driven by CNN and Long Short-Term Memory (LSTM) networks. The networks are analyzing the acoustic and linguistic features of the voice to determine the possible emotional cues of the person in interaction. Relying on the multimodal information fusion, the system then fuses data into a single hypothesis. Results of such reasoning are used to autonomously generate the robot responses which are shown in a form of non-verbal facial animations projected on the ‘face’ surface of the affective robot head PLEA. Built-in functionalities of the robot can provide a degree of situational embodiment, self-explainability and context-driven interaction."
The full paper is available through SpringerLink, provided that you have proper access rights.