Multi-task semi-supervised adversarial autoencoding for speech emotion recognition

Latif, Siddique and Rana, Rajib ORCID: and Khalifa, Sara and Jurdak, Raja and Epps, Julien and Schuller, Bjorn W. (2020) Multi-task semi-supervised adversarial autoencoding for speech emotion recognition. IEEE Transactions on Affective Computing.


Inspite the emerging importance of Speech Emotion Recognition (SER), the state-of-the-art accuracy is quite low and needs improvement to make commercial applications of SER viable. A key underlying reason for the low accuracy is the scarcity of emotion datasets, which is a challenge for developing any robust machine learning model in general. In this paper, we propose a solution to this problem: a multi-task learning framework that uses auxiliary tasks for which data is abundantly available. We show that utilisation of this additional data can improve the primary task of SER for which only limited labelled data is available. In particular, we use gender identifications and speaker recognition as auxiliary tasks, which allow the use of very large datasets, e.g., speaker classification datasets. To maximise the benefit of multi-task learning, we further use an adversarial autoencoder (AAE) within our framework, which has a strong capability to learn powerful and discriminative features. Furthermore, the unsupervised AAE in combination with the supervised classification networks enables semi-supervised learning which incorporates a discriminative component in the AAE unsupervised training pipeline. The proposed model is rigorously evaluated for categorical and dimensional emotion, and cross-corpus scenarios. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on two publicly available dataset.

Statistics for USQ ePrint 38567
Statistics for this ePrint Item
Item Type: Article (Commonwealth Reporting Category C)
Refereed: Yes
Item Status: Live Archive
Additional Information: Published online: 1 April 2020. Permanent restricted access to ArticleFirst version, in accordance with the copyright policy of the publisher.
Faculty/School / Institute/Centre: Current - Institute for Resilient Regions
Faculty/School / Institute/Centre: Current - Institute for Resilient Regions
Date Deposited: 03 Apr 2020 06:51
Last Modified: 08 Jun 2021 00:29
Uncontrolled Keywords: speech emotion recognition, multi task learning, representation learning
Fields of Research (2008): 08 Information and Computing Sciences > 0801 Artificial Intelligence and Image Processing > 080109 Pattern Recognition and Data Mining
Fields of Research (2020): 46 INFORMATION AND COMPUTING SCIENCES > 4602 Artificial intelligence > 460212 Speech recognition
Identification Number or DOI:

Actions (login required)

View Item Archive Repository Staff Only