CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset ...

1 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718 3727 July 5 - 10, 2020 Association for Computational Linguistics3718CH-SIMS: A Chinese Multimodal Sentiment Analysis Datasetwith Fine-grained Annotations of ModalityWenmeng Yu1, Hua Xu2 , Fanyang Meng, Yilin Zhu, Yixiao Ma,Jiele Wu, Jiyun Zou, Kaicheng YangState Key Laboratory of Intelligent Technology and Systems,Department of Computer Science and Technology, Tsinghua University, Beijing, ChinaBeijing National Research Center for Information Science and studies in Multimodal Sentiment anal-ysis have used limited datasets, which onlycontain unified Multimodal annotations. How-ever, the unified annotations do not alwaysreflect the independent Sentiment of singlemodalities and limit the model to capture thedifference between this pa-per, we introduce a Chinese single- and multi-modal Sentiment Analysis Dataset , CH-SIMS,which contains 2,281 refined video segmentsin the wild with both Multimodal and in-dependent unimodal allowsresearchers to study the interaction betweenmodalities or use independent unimodal anno-tations for unimodal Sentiment Analysis .

2 Fur-thermore, we propose a multi-task learningframework based on late fusion as the experiments on the CH-SIMS showthat our methods achieve state-of-the-art per-formance and learn more distinctive unimodalrepresentations. The full Dataset and codes areavailable for use IntroductionSentiment Analysis is an important research area inNatural Language Processing (NLP). It has wideapplications for other NLP tasks, such as opinionmining, dialogue generation, and user behavioranalysis. Previous study (Pang et al., 2008; Liuand Zhang, 2012) mainly focused on text sentimentanalysis and achieved impressive results. However,using text alone is not sufficient to determine thespeaker s sentimental state, and text can be mis-leading.

3 With the booming of short video applica-tions, nonverbal behaviors (vision and audio) areintroduced to solve the above shortcomings (Zadehet al., 2016; Poria et al., 2017).In Multimodal Sentiment Analysis , intra-modalrepresentation and inter-modal fusion are two im- Corresponding AuthorM: Negative M: Negative T : Positive A : Weakly Positive V : Negative (Others)(Ours) .. It is too unexpected Figure 1: An example of the annotation difference be-tween CH-SIMS and other datasets. For each mul-timodal clip, in addition to Multimodal annotations,our proposed Dataset has independent unimodal anno-tations. M: Multimodal , T: Text, A: Audio, V: and challenging subtasks (Baltru saitis et al.)

4 ,2018; Guo et al., 2019). For intra-modal represen-tation, it is essential to consider the temporal orspatial characteristics in different modalities. Themethods based on Convolutional Neural Network(CNN), Long Short-term Memory (LSTM) net-work and Deep Neural Network (DNN) are threerepresentative approaches to extract unimodal fea-tures (Cambria et al., 2017; Zadeh et al., 2017,2018a). For inter-modal fusion, numerous methodshave been proposed in recent years. For exam-ple, concatenation (Cambria et al., 2017), TensorFusion Network (TFN) (Zadeh et al., 2017), Low-rank Multimodal Fusion (LMF) (Liu et al., 2018),3719 Memory Fusion Network (MFN) (Zadeh et al.,2018a), Dynamic Fusion Graph (DFG) (Zadehet al.

5 , 2018b), and others. In this paper, we mainlyconsider late-fusion methods that perform intra-modal representation learning first and then employinter-modal fusion. An intuitive idea is that thegreater the difference between inter-modal repre-sentations, the better the complementarity of inter-modal fusion. However, it is not easy for existinglate-fusion models to learn the differences betweendifferent modalities, further limits the performanceof fusion. The reason is that the existing multi-modal Sentiment datasets only contain a unifiedmultimodal annotation for each Multimodal seg-ment, which is not always suitable for all modali-ties. In other words, all modalities share a standardannotation during intra-modal representation learn-ing.

6 Further, these unified supervisions will guideintra-modal representations to be more consistentand less validate the above Analysis , in this paper, wepropose a Chinese Multimodal Sentiment analy-sis Dataset with independent unimodal annotations,CH-SIMS. Figure 1 shows an example of the anno-tation difference between our proposed Dataset andthe other existing Multimodal datasets. SIMS has2,281 refined video clips collected from differentmovies, TV serials, and variety shows with sponta-neous expressions, various head poses, occlusions,and illuminations. The CHEAVD (Li et al., 2017)is also a Chinese Multimodal Dataset , but it onlycontains two modalities (vision and audio) and oneunified annotation.

7 In contrast, SIMS has threemodalities and unimodal annotations except formultimodal annotations for each clip. Therefore,researchers can use SIMS to do both unimodal andmultimodal Sentiment Analysis tasks. Furthermore,researchers can develop new methods for multi-modal Sentiment Analysis with these additional on SIMS, we propose a Multimodal multi-task learning framework using unimodal and mul-timodal annotations. In this framework, the uni-modal and Multimodal tasks share the feature repre-sentation sub-network in the bottom. It is suitablefor all Multimodal models based on , we introduce three late-fusion models, in-cluding TFN, LMF, and Late-Fusion DNN (LF-DNN), into our framework.

8 With unimodal tasks,the performance of Multimodal task is significantlyincreased. Furthermore, we make a detailed discus-sion on Multimodal Sentiment Analysis , unimodalsentiment Analysis and multi-task learning. Lastly,we verify that the introduction of unimodal annota-tions can effectively expand the difference betweendifferent modalities and obtain better performancein inter-modal this work, we provide a new perspective formultimodal Sentiment Analysis . Our main contribu-tions in this paper can be summarized as follows: We propose a Chinese Multimodal sentimentanalysis Dataset with more fine-grained anno-tations of modality, CH-SIMS. These addi-tional annotations make our Dataset availablefor both unimodal and Multimodal sentimentanalysis.

9 We propose a Multimodal multi-task learn-ing framework, which is suitable for all late-fusion methods in Multimodal Sentiment anal-ysis. Besides, we introduce three late-fusionmodels into this framework as strong base-lines for SIMS. The benchmark experiments on the SIMS show that our methods learn more distinctiveunimodal representations and achieve state-of-the-art Related WorkIn this section, we briefly review related work inmultimodal datasets, Multimodal Sentiment analy-sis, and multi-task Multimodal DatasetsTo meet the needs of Multimodal Sentiment anal-ysis and emotion recognition, researchers haveproposed various of Multimodal datasets, includ-ing IEMOCAP (Busso et al.)

10 , 2008), YouTube(Morency et al., 2011), MOUD (P erez-Rosas et al.,2013), ICT-MMMO (W ollmer et al., 2013), MOSI(Zadeh et al., 2016), CMU-MOSEI (Zadeh et al.,2018b) and so on. In addition, Li et al. (2017)proposed a Chinese emotional audio-visual datasetand Poria et al. (2018) proposed a multi-party emo-tional, conversational Dataset containing more thantwo speakers per dialogue. However, these existingmultimodal datasets only contain a unified multi-modal annotation for each Multimodal corpus. Incontrast, SIMS contains both unimodal and multi-modal #Total number of videos60 Total number of segments2,281- Male1,500- Female781 Total number of distinct speakers474 Average length of segments (s) word count per segments15 Table 1: Statistics of SIMS Multimodal Sentiment AnalysisMultimodal Sentiment Analysis has become a majorresearch topic that integrates verbal and nonver-bal behaviors.