Sentribute: Image Sentiment Analysis from a Mid …

1 sentribute : Image Sentiment Analysis from a Mid-levelPerspectiveJianbo YuanDepartment of Electrical & ComputerEngineeringUniversity of RochesterRochester, NY YouDepartment of Computer ScienceUniversity of RochesterRochester, NY McdonoughDepartment of Electrical & ComputerEngineeringUniversity of RochesterRochester, NY LuoDepartment of Computer ScienceUniversity of RochesterRochester, NY content Analysis has always been important yet chal-lenging. Thanks to the popularity of social networks, im-ages become an convenient carrier for information di usionamong online users. To understand the di usion patternsand di erent aspects of the social images, we need to inter-pret the images first. Similar to textual content, images alsocarry di erent levels of Sentiment to their viewers. However,di erent from text, where Sentiment Analysis can use easilyaccessible semantic and context information, how to extractand interpret the Sentiment of an Image remains quite chal-lenging.

2 In this paper, we propose an Image Sentiment pre-diction framework, which leverages the mid-level attributesof an Image to predict its Sentiment . This makes the senti-ment classification results more interpretable than directlyusing the low-level features of an Image . To obtain a bet-ter performance on images containing faces, we introduceeigenface-based facial expression detection as an additionalmid-level attributes. An empirical study of the proposedframework shows improved performance in terms of predic-tion accuracy. More importantly, by inspecting the predic-tion results, we are able to discover interesting relationshipsbetween mid-level attribute and Image and Subject [Database management]: Database Applications; [Information Storage and Retrieval]: ContentAnalysis and Retrieval; [Pattern Recognition]: Ap-plicationsPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page.

3 To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a 13, August 11 2013, Chicago, USAC opyright 2013 ACM 978-1-4503-2332-1/13/08 ..$ TermsAlgorithms, Experimentation, ApplicationKeywordsImage Sentiment , Analysis , Mid-level Attributes, Visual Con-tent1. INTRODUCTIONN owadays, social networks such as Twitter and microblogsuch as Weibo become major platforms of information ex-change and communication between users, between whichthe common information carrier is tweets. A recent studyshows that images constitute about 36 percent of all theshared links on Twitter1, which makes visual data miningan interesting and active area to explore. As an old sayinghas it, an Image is worth a thousand words. Much alike tex-tual content based mining approach, extensive studies havebeen done regarding aesthetics and emotions in images [3,8, 28].

4 In this paper, we are focusing on Sentiment analysisbased on visual information far Analysis of textual information has been well de-veloped in areas including opinion mining [18, 20], humandecision making [20], brand monitoring [9], stock marketprediction [1], political voting forecasts [18, 25] and intelli-gence gathering [31]. Figure 1 shows and example of imagetweets. In contrast, Analysis of visual information coversareas such as Image information retrieval [4, 33], aestheticsgrading [15] and the progress is relatively networks such as Twitter and microblogs such asWeibo provide billions of pieces of both textual and visual in-formation, making it possible to detect Sentiment indicatedby both textual and visual data respectively. However, sen-timent Analysis based on a visual perspective is still in itsinfancy. With respect to Sentiment Analysis , much work hasbeen done on textual information based Sentiment Analysis [18, 20, 29], as well as online Sentiment dictionary [5, 24].

5 1 #. 1: Selected images crawled from Twitter showing(left column) positive Sentiment and (right column) and concept learning approaches [6, 19, 16, 22]based on visual features is another way of Sentiment analysiswithout employing textual information. However, semanticsand concept learning approaches are hampered by the limi-tations of object classifier accuracy The Analysis of aesthetics[3, 15], interestingness [8] and a ect or emotions [10, 14, 17,32] of images are most related to Sentiment Analysis basedon visual content. Aiming to conduct visual content basedsentiment Analysis , current approaches includes employinglow-level features [10, 11, 12], via facial expression detec-tion [27] and user intent [7]. Sentiment Analysis approachesbased on low-level features has the limitation of low inter-pretability, which in turn makes it undesirable for high-leveluse.

6 Metadata of images is another source of information forhigh-level feature learning [2]. However, not all images con-tain such kind of data. Therefore, we proposed sentribute ,an Image Sentiment Analysis algorithm based on to the state-of-the-art algorithms, our maincontribution to this area is two-fold: first, we propose Sen-tribute, an Image - Sentiment Analysis algorithm based on 102mid-level attributes, of which results are easier to interpretand ready-to-use for high-level understanding. Second, weintroduce eigenface to facial Sentiment recognition as a so-lution for Sentiment Analysis on images containing is simple but powerful, especially in cases of extremefacial expressions, and contributed an 18% gain in accuracyover decision making only based on mid-level attributes, and30% over the state of art methods based on low level remainder of this paper is organized as follows: in Sec-tion 2, we present an overview of our proposed Sentributeframework.

7 Section 3 provides details for sentribute , includ-ing low-level feature extraction, mid-level attribute gener-ation, Image Sentiment prediction, and decision correctionbased on facial Sentiment recognition. Then in Section 4,we test our algorithm on 810 images crawled from Twitterand make a comparison with the state of the art method,which makes prediction based on low-level features and tex-tual information only. Finally, we summarize our findingsand possible future extensions of our current work in FRAMEWORK OVERVIEWF igure 2 presents our proposed sentribute framework. Theidea for this algorithm is as follows: first of all, we extractscene descriptor low-level features from the SUN Database[7] and use these four features to train our classifiers by Lib-linear [10] for generating 102 predefined mid-level attributes,and then use these attributes to predict sentiments.

8 Mean-while, facial sentiments are predicted using eigenfaces. Thismethod generates really good results especially in cases ofpredicting strong positive and negative sentiments, whichmakes it possible to combine these two predictions and gen-erate a better result for predicting Image sentiments withfaces. To illustrate how facial Sentiment help refine ourprediction based on only mid-level attributes, we presentan example in Section 4, of how to correct our false posi-tive/negative prediction based on facial Sentiment SENTRIBUTEIn this section we outline the design and construction ofthe proposed sentribute , a novel Image Sentiment predictionmethod based on mid-level attributes, together with a de-cision refine mechanism for images containing people. Forimage Sentiment Analysis , we conclude the procedure start-ing from dataset introduction, low-level feature selection,building mid-level attribute classifier, Image Sentiment pre-diction.

9 As for facial Sentiment recognition, we introduceeigenface to fulfill our DatasetOur proposed algorithm mainly contains three steps: firstis to generate mid-level attributes labels. For this part, wetrain our classifier using SUN Database2, the first large-scale scene attribute database, initially designed for high-level scene understanding and fine-grained scene recognition[21]. This database includes more than 800 categories and14,340 images, as well as discriminative attributes labeledby crowd-sourced human studies. Attributes labels are pre-sented in form of zero to three votes, of which 0 vote meansthis Image is the least correlated with this attribute, andthree votes means the most correlated. Due to this votingmechanism, we have an option of selecting which set of im-ages to be labeled as positive: images with more than onevote, introduced as soft decision (SD), or images with morethan two votes, introduced as hard decision (HD).

10 Second step of our algorithm is to train Sentiment pre-dicting classifiers with images crawled from Twitter together2 ContentsExtract Low-level FeaturesFace DetectionFacial Expression DetectionGenerate 102 Mid-level AttributesSentiment Prediction(decision fusion)Four Scene DescriptorsSentribute: Algorithm FrameworkEigenface modelImages contain FacesTraining ClassifiersAsymmetric BaggingFigure 2: Selected images crawled from Twitter showing (a) positive Sentiment and (b) negative their textual data covering more than 800 images. Twit-ter is currently one of the most popular microblog ground truth is obtained from visual sentimentontology3with permission of the authors. The dataset in-cludes 1340 positive, 223 negative and 552 neutral imagetweets. For testing, we randomly select 810 images, onlycontaining positive (660 tweets) and negative (150 tweets).