Sentiment Analysis of Twitter Data - Columbia University

1 Sentiment Analysis of Twitter DataApoorv Agarwal Boyi Xie Ilia Vovsha Owen Rambow Rebecca PassonneauDepartment of Computer ScienceColumbia UniversityNew York, NY 10027 USA{apoorv@cs, xie@cs, iv2121@, rambow@ccls, examine Sentiment Analysis on Twitterdata. The contributions of this paper are: (1)We introduce POS-specific prior polarity fea-tures. (2) We explore the use of a tree kernel toobviate the need for tedious feature engineer-ing. The new features (in conjunction withpreviously proposed features) and the tree ker-nel perform approximately at the same level,both outperforming the state-of-the-art IntroductionMicroblogging websites have evolved to become asource of varied kind of information.}

2 This is due tonature of microblogs on which people post real timemessages about their opinions on a variety of topics,discuss current issues, complain, and express posi-tive Sentiment for products they use in daily life. Infact, companies manufacturing such products havestarted to poll these microblogs to get a sense of gen-eral Sentiment for their product. Many times thesecompanies study user reactions and reply to users onmicroblogs. One challenge is to build technology todetect and summarize an overall this paper, we look at one such popular mi-croblog called Twitter and build models for classify-ing tweets into positive, negative and neutral senti-ment. We build models for two classification tasks:a binary task of classifying Sentiment into positiveand negative classes and a 3-way task of classi-fying Sentiment into positive, negative and neutralclasses.

3 We experiment with three types of models:unigram model, a feature based model and a treekernel based model. For the feature based modelwe use some of the features proposed in past liter-ature and propose new features. For the tree ker-nel based model we design a new tree representa-tion for tweets. We use a unigram model, previouslyshown to work well for Sentiment Analysis for Twit-ter data, as our baseline. Our experiments show thata unigram model is indeed a hard baseline achievingover 20% over the chance baseline for both classifi-cation tasks. Our feature based model that uses only100 features achieves similar accuracy as the uni-gram model that uses over 10,000 features.

4 Our treekernel based model outperforms both these modelsby a significant margin. We also experiment witha combination of models: combining unigrams withour features and combining our features with the treekernel. Both these combinations outperform the un-igram baseline by over 4% for both classificationtasks. In this paper, we present extensive featureanalysis of the 100 features we propose. Our ex-periments show that features that have to do withTwitter-specific features (emoticons, hashtags etc.)add value to the classifier but only marginally. Fea-tures that combine prior polarity of words with theirparts-of-speech tags are most important for both theclassification tasks.

5 Thus, we see that standard nat-ural language processing tools are useful even ina genre which is quite different from the genre onwhich they were trained (newswire). Furthermore,we also show that the tree kernel model performsroughly as well as the best feature based models,even though it does not require detailed feature use manually annotated Twitter data for ourexperiments. One advantage of this data, over pre-viously used data-sets, is that the tweets are col-lected in a streaming fashion and therefore representa true sample of actual tweets in terms of languageuse and content. Our new data set is available toother researchers. In this paper we also introducetwo resources which are available (contact the firstauthor): 1) a hand annotated dictionary for emoti-cons that maps emoticons to their polarity and 2)an acronym dictionary collected from the web withEnglish translations of over 5000 frequently rest of the paper is organized as follows.

6 Insection 2, we discuss classification tasks like sen-timent Analysis on micro-blog data. In section 3,we give details about the data. In section 4 we dis-cuss our pre-processing technique and additional re-sources. In section 5 we present our prior polarityscoring scheme. In section 6 we present the designof our tree kernel. In section 7 we give details of ourfeature based approach. In section 8 we present ourexperiments and discuss the results. We concludeand give future directions of research in section Literature SurveySentiment Analysis has been handled as a NaturalLanguage Processing task at many levels of gran-ularity. Starting from being a document level classi-fication task (Turney, 2002; Pang and Lee, 2004), ithas been handled at the sentence level (Hu and Liu,2004; Kim and Hovy, 2004) and more recently atthe phrase level (Wilson et al.)

7 , 2005; Agarwal et al.,2009).Microblog data like Twitter , on which users postreal time reactions to and opinions about every-thing , poses newer and different challenges. Someof the early and recent results on Sentiment analysisof Twitter data are by Go et al. (2009), (Berminghamand Smeaton, 2010) and Pak and Paroubek (2010).Go et al. (2009) use distant learning to acquire senti-ment data. They use tweets ending in positive emoti-cons like :) :-) as positive and negative emoti-cons like :( :-( as negative. They build mod-els using Naive Bayes, MaxEnt and Support Vec-tor Machines (SVM), and they report SVM outper-forms other classifiers. In terms of feature space,they try a Unigram, Bigram model in conjunctionwith parts-of-speech (POS) features.

8 They note thatthe unigram model outperforms all other , bigrams and POS features do not and Paroubek (2010) collect data following asimilar distant learning paradigm. They perform adifferent classification task though: subjective ver-sus objective. For subjective data they collect thetweets ending with emoticons in the same manneras Go et al. (2009). For objective data they crawltwitter accounts of popular newspapers like NewYork Times , Washington Posts etc. They re-port that POS and bigrams both help (contrary toresults presented by Go et al. (2009)). Both theseapproaches, however, are primarily based on ngrammodels. Moreover, the data they use for training andtesting is collected by search queries and is thereforebiased.

9 In contrast, we present features that achievea significant gain over a unigram baseline. In addi-tion we explore a different method of data represen-tation and report significant improvement over theunigram models. Another contribution of this paperis that we report results on manually annotated datathat does not suffer from any known biases. Ourdata is a random sample of streaming tweets unlikedata collected by using specific queries. The sizeof our hand-labeled data allows us to perform cross-validation experiments and check for the variance inperformance of the classifier across significant effort for Sentiment classifica-tion on Twitter data is by Barbosa and Feng (2010).

10 They use polarity predictions from three websites asnoisy labels to train a model and use 1000 manuallylabeled tweets for tuning and another 1000 manu-ally labeled tweets for testing. They however donot mention how they collect their test data. Theypropose the use of syntax features of tweets likeretweet, hashtags, link, punctuation and exclamationmarks in conjunction with features like prior polar-ity of words and POS of words. We extend theirapproach by using real valued prior polarity, and bycombining prior polarity with POS. Our results showthat the features that enhance the performance of ourclassifiers the most are features that combine priorpolarity of words with their parts of speech.