Thumbs up? Sentiment Classification using Machine …

1 Thumbs up? Sentiment Classification using Machine LearningTechniquesBo PangandLillian LeeDepartment of Computer ScienceCornell UniversityIthaca, NY 14853 VaithyanathanIBM Almaden Research Center650 Harry Jose, CA 95120 consider the problem of classifying doc-uments not by topic, but by overall senti-ment, , determining whether a reviewis positive or negative. using movie re-views as data, we find that standard ma-chine learning techniques definitively out-perform human-produced baselines. How-ever, the three Machine learning methodswe employed (Naive Bayes, maximum en-tropy classification, and support vector ma-chines) do not perform as well on sentimentclassification as on traditional topic-basedcategorization. We conclude by examiningfactors that make the Sentiment classifica-tion problem more info:Proceedings of EMNLP2002, pp. 79 IntroductionToday, very large amounts of information are avail-able in on-line documents.

2 As part of the effort tobetter organize this information for users, researchershave been actively investigating the problem of au-tomatic text bulk of such work has focused ontopicalcat-egorization, attempting to sort documents accord-ing to their subject matter ( , sports vs. poli-tics). However, recent years have seen rapid growthin on-line discussion groups and review sites ( ,the New York Times Books web page) where a cru-cial characteristic of the posted articles is theirsenti-ment, or overall opinion towards the subject matter for example, whether a product review is pos-itive or negative. Labeling these articles with theirsentiment would provide succinct summaries to read-ers; indeed, these labels are part of the appeal andvalue-add of such sites as ,which both labels movie reviews that do not con-tain explicit rating indicators and normalizes thedifferent rating schemes that individual reviewersuse.

3 Sentiment classification would also be helpful inbusiness intelligence applications ( MindfulEye sLexant system1) and recommender systems ( ,Terveen et al. (1997), Tatemura (2000)), where userinput and feedback could be quickly summarized; in-deed, in general, free-form survey responses given innatural language format could be processed usingsentiment categorization. Moreover, there are alsopotential applications to message filtering; for exam-ple, one might be able to use Sentiment informationto recognize and discard flames (Spertus, 1997).In this paper, we examine the effectiveness of ap-plying Machine learning techniques to the sentimentclassification problem. A challenging aspect of thisproblem that seems to distinguish it from traditionaltopic-based classification is that while topics are of-ten identifiable by keywords alone, Sentiment can beexpressed in a more subtle manner.

4 For example, thesentence How could anyone sit through this movie? contains no single word that is obviously negative.(See Section 7 for more examples). Thus, sentimentseems to require moreunderstandingthan the usualtopic-based classification. So, apart from presentingour results obtained via Machine learning techniques,we also analyze the problem to gain a better under-standing of how difficult it Previous WorkThis section briefly surveys previous work on non-topic-based text area of research concentrates on classifyingdocuments according to theirsourceorsource style,with statistically-detected stylistic variation (Biber,1988) serving as an important cue. Examples in-clude author, publisher ( , theNew York Daily News), native-language background, and1 brow ( , high-brow vs. popular , or low-brow)(Mosteller and Wallace, 1984; Argamon-Engelson etal., 1998; Tomokiyo and Jones, 2001; Kessler et al.)

5 ,1997).Another, more related area of research is that ofdetermining thegenreof texts; subjective genres,such as editorial , are often one of the possiblecategories (Karlgren and Cutting, 1994; Kessler etal., 1997; Finn et al., 2002). Other work explicitlyattempts to find features indicating that subjectivelanguage is being used (Hatzivassiloglou and Wiebe,2000; Wiebe et al., 2001). But, while techniques forgenre categorization and subjectivity detection canhelp usrecognizedocuments that express an opin-ion, they do not address our specificclassificationtask of determining what that opinion actually previous research on Sentiment -based classi-fication has been at least partially of this work focuses on classifying the semanticorientation of individual words or phrases, using lin-guistic heuristics or a pre-selected set of seed words(Hatzivassiloglou and McKeown, 1997; Turney andLittman, 2002).

6 Past work on Sentiment -based cat-egorization of entire documents has often involvedeither the use of models inspired by cognitive lin-guistics (Hearst, 1992; Sack, 1994) or the manual orsemi-manual construction of discriminant-word lex-icons (Huettner and Subasic, 2000; Das and Chen,2001; Tong, 2001). Interestingly, our baseline exper-iments, described in Section 4, show that humansmay not always have the best intuition for choosingdiscriminating s (2002) work on classification of reviewsis perhaps the closest to applied a spe-cific unsupervised learning technique based on themutual information between document phrases andthe words excellent and poor , where the mu-tual information is computed using statistics gath-ered by a search engine. In contrast, we utilize sev-eral completely prior-knowledge-free supervised ma-chine learning methods, with the goal of understand-ing the inherent difficulty of the The Movie-Review DomainFor our experiments, we chose to work with moviereviews.

7 This domain is experimentally convenientbecause there are large on-line collections of such re-views, and because reviewers often summarize theiroverall Sentiment with a Machine -extractablerat-ingindicator, such as a number of stars; hence, wedid not need to hand-label the data for supervised2 Indeed, although our choice of title was completelyindependent of his, our selections were eerily or evaluation purposes. We also note thatTurney (2002) found movie reviews to be the mostdifficult of several domains for Sentiment classifica-tion, reporting an accuracy of on a 120-document set (random-choice performance: 50%).But we stress that the Machine learning methods andfeatures we use are not specific to movie reviews, andshould be easily applicable to other domains as longas sufficient training data data source was the Internet Movie Database(IMDb) archive of selected only reviews where the au-thor rating was expressed either with stars or somenumerical value (other conventions varied too widelyto allow for automatic processing).

8 Ratings wereautomatically extracted and converted into one ofthree categories: positive, negative, or neutral. Forthe work described in this paper, we concentratedonly on discriminating between positive and nega-tive Sentiment . To avoid domination of the corpusby a small number of prolific reviewers, we imposeda limit of fewer than 20 reviews per author per sen-timent category, yielding a corpus of 752 negativeand 1301 positive reviews, with a total of 144 re-viewers represented. This dataset will be availableon-line (the URL contains hyphens onlyaround the word review ).4 A Closer Look At the ProblemIntuitions seem to differ as to the difficulty of the sen-timent detection problem. An expert on using ma-chine learning for text categorization predicted rela-tively low performance for automatic methods. Onthe other hand, it seems that distinguishing positivefrom negative reviews is relatively easy for humans,especially in comparison to the standard text catego-rization problem, where topics can be closely might also suspect that there are certain wordspeople tend to use to express strong sentiments, sothat it might suffice to simply produce a list of suchwords by introspection and rely on them alone toclassify the test this latter hypothesis, we asked two gradu-ate students in computer science to (independently)choose good indicator words for positive and nega-tive sentiments in movie reviews.

9 Their selections,shown in Figure 1, seem intuitively plausible. Wethen converted their responses into simple decisionprocedures that essentially count the number of theproposed positive and negative words in a given doc-ument. We applied these procedures to uniformly-3 word listsAccuracyTiesHuman 1positive:dazzling, brilliant, phenomenal, excellent, fantastic58%75%negative:suck, terrible, awful, unwatchable, hideousHuman 2positive:gripping, mesmerizing, riveting, spectacular, cool,64%39%awesome, thrilling, badass, excellent, moving, excitingnegative:bad, cliched, sucks, boring, stupid, slowFigure 1: Baseline results for human word lists. Data: 700 positive and 700 negative word listsAccuracyTiesHuman 3 + statspositive:love, wonderful, best, great, superb, still, beautiful69%16%negative:bad, worst, stupid, waste, boring, ?, !Figure 2: Results for baseline using introspection and simple statistics of the data (includingtestdata).

10 Distributed data, so that the random-choice baselineresult would be 50%. As shown in Figure 1, theaccuracy percentage of documents classified cor-rectly for the human-based classifiers were 58%and 64%, that the tie rates percentage of documents where the two sentimentswere rated equally likely are quite high5(we chosea tie breaking policy that maximized the accuracy ofthe baselines).While the tie rates suggest that the brevity ofthe human-produced lists is a factor in the relativelypoor performance results, it is not the case that sizealone necessarily limits accuracy. Based on a verypreliminary examination of frequency counts in theentire corpus (includingtestdata) plus introspection,we created a list of seven positive and seven negativewords (including punctuation), shown in Figure that figure indicates, using these words raised theaccuracy to 69%. Also, although this third list is ofcomparable length to the other two, it has a muchlower tie rate of 16%.