Featurizing Text: Converting Text into Predictors for ...

1 Featurizing Text: Converting Text into Predictors forRegression AnalysisDean P. Foster Mark Liberman Robert A. Stine Department of StatisticsThe Wharton School of the University of PennsylvaniaPhiladelphia, PA 19104-6340 October 18, 2013 AbstractModern data streams routinely combine text with the familiar numerical data usedin regression analysis . For example, listings for real estate that show the price of aproperty typically include a verbal description. Some descriptions include numericaldata, such as the number of rooms or the size of the home.

2 Many others, however, onlyverbally describe the property, often using an idiosyncratic vernacular. For modelingsuch data, we describe several methods that that convert such text into numericalfeatures suitable for regression analysis . The proposed Featurizing techniques createregressors directly from text, requiring minimal user input. The techniques rangenaive to subtle. One can simply use raw counts of words, obtain principal componentsfrom these counts, or build regressors from counts of adjacent words.

3 Our examplethat models real estate prices illustrates the surprising success of these methods. Topartially explain this success, we offer a motivating probabilistic model. Because thederived regressors are difficult to interpret, we further show how the presence of partialquantitative features extracted from text can elucidate the structure of a Phrases: sentiment analysis , n-gram, latent semantic analysis , text mining Research supported by NSF grant 11067431 Featurizing Text (DRAFT, October 18, 2013)21 IntroductionModern data streams routinely combine text with numerical data suitable for in regres-sion analysis .

4 For example, patient medical records combine lab measurements withphysician comments and online product ratings such as those at Amazon or Netflixblend explicit characteristics with verbal commentary. As a specific example, we builda regression model to predict the price of real estate from its listing. The listings weuse are verbal rather than numerical data obtained by filling out a spreadsheet-likeform. Here are four such listings for Chicago, IL, extracted (with permission) on June 12, 2013:$399000 Stunning skyline views like something from a postcard are yourswith this large 2 bed, 2 bath loft in Dearborn Tower!

5 Detailedhrdwd floors throughout the unit compliment an open kitchen andspacious living-room and dining-room /w walk-in closet, steamshower and marble entry. Parking available.$13000 4 bedroom, 2 bath 2 story frame home. Property features alarge kitchen, living-room and a full basement. This is a Fannie MaeHomepath property.$65000 Great short sale Brick 2 flat with 3 bdrmeach unit. 4 or more cars parking. Easy to show.$29900 This 3 flat with all 3 bed units is truly a greatinvestment!! This property also comes with a full attic that hasthe potential of a build-out-thats a possible 4 unit building in agreat area!

6 ! Blocks from lake and transportation. Looking for adeal in todays market - here is the one!!!The only numerical data common to the listings is the price that appears at the head ofeach listing. Some listings include further numerical data, such as the number of roomsor occasionally the size of the property (number of square feet). Many listings, however, Featurizing Text (DRAFT, October 18, 2013)3provide only a verbal description, often written in an idiosyncratic vernacular familiaronly to those who are house hunting. Some authors write in sentences, others not,and a variety of abbreviations appear.

7 The style of punctuation varies from spartanto effusive (particularly exclamation marks), and the length of the listing runs fromseveral words to a long obvious approach to building regressors from text data relies on a substantiveanalysis of the text. For example, sentiment analysis constructs a domain-specificlexicon of positive and negative words. In the context of real estate, words such as modern and spacious might be flagged as positive indicators (and so be associatedwith more expensive properties), whereas Fannie Mae and fixer-upper would bemarked as negative indicators.

8 The development of such lexicons has been an activearea of research in sentiment analysis over the past decade (Taboada, Brooke, Tofiloski,Voli and Stede, 2011). The development of a lexicon require substantial knowledge ofthe context and the results are known to be domain specific. Each new problem requiresa new lexicon. The lexicon for pricing homes would be quite different from the lexiconfor diagnosing patient health. Our approach is also domain specific, but requires littleuser input and so can be highly contrast to substantively oriented modeling, we propose a version of supervisedsentiment analysis that converts text into conventional explanatory variables.

9 We con-vert the text into conventional numerical regressors (featurize) by exploiting methodsfrom computational linguistics that are familiar to statisticians. These so-called vectorspace models (Turney and Pantel, 2010), such as latent semantic analysis (LSA), makeuse of singular value decompositions of the bag-of-words and bigram representations oftext. (This connection leads to methods being described as a spectral algorithm for .)These representations map words into points in a vector space defined by counts. Thisapproach is highly automated with little need for human intervention, though it makesit easy to exploit such investments when available.

10 The derived regressors can be usedalone or in combination with traditional variables, such as those obtained from a lexi-con or other semantic model. We use the example of real estate listings to illustrate theimpact of various choices on the predictive accuracy. For example, a regression usingthe automated features produced by this analysis explains over two-thirds of the vari-ation in listed prices for real estate in Chicago. The addition of several substantivelyFeaturizing Text (DRAFT, October 18, 2013)4derived variables adds little. Though we do not emphasize its use here, variable selec-tion can be employed to reduce the ensemble of regressors without sacrificing emphasis on predictive accuracy does not necessarily produce an interpretablemodel, and one can use other data to create such structure.