tidytext: Text Mining and Analysis Using Tidy Data ...

1 tidytext : Text Mining and Analysis Using tidy DataPrinciples in RJulia Silge1and David Robinson21 Datassist2 Stack Review Repository ArchiveLicenceAuthors of JOSS papers retaincopyright and release the work un-der a Creative Commons Attri-bution International License(CC-BY).SummaryThe tidytext package (Silge, Robinson, and Hester 2016) is an R package (R Core Team2016) for text Mining Using tidy data principles . As described by Hadley Wickham (Wick-ham 2014), tidy data has a specific structure: each variable is a column each observation is a row each type of observational unit is a tableTidy data sets allow manipulation with a standard set of tidy tools, including popu-lar packages such as dplyr (Wickham, Francois, and RStudio 2015), ggplot2 (Wickham,Chang, and RStudio 2016), and broom (Robinson et al.)

2 2015). These tools do not yet,however, have the infrastructure to work fluently with text data and natural languageprocessing tools. In developing this package, we provide functions and supporting datasets to allow conversion of text to and from tidy formats, and to switch seamlessly betweentidy tools and existing text Mining define the tidy text format as being one-token-per-document-per-row, and providefunctionality to tokenize by commonly used units of text including words, n-grams, andsentences. At the same time, the tidytext package doesn t expect a user to keep textdata in a tidy form at all times during an Analysis . The package includes functions totidyobjects (see the broom package (Robinson et al.)

3 2015)) from popular text miningR packages such as tm (Ingo Feinerer and Meyer 2008) and quanteda (Benoit and Nulty2016). This allows, for example, a workflow with easy reading, filtering, and processingto be done Using dplyr and other tidy tools, after which the data can be converted intoa document-term matrix for machine learning applications. The models can then bere-converted into a tidy form for interpretation and visualization with following is an example visualization made Using tidytext s text Mining and sentimentanalysis , Kenneth, and Paul Nulty. : Quantitative Analysis of Feinerer, Kurt Hornik, and David Meyer. 2008. Text Mining Infrastructure in R.

4 Journal of Statistical Software25 (5): 1 Core Team. : A Language and Environment for Statistical Computing. Vienna,Austria: R Foundation for Statistical , David, Matthieu Gomez, Boris Demeshev, Dieter Menne, Benjamin Nutter,Silge et al., (2016). tidytext : Text Mining and Analysis Using tidy data principles in of Open Source Software, 1(3), 37, 1:Sentiment in Jane Austen s NovelsSilge et al., (2016). tidytext : Text Mining and Analysis Using tidy data principles in of Open Source Software, 1(3), 37, Johnston, Ben Bolker, Francois Briatte, and Hadley Wickham. : Con-vert Statistical Analysis Objects into tidy data , Julia, David Robinson, and Jim Hester.

5 2016. tidytext : Text Mining Using Dplyr,Ggplot2, and Other tidy Tools. , Hadley. 2014. tidy data . Journal of Statistical Software59 (1): 1 , Hadley, Winston Chang, and RStudio. : An Implementation ofthe Grammar of , Hadley, Romain Francois, and RStudio. : A Grammar of et al., (2016). tidytext : Text Mining and Analysis Using tidy data principles in of Open Source Software, 1(3), 37.