Syntactic Annotations for the Google Books NGram Corpus

1 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 169 174,Jeju, Republic of Korea, 8-14 July 2012 Association for Computational LinguisticsSyntactic Annotations for the Google Books NGram CorpusYuri Lin, Jean-Baptiste Michel, Erez Lieberman Aiden,Jon Orwant, Will Brockman and Slav Petrov Google present a new edition of the Google BooksNgram Corpus , which describes how oftenwords and phrases were used over a periodof five centuries, in eight languages; it reflects6% of all Books ever published. This new edi-tion introduces Syntactic Annotations : wordsare tagged with their part-of-speech, and head-modifier relationships are recorded.

2 The an-notations are produced automatically with sta-tistical models that are specifically adapted tohistorical text. The Corpus will facilitate thestudy of linguistic trends, especially those re-lated to the evolution of IntroductionThe Google Books NGram Corpus (Michel et al.,2011) has enabled the quantitative analysis of lin-guistic and cultural trends as reflected in millionsof Books written over the past five centuries. Thecorpus consists of words and phrases ( , ngrams)and their usage frequency over time. The data isavailable for download, and can also be viewedthrough the interactive Google Books NGram sheer quantity of and broad historical scopeof the data has enabled a wide range of analyses(Michel et al.)

3 , 2011; Ravallion, 2011). Of course,examining raw NGram frequencies is of limited util-ity when studying many aspects of linguistic change,particularly the ones related to syntax. For instance,most English verbs are regular (their past tense isformed by adding -ed), and the few exceptions,known as irregular verbs, tend to regularize over the Corresponding Frequencyburntburnt_VERB burnt_ADJburnedburned_VERB burned_ADJF igure 1: Usage frequencies ofburnedandburntovertime, showing thatburnedbecame the dominant spellingaround 1880. Our new Syntactic Annotations enable amore refined analysis, suggesting that the crossing-pointfor the verb usage ( ) wasdecades (Lieberman et al.

4 , 2007). Figure 1 illus-trates howburnedgradually overtookburnt, becom-ing more frequent around 1880. Unfortunately, as astudy of verb regularization, this analysis is skewedby a significant confound: both words can serveas either verbs ( ,the house burnt) or adjectives( ,the burnt toast). Because many words havemultiple Syntactic interpretations, such confoundsoften limit the utility of raw NGram frequency this work we provide a new edition of theGoogle Books NGram Corpus that contains over 8million Books , or 6% of all Books ever published ( 3). Moreover, we include Syntactic anal-ysis in order to facilitate a fine-grained analysis ofthe evolution of syntax.

5 Ngrams are annotated withpart-of-speech tags ( , in the phrasehe burnt thetoast,burntis a verb; inthe burnt toast,burntis anadjective) and head-modifier dependencies ( , inthe phrasethe little black book,littlemodifiesbook).The annotated ngrams are far more useful for ex-169amining the evolution of grammar and syntax. Forour study of the regularization of the verbburn,the availability of Syntactic Annotations resolves theverb vs. adjective ambiguity in the original data, al-lowing us to only examine instances whereburntandburnedappear as verbs. This more refined anal-ysis suggests a crossover date for the frequency ofthe verb forms that is several decades earlier thanthe overall (verbs and adjectives) use state-of-the-art statistical part-of-speechtaggers and dependency parsers to produce syntac-tic Annotations for eight languages in the GoogleBooks collection.

6 The Annotations consist of 12 lan-guage universal part-of-speech tags and unlabeledhead-modifier dependencies. Section 4 describes themodels that we used and the format of the annota-tions in detail. We assess the expected annotationaccuracies experimentally and discuss how we adaptthe taggers and parsers to historical text in Section annotated ngrams are available as a new editionof the Google Books NGram Corpus ; we providesome examples from the new Corpus in Figure Related WorkMichel et al. (2011) described the construction ofthe first edition of the Google Books NGram Corpusand used it to quantitatively analyze a variety of top-ics ranging from language growth to public related NGram Viewer has become a populartool for examining language trends by experts andnon-experts addition to studying frequency patterns in thedata, researchers have also attempted to analyze thegrammatical function of the ngrams (Davies, 2011).

7 Such endeavors are hampered by the fact that theNgram Corpus provides only aggregate statistics inthe form of NGram counts and not the full sen-tences. Furthermore, only ngrams that pass certainoccurrence thresholds are publicly available, makingany further aggregation attempt futile: in heavy taildistributions like the ones common in natural lan-guages, the counts of rare events (that do not passthe frequency threshold) can have a large cumula-tive contrast, because we have access to the fulltext, we can annotate ngrams to reflect the particu-lar grammatical functions they take in the sentencesLanguage#Volumes#TokensEnglish4 ,541,627468,491,999,592 Spanish854,64983,967,471,303 French792,118102,174,681,393 German657,99164,784,628,286 Russian591,31067,137,666,353 Italian305,76340,288,810,817 Chinese302,65226,859,461,025 Hebrew70,6368,172,543,728 Table 1.

8 Number of volumes and tokens for each lan-guage in our Corpus . The total collection contains morethan 6% of all Books ever were extracted from, and can also account forthe contribution of rare ngrams to otherwise frequentgrammatical NGram CorpusThe Google Books NGram Corpus has been avail-able 2010. This work presents new corpora thathave been extracted from an even larger book collec-tion, adds a new language (Italian), and introducessyntactically annotated ngrams. The new corporaare available in addition to the already existing Books DataThe new edition of the NGram Corpus supports theeight languages shown in Table 1.

9 The book vol-umes were selected from the larger collection of allbooks digitized at Google following exactly the pro-cedure described in Michel et al. (2011). The newedition contains data from 8,116,746 Books , or over6% of all Books ever published. The English cor-pus alone comprises close to half a trillion collection of Books is much larger than anyother digitized collection; its generation required asubstantial effort involving obtaining and manuallyscanning millions of Raw NgramsWe extract ngrams in a similar way to the first edi-tion of the Corpus (Michel et al., 2011), but withsome notable differences. Previously, tokenizationwas done on whitespace characters and all ngramsoccurring on a given page were extracted, includ-ing ones that span sentence boundaries, but omitting170 TagEnglishSpanishFrenchGermanRussian1 ItalianChineseHebrewADJother, such mayor, gran tous, m eme anderen, erstenvse, to stesso, grande , !

10 , !ADPof, inde, ende,`ain, vonv, nadi, in , !, !ADVnot, whenno, m asne, plusauch, sotak, boleenon, pi u , !, !CONJand, ory, queet, queund, da i, qtoche, ed , !, !DETthe, ala, ella, lesder, die-la, il , !NOUN time, people parte, a nos temps, partieZeit, Jahreego, onparte, tempo , !, !PRONit, Ique, sequi, ilsich, die-che, si , !, !VERBis, wases, haest, sontist, werdenbylo, byl e, sono , N!, !Table 2: The two most common words for some POS tags in the new Google Books NGram Corpus for all that span page , we perform tokenization and sentenceboundary detection by applying a set of manuallydevised rules (except for Chinese, where a statisticalsystem is used for segmentation).