CHAPTER Vector Semantics and Embeddings

1 Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright 2021. Allrights reserved. Draft of December 29, Semantics andEmbeddings Nets are for fish;Once you get the fish, you can forget the net. Words are for meaning;Once you get the meaning, you can forget the words (Zhuangzi), CHAPTER 26 The asphalt that Los Angeles is famous for occurs mainly on its freeways. Butin the middle of the city is another patch of asphalt, the La Brea tar pits, and thisasphalt preserves millions of fossil bones from the last of the Ice Ages of the Pleis-tocene Epoch. One of these fossils is theSmilodon, or saber-toothed tiger, instantlyrecognizable by its long canines.

2 Five million years ago or so, a completely differentsabre-tooth tiger calledThylacosmiluslivedin Argentina and other parts of South Amer-ica. Thylacosmilus was a marsupial whereasSmilodon was a placental mammal, but Thy-lacosmilus had the same long upper caninesand, like Smilodon, had a protective boneflange on the lower jaw. The similarity ofthese two mammals is one of many examplesof parallel or convergent evolution, in which particular contexts or environmentslead to the evolution of very similar structures in different species (Gould, 1980).The role of context is also important in the similarity of a less biological kindof organism: the word.

3 Words that occur insimilar contextstend to havesimilarmeanings. This link between similarity in how words are distributed and similarityin what they mean is called thedistributional hypothesis. The hypothesis wasdistributionalhypothesisfirst formulated in the 1950s by linguists like Joos (1950), Harris (1954), and Firth(1957), who noticed that words which are synonyms (likeoculistandeye-doctor)tended to occur in the same environment ( , near words likeeyeorexamined)with the amount of meaning difference between two words corresponding roughlyto the amount of difference in their environments (Harris, 1954, 157).

4 In this CHAPTER we introducevector Semantics , which instantiates this linguisticvectorsemanticshypothesis by learning representations of the meaning of words, calledembeddings,embeddingsdirectly from their distributions in texts. These representations are used in every nat-ural language processing application that makes use of meaning, and thestatic em-beddingswe introduce here underlie the more powerful dynamic orcontextualizedembeddingslikeBERT that we will see in CHAPTER word representations are also the first example in this book ofrepre-sentation learning, automatically learning useful representations of the input suchself-supervisedways to learn representations of the input, instead ofcreating representations by hand viafeature engineering, is an important focus ofNLP research (Bengio et al.)

5 , 2013).2 CHAPTER6 VECTORSEMANTICS Lexical SemanticsLet s begin by introducing some basic principles of word meaning. How shouldwe represent the meaning of a word? In the n-gram models of CHAPTER 3, and inclassical NLP applications, our only representation of a word is as a string of letters,or an index in a vocabulary list. This representation is not that different from atradition in philosophy, perhaps you ve seen it in introductory logic classes, in whichthe meaning of words is represented by just spelling the word with small capitalletters; representing the meaning of dog asDOG, and cat asCAT, or by using anapostrophe (DOG ).

6 Representing the meaning of a word by capitalizing it is a pretty unsatisfactorymodel. You might have seen a version of a joke due originally to semanticist BarbaraPartee (Carlson, 1977):Q: What s the meaning of life?A:LIFE Surely we can do better than this! After all, we ll want a model of word meaningto do all sorts of things for us. It should tell us that some words have similar mean-ings (catis similar todog), others are antonyms (coldis the opposite ofhot), somehave positive connotations (happy) while others have negative connotations (sad).It should represent the fact that the meanings ofbuy,sell, andpayoffer differingperspectives on the same underlying purchasing event (If I buy something from you,you ve probably sold it to me, and I likely paid you).

7 More generally, a model ofword meaning should allow us to draw inferences to address meaning-related taskslike question-answering or this section we summarize some of these desiderata, drawing on results in thelinguistic study of word meaning, which is calledlexical Semantics ; we ll return tolexicalsemanticsand expand on this list in CHAPTER 18 and CHAPTER and SensesLet s start by looking at how one word (we ll choosemouse)might be defined in a dictionary (simplified from the online dictionary WordNet):mouse (N)1. any of numerous small a hand-operated device that controls a the formmouseis thelemma, also called thecitation form.

8 The formlemmacitation formmousewould also be the lemma for the wordmice; dictionaries don t have separatedefinitions for inflected forms likemice. Similarlysingis the lemma forsing,sang,sung. In many languages the infinitive form is used as the lemma for the verb, soSpanishdormir to sleep is the lemma forduermes you sleep . The specific formssungorcarpetsorsingorduermesare the example above shows, each lemma can have multiple meanings; thelemmamousecan refer to the rodent or the cursor control device. We call eachof these aspects of the meaning ofmouseaword sense. The fact that lemmas canbepolysemous(have multiple senses) can make interpretation difficult (is someonewho types mouse info into a search engine looking for a pet or a tool?)

9 CHAPTER 18will discuss the problem of polysemy, and introduceword sense disambiguation,the task of determining which sense of a word is being used in a particular important component of word meaning is the relationship be-tween word senses. For example when one word has a sense whose meaning LEXICALSEMANTICS3identical to a sense of another word, or nearly identical, we say the two senses ofthose two words aresynonyms. Synonyms include such pairs assynonymcouch/sofa vomit/throw up filbert/hazelnut car/automobileA more formal definition of synonymy (between words rather than senses) is thattwo words are synonymous if they are substitutable for one another in any sentencewithout changing thetruth conditionsof the sentence, the situations in which thesentence would be true.

10 We often say in this case that the two words have the samepropositional substitutions between some pairs of words likecar/automobileorwa-ter/H2 Oare truth preserving, the words are still not identical in meaning. Indeed,probably no two words are absolutely identical in meaning. One of the fundamentaltenets of Semantics , called theprinciple of contrast(Girard 1718, Br eal 1897, Clarkprinciple ofcontrast1987), states that a difference in linguistic form is always associated with some dif-ference in meaning. For example, the wordH2 Ois used in scientific contexts andwould be inappropriate in a hiking guide waterwould be more appropriate andthis genre difference is part of the meaning of the word.