CHAPTER Vector Semantics and Embeddings

1 Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright 2021. Allrights reserved. Draft of December 29, Semantics andEmbeddings Nets are for fish;Once you get the fish, you can forget the net. Words are for meaning;Once you get the meaning, you can forget the words (Zhuangzi), CHAPTER 26 The asphalt that Los Angeles is famous for occurs mainly on its freeways. Butin the middle of the city is another patch of asphalt, the La Brea tar pits, and thisasphalt preserves millions of fossil bones from the last of the Ice Ages of the Pleis-tocene Epoch. One of these fossils is theSmilodon, or saber-toothed tiger, instantlyrecognizable by its long canines. Five million years ago or so, a completely differentsabre-tooth tiger calledThylacosmiluslivedin Argentina and other parts of South Amer-ica. Thylacosmilus was a marsupial whereasSmilodon was a placental mammal, but Thy-lacosmilus had the same long upper caninesand, like Smilodon, had a protective boneflange on the lower jaw.

2 The similarity ofthese two mammals is one of many examplesof parallel or convergent evolution, in which particular contexts or environmentslead to the evolution of very similar structures in different species (Gould, 1980).The role of context is also important in the similarity of a less biological kindof organism: the word. Words that occur insimilar contextstend to havesimilarmeanings. This link between similarity in how words are distributed and similarityin what they mean is called thedistributional hypothesis. The hypothesis wasdistributionalhypothesisfirst formulated in the 1950s by linguists like Joos (1950), Harris (1954), and Firth(1957), who noticed that words which are synonyms (likeoculistandeye-doctor)tended to occur in the same environment ( , near words likeeyeorexamined)with the amount of meaning difference between two words corresponding roughlyto the amount of difference in their environments (Harris, 1954, 157).

3 In this CHAPTER we introducevector Semantics , which instantiates this linguisticvectorsemanticshypothesis by learning representations of the meaning of words, calledembeddings,embeddingsdirectly from their distributions in texts. These representations are used in every nat-ural language processing application that makes use of meaning, and thestatic em-beddingswe introduce here underlie the more powerful dynamic orcontextualizedembeddingslikeBERT that we will see in CHAPTER word representations are also the first example in this book ofrepre-sentation learning, automatically learning useful representations of the input suchself-supervisedways to learn representations of the input, instead ofcreating representations by hand viafeature engineering, is an important focus ofNLP research (Bengio et al., 2013).2 CHAPTER6 VECTORSEMANTICS Lexical SemanticsLet s begin by introducing some basic principles of word meaning.

4 How shouldwe represent the meaning of a word? In the n-gram models of CHAPTER 3, and inclassical NLP applications, our only representation of a word is as a string of letters,or an index in a vocabulary list. This representation is not that different from atradition in philosophy, perhaps you ve seen it in introductory logic classes, in whichthe meaning of words is represented by just spelling the word with small capitalletters; representing the meaning of dog asDOG, and cat asCAT, or by using anapostrophe (DOG ).Representing the meaning of a word by capitalizing it is a pretty unsatisfactorymodel. You might have seen a version of a joke due originally to semanticist BarbaraPartee (Carlson, 1977):Q: What s the meaning of life?A:LIFE Surely we can do better than this! After all, we ll want a model of word meaningto do all sorts of things for us.

5 It should tell us that some words have similar mean-ings (catis similar todog), others are antonyms (coldis the opposite ofhot), somehave positive connotations (happy) while others have negative connotations (sad).It should represent the fact that the meanings ofbuy,sell, andpayoffer differingperspectives on the same underlying purchasing event (If I buy something from you,you ve probably sold it to me, and I likely paid you). More generally, a model ofword meaning should allow us to draw inferences to address meaning-related taskslike question-answering or this section we summarize some of these desiderata, drawing on results in thelinguistic study of word meaning, which is calledlexical Semantics ; we ll return tolexicalsemanticsand expand on this list in CHAPTER 18 and CHAPTER and SensesLet s start by looking at how one word (we ll choosemouse)might be defined in a dictionary (simplified from the online dictionary WordNet):mouse (N)1.

6 Any of numerous small a hand-operated device that controls a the formmouseis thelemma, also called thecitation form. The formlemmacitation formmousewould also be the lemma for the wordmice; dictionaries don t have separatedefinitions for inflected forms likemice. Similarlysingis the lemma forsing,sang,sung. In many languages the infinitive form is used as the lemma for the verb, soSpanishdormir to sleep is the lemma forduermes you sleep . The specific formssungorcarpetsorsingorduermesare the example above shows, each lemma can have multiple meanings; thelemmamousecan refer to the rodent or the cursor control device. We call eachof these aspects of the meaning ofmouseaword sense. The fact that lemmas canbepolysemous(have multiple senses) can make interpretation difficult (is someonewho types mouse info into a search engine looking for a pet or a tool?)

7 CHAPTER 18will discuss the problem of polysemy, and introduceword sense disambiguation,the task of determining which sense of a word is being used in a particular important component of word meaning is the relationship be-tween word senses. For example when one word has a sense whose meaning LEXICALSEMANTICS3identical to a sense of another word, or nearly identical, we say the two senses ofthose two words aresynonyms. Synonyms include such pairs assynonymcouch/sofa vomit/throw up filbert/hazelnut car/automobileA more formal definition of synonymy (between words rather than senses) is thattwo words are synonymous if they are substitutable for one another in any sentencewithout changing thetruth conditionsof the sentence, the situations in which thesentence would be true. We often say in this case that the two words have the samepropositional substitutions between some pairs of words likecar/automobileorwa-ter/H2 Oare truth preserving, the words are still not identical in meaning.

8 Indeed,probably no two words are absolutely identical in meaning. One of the fundamentaltenets of Semantics , called theprinciple of contrast(Girard 1718, Br eal 1897, Clarkprinciple ofcontrast1987), states that a difference in linguistic form is always associated with some dif- ference in meaning. For example, the wordH2 Ois used in scientific contexts andwould be inappropriate in a hiking guide waterwould be more appropriate andthis genre difference is part of the meaning of the word. In practice, the wordsyn-onymis therefore used to describe a relationship of approximate or rough SimilarityWhile words don t have many synonyms, most words do havelots not a synonym ofdog, butcatsanddogsare certainlysimilar words. In moving from synonymy to similarity, it will be useful to shift fromtalking about relations between word senses (like synonymy) to relations betweenwords (like similarity).

9 Dealing with words avoids having to commit to a particularrepresentation of word senses, which will turn out to simplify our notion of wordsimilarityis very useful in larger semantic tasks. Knowingsimilarityhow similar two words are can help in computing how similar the meaning of twophrases or sentences are, a very important component of tasks like question answer-ing, paraphrasing, and summarization. One way of getting values for word similarityis to ask humans to judge how similar one word is to another. A number of datasetshave resulted from such experiments. For example the SimLex-999 dataset (Hillet al., 2015) gives values on a scale from 0 to 10, like the examples below, whichrange from near-synonyms (vanish,disappear) to pairs that scarcely seem to haveanything in common (hole,agreement) RelatednessThe meaning of two words can be related in ways other thansimilarity.

10 One such class of connections is called wordrelatedness(Budanitskyrelatednessand Hirst, 2006), also traditionally called wordassociationin the meanings of the wordscoffeeandcup. Coffee is not similar to cup;they share practically no features (coffee is a plant or a beverage, while a cup is amanufactured object with a particular shape). But coffee and cup are clearly related;they are associated by co-participating in an everyday event (the event of drinkingcoffee out of a cup). Similarlyscalpelandsurgeonare not similar but are relatedeventively (a surgeon tends to make use of a scalpel).One common kind of relatedness between words is if they belong to the samesemantic field. A semantic field is a set of words which cover a particular semanticsemantic field4 CHAPTER6 VECTORSEMANTICS ANDEMBEDDINGS domain and bear structured relations with each other.