20.4.2 Selectional Restrictions and Selectional Preferences
One of the earliest knowledge-sources for sense disambiguation is the notion of selectional restrictions defined in Ch. 19. For example the verb eat might have a restriction that its THEME argument be [+FOOD]. In early systems, selectional restrictions were used to rule out senses that violate the selectional restrictions of neighboring words (Katz and Fodor, 1963; Hirst, 1987). Consider the following pair of WSJ examples of the word dish:
"In our house, everybody has a career and none of them includes washing dishes," he says.
(20.13) In her tiny kitchen at home, Ms. Chen works efficiently, stir-frying several simple dishes, including braised pig's ears and chicken livers with green peppers.
These correspond to WordNet dish $ ^{1} $ (a piece of dishware normally used as a container for holding or serving food), with hypernyms like artifact, and dish $ ^{2} $ (a particular item of prepared food) with hypernyms like food.
The fact that we perceive no ambiguity in these examples can be attributed to the selectional restrictions imposed by wash and stir-fry on their THEME semantic roles. The restrictions imposed by wash (perhaps [+WASHABLE]) conflict with dish $ ^{2} $. The restrictions on stir-fry ([+EDIBLE]) conflict with dish $ ^{1} $. In early systems, the predicate strictly selected the correct sense of an ambiguous argument by eliminating the sense that fails to match one of its selectional restrictions. But such hard constraints have a number of problems. The main problem is that selectional restriction violations often occur in well-formed sentences, either because they are negated as in (20.14), or because selectional restrictions are overstated as in (20.15):
(20.14) But it fell apart in 1931, perhaps because people realized you can't eat gold for lunch if you're hungry.
(20.15) In his two championship trials, Mr. Kulkarni ate glass on an empty stomach, accompanied only by water and tea.
As Hirst (1987) observes, examples like these often result in the elimination of all senses, bringing semantic analysis to a halt. Modern models thus adopt the view of selectional restrictions as preferences, rather than rigid requirements. Although there have been many instantiations of this approach over the years (e.g., Wilks, 1975c, 1975b, 1978), we'll discuss a member of the popular probabilistic or information-theoretic family of approaches: Resnik's (1997) model of selectional association.
Resnik first defines the selectional preference strength as the general amount of information that a predicate tells us about the semantic class of its arguments. For
example, the verb eat tells us a lot about the semantic class of its direct object, since they tend to be edible. The verb be, by contrast, tells us less about its direct objects. The selectional preference strength can be defined by the difference in information between two distributions: the distribution of expected semantic classes $ P(c) $ (how likely is it that a direct object will fall into class c) and the distribution of expected semantic classes for the particular verb $ P(c|v) $ (how likely is it that the direct object of specific verb v will fall into semantic class c). The greater the difference between these distributions, the more information the verb is giving us about possible objects. This difference can be quantified by the relative entropy between these two distributions, or Kullback-Leibler divergence (Kullback and Leibler, 1951). The Kullback-Leibler or KL divergence $ D(P||Q) $ can be used to express the difference between two probability distributions P and Q, and will be discussed further when we discuss word similarity in Equation (20.50).
$$ \begin{array}{c} D(P||Q)~=~\sum_{x} P(x)\log\frac{P(x)}{Q(x)}\end{array} $$
The selectional preference $S_{R}(v)$ uses the KL divergence to express how much information, in bits, the verb $v$ expresses about the possible semantic class of its argument.
$$ \begin{align*}S_{R}(v)&=D(P(c|v)||P(c))\\&=\sum_{c}P(c|v)\log\frac{P(c|v)}{P(c)}\end{align*} $$
Resnik then defines the selectional association of a particular class and verb as the relative contribution of that class to the general selectional preference of the verb:
$$ A_{R}(v,c)~=~\frac{1}{S_{R}(p)}P(c|v)\log\frac{P(c|v)}{P(c)} $$
The selectional association is thus a probabilistic measure of the strength of association between a predicate and a class dominating the argument to the predicate. Resnik estimates the probabilities for these associations by parsing a corpus, counting all the times each predicate occurs with each argument word, and assuming that each word is a partial observation of all the WordNet concepts containing the word. The following table from Resnik (1996) shows some sample high and low selectional associations for verbs and some WordNet semantic classes of their direct objects.
| Verb | Direct Object Semantic Class | Assoc | Direct Object Semantic Class | Assoc |
| read | WRITING | 6.80 | ACTIVITY | -0.20 |
| write | WRITING | 7.26 | COMMERCE | 0 |
| see | ENTITY | 5.79 | METHOD | -0.01 |
Resnik (1998) shows that these selectional associations can be used to perform a limited form of word sense disambiguation. Roughly speaking the algorithm selects as the correct sense for an argument the one that has the highest selectional association between one of its ancestor hypernyms and the predicate.
While we have presented only the Resnik model of selectional preferences, there are other more recent models, using probabilistic methods and using other relations than just direct object; see the end of the chapter for a brief summary. In general, selectional restriction approaches perform as well as other unsupervised approaches at sense disambiguation, but not as well as Lesk or as supervised approaches.
20.5 MINIMALLY SUPERVISED WSD: BOOTSTRAPPING
Both the supervised approach and the dictionary-based approach to WSD require large hand-built resources; supervised training sets in one case, large dictionaries in the other. We can instead use bootstrapping algorithms, often called semi-supervised learning or minimally supervised learning, which need only a very small hand-labeled training set. The most widely emulated bootstrapping algorithm for WSD is the Yarowsky algorithm (Yarowsky, 1995).
The goal of the Yarowsky algorithm is to learn a classifier for a target word (in a lexical-sample task). The algorithm is given a small seed-set $ \Lambda_0 $ of labeled instances of each sense, and a much larger unlabeled corpus $ V_0 $. The algorithm first trains an initial decision-list classifier on the seed-set $ \Lambda_0 $. It then uses this classifier to label the unlabeled corpus $ V_0 $. The algorithm then selects the examples in $ V_0 $ that it is most confident about, removes them, and adds them to the training set (call it now $ \Lambda_1 $). The algorithm then trains a new decision list classifier (a new set of rules) on $ \Lambda_1 $, and iterates by applying the classifier to the now-smaller unlabeled set $ V_1 $, extracting a new training set $ \Lambda_2 $ and so on. With each iteration of this process, the training corpus grows and the untagged corpus shrinks. The process is repeated until some sufficiently low error-rate on the training set is reached, or until no further examples from the untagged corpus are above threshold.
The key to any bootstrapping approach lies in its ability to create a larger training set from a small set of seeds. This requires an accurate initial set of seeds and a good confidence metric for picking good new examples to add to the training set. The confidence metric used by Yarowsky (1995) is the measure described earlier in Sec. 20.2.2, the log-likelihood ratio of the decision-list rule that classified the example.
One way to generate the initial seeds is to hand-label a small set of examples (Hearst, 1991). Instead of hand-labeling, it is also possible to use a heuristic to automatically select accurate seeds. Yarowsky (1995) used the One Sense per Collocation heuristic, which relies on the intuition that certain words or phrases strongly associated with the target senses tend not to occur with the other sense. Yarowsky defines his seed set by choosing a single collocation for each sense. As an illustration of this technique, consider generating seed sentences for the fish and musical senses of bass. Without too much thought, we might come up with fish as a reasonable indicator of bass $ ^{1} $, and play as a reasonable indicator of bass $ ^{2} $. Figure 20.5 shows a partial result of such a search for the strings “fish” and “play” in a corpus of bass examples drawn from the WSJ.
We can also suggest collocates automatically, for example extracting words from machine readable dictionary entries, and selecting seeds using collocational statistics such as those described in Sec. 20.7 (Yarowsky, 1995).
| We need more good teachers – right now, there are only a half a dozen who can play the free bass with ease. |
| An electric guitar and bass player stand off to one side, not really part of the scene, just as a sort of nod to gringo expectations perhaps. |
| When the New Jersey Jazz Society, in a fund-raiser for the American Jazz Hall of Fame, honors this historic night next Saturday, Harry Goodman, Mr. Goodman's brother and bass player at the original concert, will be in the audience with other family members. |
| The researchers said the worms spend part of their life cycle in such fish as Pacific salmon and striped bass and Pacific rockfish or snapper. |
| And it all started when fishermen decided the striped bass in Lake Mead were too skinny. Though still a far cry from the lake's record 52-pound bass of a decade ago, “you could fillet these fish again, and that made people very, very happy,” Mr. Paulson says. |
| Figure 20.5 Samples of bass sentences extracted from the WSJ using the simple correlates play and fish. |
The original Yarowsky algorithm also makes use of a second heuristic, called One Sense Per Discourse, based on the work of Gale et al. (1992c), who noticed that a particular word appearing multiple times in a text or discourse often appeared with the same sense. Yarowsky (1995), for example, showed in a corpus of 37,232 examples that every time the word bass occurred more than once in a discourse, that it occurred in only the fish or only the music coarse-grain sense throughout the discourse. The validity of this heuristic depends on the granularity of the sense inventory and is not valid.
in every discourse situation; it seems to be true mostly for coarse-grain senses, and particularly for cases of homonymy rather than polysemy (Krovetz, 1998). Nonetheless, it has still been useful in a number of unsupervised and semi-supervised sense disambiguation situations.
20.6 WORD SIMILARITY: THESAURUS METHODS
We turn now to the computation of various semantic relations that hold between words. We saw in Ch. 19 that such relations include synonymy, antonymy, hyponymy, hypernymy, and meronymy. Of these, the one that has been most computationally developed and has the greatest number of applications is the idea of word synonymy and similarity.
Synonymy is a binary relation between words; two words are either synonyms or not. For most computational purposes we use instead a looser metric of word similarity or semantic distance. Two words are more similar if they share more features of meaning, or are near-synonyms. Two words are less similar, or have greater semantic distance, if they have fewer common meaning elements. Although we have described them as relations between words, synonymy, similarity, and distance are actually relations between word senses. For example of the two senses of bank, we might say that the financial sense is similar to one of the senses of fund while the riparian sense is more similar to one of the senses of slope. In the next few sections of this chapter, we will need to compute these relations over both words and senses.
The ability to compute word similarity is a useful part of many language understanding applications. In information retrieval or question answering we might want to retrieve documents whose words have similar meanings to the query words. In summarization, generation, and machine translation, we need to know whether two words are similar to know if we can substitute one for the other in particular contexts. In language modeling, we can use semantic similarity to cluster words for class-based models. One interesting class of applications for word similarity is automatic grading of student responses. For example algorithms for automatic essay grading use word similarity to determine if an essay is similar in meaning to a correct answer. We can also use word-similarity as part of an algorithm to take an exam, such as a multiple-choice vocabulary test. Automatically taking exams is useful in test designs in order to see how easy or hard a particular multiple-choice question or exam is.
There are two classes of algorithms for measuring word similarity. This section focuses on thesaurus-based algorithms, in which we measure the distance between two senses in an on-line thesaurus like WordNet or MeSH. The next section focuses on the distributional algorithms, in which we estimate word similarity by finding words that have similar distributions in a corpus.
The thesaurus-based algorithms use the structure of the thesaurus to define word similarity. In principle we could measure similarity using any information available in a thesaurus (meronymy, glosses, etc). In practice, however, thesaurus-based word similarity algorithms generally use only the hypernym/hyponym (is-a or subsumption) hierarchy. In WordNet, verbs and nouns are in separate hypernym hierarchies, so a
thesaurus-based algorithm for WordNet can thus only compute noun-noun similarity, or verb-verb similarity; we can't compare nouns to verbs, or do anything with adjectives or other parts of speech.
Resnik (1995) and Budanitsky and Hirst (2001) draw the important distinction between word similarity and word relatedness. Two words are similar if they are near-synonyms, or roughly substitutable in context. Word relatedness characterizes a larger set of potential relationships between words; antonyms, for example, have high relatedness, but low similarity. The words car and gasoline are very related, but not similar, while car and bicycle are similar. Word similarity is thus a subcase of word relatedness. In general, the five algorithms we describe in this section do not attempt to distinguish between similarity and semantic relatedness; for convenience we will call them similarity measures, although some would be more appropriately described as relatedness measures; we return to this question in Sec. 20.8.
The oldest and simplest thesaurus-based algorithms are based on the intuition that the shorter the path between two words or senses in the graph defined by the thesaurus hierarchy, the more similar they are. Thus a word/sense is very similar to its parents or its siblings, and less similar to words that are far away in the network. This notion can be operationalized by measuring the number of edges between the two concept nodes in the thesaurus graph. Fig. 20.6 shows an intuition; the concept dime is most similar to nickel and coin, less similar to money, and even less similar to Richter scale. Formally, we specify path length as follows:
pathlen $ (c_{1},c_{2})= $ the number of edges in the shortest path in the thesaurus graph between the sense nodes $ c_{1} $ and $ c_{2} $
Path-based similarity can be defined just as the path length, often with a log transform (Leacock and Chodorow, 1998), resulting in the following common definition of path-length based similarity:
$$ \mathrm{sim_{path}}(c_{1},c_{2})=-\log\mathrm{pathlen}(c_{1},c_{2}) $$
For most applications, we don’t have sense-tagged data, and thus we need our algorithm to give us the similarity between words rather than between senses or concepts. For any of the thesaurus-based algorithms, following Resnik (1995), we can approximate the correct similarity (which would require sense disambiguation) by just using the pair of senses for the two words that results in maximum sense similarity. Thus based on sense similarity we can define word similarity as follows:
$$ \mathrm{wordsim}(w_{1},w_{2})=\max_{\substack{c_{1}\in\mathrm{senses}(w_{1})\\ c_{2}\in\mathrm{senses}(w_{2})}}\mathrm{sim}(c_{1},c_{2}) $$
The basic path-length algorithm makes the implicit assumption that each link in the network represents a uniform distance. In practice, this assumption is not appropriate. Some links (for example those that are very deep in the WordNet hierarchy) often seem to represent an intuitively narrow distance, while other links (e.g., higher up in the WordNet hierarchy) represent an intuitively wider distance. For example, in Fig. 20.6, the distance from nickel to money (5) seems intuitively much shorter than the distance from nickel to an abstract word standard; the link between medium of exchange and standard seems wider than that between, say, coin and coinage.
It is possible to refine path-based algorithms with normalizations based on depth in the hierarchy (Wu and Palmer, 1994), but in general we'd like an approach which lets us represent the distance associated with each edge independently.
A second class of thesaurus-based similarity algorithms attempts to offer just such a fine-grained metric. These information content word similarity algorithms still rely on the structure of the thesaurus, but also add probabilistic information derived from a corpus.
Using similar notions to those we introduced earlier to define soft selectional restrictions, let's first define $ P(c) $, following Resnik (1995), as the probability that a randomly selected word in a corpus is an instance of concept c (i.e., a separate random variable, ranging over words, associated with each concept). This implies that $ P(root) = 1 $, since any word is subsumed by the root concept. Intuitively, the lower a concept in the hierarchy, the lower its probability. We train these probabilities by counting in a corpus; each word in the corpus counts as an occurrence of each concept that contains it. For example, in Fig. 20.6 above, an occurrence of the word dime would count toward the frequency of coin, currency, standard, etc. More formally, Resnik computes $ P(c) $ as follows:
$$ P(c)=\frac{\sum_{w\in\mathrm{words}(c)}count(w)}{N} $$
where words(c) is the set of words subsumed by concept c, and N is the total number of words in the corpus that are also present in the thesaurus.
Fig. 20.7, from Lin (1998b), shows a fragment of the WordNet concept hierarchy augmented with the probabilities $ P(c) $.
We now need two additional definitions. First, following basic information theory, we define the information content (IC) of a concept c as:
$$ IC(c)=-\log P(c) $$
Second, we define the lowest common subsumer or LCS of two concepts:
LCS$(c_{1},c_{2})$ = the lowest common subsumer, i.e., the lowest node in the hierarchy that subsumes (is a hypernym of) both $c_{1}$ and $c_{2}$
There are now a number of ways to use the information content of a node in a word similarity metric. The simplest way was first proposed by Resnik (1995). We think of the similarity between two words as related to their common information; the more two words have in common, the more similar they are. Resnik proposes to estimate the common amount of information by the information content of the lowest common subsumer of the two nodes. More formally, the Resnik similarity measure is:
$$ \mathrm{sim_{resnik}}(c_{1},c_{2})=-\log P(\mathrm{LCS}(c_{1},c_{2})) $$
Lin (1998b) extended the Resnik intuition by pointing out that a similarity metric between objects A and B needs to do more than measure the amount of information in common between A and B. For example, he pointed out that in addition, the more differences between A and B, the less similar they are. In summary:
commonality: the more information A and B have in common, the more similar they are.
- difference: the more differences between the information in A and B, the less similar they are
Lin measures the commonality between A and B as the information content of the proposition that states the commonality between A and B:
$$ IC(Common(A,B)) $$
He measures the difference between A and B as
$$ \mathrm{IC}(\mathrm{description}(\mathrm{A},\mathrm{B}))-\mathrm{IC}(\mathrm{common}(\mathrm{A},\mathrm{B})) $$
where description(A,B) describes A and B. Given a few additional assumptions about similarity, Lin proves the following theorem:
Similarity Theorem: The similarity between A and B is measured by the ratio between the amount of information needed to state the commonality of A and B and the information needed to fully describe what A and B are:
$$ sim_{Lin}(A,B)=\frac{\log P(common(A,B))}{\log P(description(A,B))} $$
Applying this idea to the thesaurus domain, Lin shows (in a slight modification of Resnik's assumption) that the information in common between two concepts is twice the information in the lowest common subsumer LCS $ (c_1, c_2) $. Adding in the above definitions of the information content of thesaurus concepts, the final Lin similarity function is:
$$ \mathrm{sim}_{\mathrm{Lin}}(c_{1},c_{2})=\frac{2\times\log P(LCS(c_{1},c_{2}))}{\log P(c_{1})+\log P(c_{2})} $$
For example, using $ \text{sim}_{\text{lin}} $, Lin (1998b) shows that the similarity between the concepts of hill and coast from Fig. 20.7 is:
$$ \mathrm{sim}_{\mathrm{Lin}}(\mathrm{hill,coast})=\frac{2\times\log P(\mathrm{geological-formation})}{\log P(\mathrm{hill})+\log P(\mathrm{coast}))}=0.59 $$
A very similar formula, Jiang-Conrath distance (Jiang and Conrath, 1997) (although derived in a completely different way from Lin, and expressed as a distance rather than similarity function) has been shown to work as well or better than all the other thesaurus-based methods:
$$ \mathrm{dist}_{\mathbf{JC}}(c_{1},c_{2})=2\times\log P(\mathrm{LCS}(c_{1},c_{2}))-(\log P(c_{1})+\log P(c_{2})) $$
___ local.
Finally, we describe a dictionary-based method, an extension of the Lesk algorithm for word-sense disambiguation described in Sec. 20.4.1. We call this a dictionary rather than a thesaurus method because it makes use of glosses, which are in general a property of dictionaries rather than thesauri (although WordNet does have glosses). Like the Lesk algorithm, the intuition of this Extended Gloss Overlap, or Extended Lesk measure (Banerjee and Pedersen, 2003) is that two concepts/senses in a thesaurus are similar if their glosses contain overlapping words. We'll begin by sketching an overlap function for two glosses. Consider these two concepts, with their glosses:
• drawing paper: $ \underline{\text{paper}} $ that is $ \underline{\text{specially prepared}} $ for use in drafting
• decal: the art of transferring designs from $ \underline{\text{specially prepared paper}} $ to a wood or glass or metal surface.
For each $n$-word phrase that occurs in both glosses, Extended Lesk adds in a score of $n^{2}$ (the relation is non-linear because of the Zipfian relationship between lengths of phrases and their corpus frequencies; longer overlaps are rare so should be weighted
more heavily). Here the overlapping phrases are paper and specially prepared, for a total similarity score of $ 1^{2}+2^{2}=5 $.
Given such an overlap function, when comparing two concepts (synsets), Extended Lesk not only looks for overlap between their glosses, but also between the glosses of the senses which are hypernyms, hyponyms, meronyms, and other relations of the two concepts. For example if we just considered hyponyms, and defined gloss(hypo(A)) as the concatenation of all the glosses of all the hyponym senses of A, the total relatedness between two concepts A and B might be:
$$ \begin{aligned}similarity(A,B)=&\;overlap(gloss(A),gloss(B))\\&+overlap(gloss(hypo(A)),gloss(hypo(B)))\\&+overlap(gloss(A),gloss(hypo(B)))\\&+overlap(gloss(hypo(A)),gloss(B))\end{aligned} $$
Let RELS be the set of possible WordNet relations whose glosses we compare; assuming a basic overlap measure as sketched above, we can then define the Extended Lesk overlap measure as:
$$ \mathrm{sim_{eLesk}}(c_{1},c_{2})=\sum_{r,q\in RELS}\mathrm{overlap}(\mathrm{gloss}(r(c_{1})),\mathrm{gloss}(q(c_{2}))) $$
$$ \begin{align*}\operatorname{sim}_{\operatorname{path}}(c_1,c_2)&=-\log\operatorname{pathlen}(c_1,c_2)\\\operatorname{sim}_{\operatorname{Resnik}}(c_1,c_2)&=-\log P(\operatorname{LCS}(c_1,c_2))\\\operatorname{sim}_{\operatorname{Lin}}(c_1,c_2)&=\frac{2\times\log P(\operatorname{LCS}(c_1,c_2))}{\log P(c_1)+\log P(c_2)}\\\operatorname{sim}_{\mathrm{jc}}(c_1,c_2)&=\frac{1}{2\times\log P(\operatorname{LCS}(c_1,c_2))-(\log P(c_1)+\log P(c_2))}\\\operatorname{sim}_{\mathrm{eLesk}}(c_1,c_2)&=\sum_{r,q\in\operatorname{RELS}}\operatorname{overlap}(\operatorname{gloss}(r(c_1)),\operatorname{gloss}(q(c_2)))\end{align*} $$
Fig. 20.8 summarizes the five similarity measures we have described in this section. The publicly available Wordnet::Similarity package implementing all these and other thesaurus-based word similarity measures is described in Pedersen et al. (2004).
Evaluating Thesaurus-based Similarity Which of these similarity measures is best? Word similarity measures have been evaluated in two ways. One intrinsic method is to compute the correlation coefficient between word similarity scores from an algorithm and word similarity ratings assigned by humans; such human ratings have been obtained for 65 word pairs by Rubenstein and Goodenough (1965), and 30 word pairs by
Miller and Charles (1991). Another more extrinsic evaluation method is to embed the similarity measure in some end application like detection of malapropisms (real-word spelling errors) (Budanitsky and Hirst, 2006; Hirst and Budanitsky, 2005), or other NLP applications like word-sense disambiguation (Patwardhan et al., 2003; McCarthy et al., 2004) and evaluate its impact on end-to-end performance. All of these evaluations suggest that all the above measures perform relatively well, and that of these, Jiang-Conrath similarity and Extended Lesk similarity are two of the best approaches, depending on the application.
20.7 WORD SIMILARITY: DISTRIBUTIONAL METHODS
The previous section showed how to compute similarity between any two senses in a thesaurus, and by extension between any two words in the thesaurus hierarchy. But of course we don't have such thesauri for every language. Even for languages where we do have such resources, thesaurus-based methods have a number of limitations. The obvious limitation is that thesauri often lack words, especially new or domain-specific words. In addition, thesaurus-based methods only work if rich hyponymy knowledge is present in the thesaurus. While we have this for nouns, hyponym information for verbs tends to be much sparser, and doesn't exist at all for adjectives and adverbs. Finally, it is more difficult with thesaurus-based methods to compare words in different hierarchies, such as nouns with verbs.
For these reasons, methods which can automatically extract synonyms and other word relations from corpora have been developed. In this section we introduce such distributional methods, which can be applied directly to supply a word-relatedness measure for NLP tasks. Distributional methods can also be used for automatic thesaurus generation for automatically populating or augmenting on-line thesauruses like WordNet with new synonyms and, as we will see in Sec. 20.8, with other relations like hyponymy and meronymy.
The intuition of distributional methods is that the meaning of a word is related to the distribution of words around it; in the famous dictum of Firth (1957), “You shall know a word by the company it keeps!”. Consider the following example, modified by Lin (1998a) from (?):
(20.31) A bottle of tezgüino is on the table.
Everybody likes tezgúino.
Tezgüino makes you drunk.
We make tezgüino out of corn.
The contexts in which tezgüino occurs suggest that it might be some kind of fermented alcoholic drink made from corn. The distributional method tries to capture this intuition by representing features of the context of tezgüino that might overlap with features of similar words like beer, liquor, tequila, and so on. For example such features might be occurs before drunk or occurs after bottle or is the direct object of likes.
We can then represent a word $w$ as a feature vector just as we saw with the bag-of-words features in Sec. 20.2. For example, suppose we had one binary feature $f_i$ representing each of the $N$ words in the lexicon $v_i$. The feature means $w$ occurs in the
neighborhood of word $ v_i $, and hence takes the value 1 if w and $ v_i $ occur in some context window, and 0 otherwise. We could represent the meaning of word w as the feature vector
$$ \vec{w}=\left(f_{1},f_{2},f_{3},\cdots,f_{N}\right) $$
If $w = tezgüino$, $v_1 = bottle$, $v_2 = drunk$, and $v_3 = matrix$, the co-occurrence vector for $w$ from the corpus above would be:
$$ \overrightarrow{w}=(1,1,0,\cdots) $$
Given two words represented by such sparse feature vectors, we can apply a vector distance measure and say that the words are similar if the two vectors are close by this measure. Fig. 20.9 shows an intuition about vector similarity for the four words apricot, pineapple, digital, and information. Based on the meanings of these four words, we would like a metric that shows apricot and pineapple to be similar, digital and information, to be similar, and the other four pairings to produce low similarity. For each word, Fig. 20.9 shows a short piece (8 dimensions) of the (binary) word co-occurrence vectors, computed from words that occur within a two-line context in the Brown corpus. The reader should convince themselves that the vectors for apricot and pineapple are indeed more similar than those of, say, apricot and information. For pedagogical purposes we've shown the context words that are particularly good at discrimination. Note that since vocabularies are quite large (10,000-100,000 words) and most words don't occur near each other in any corpus, real vectors are quite sparse.
| arts | boil | data | function | large | sugar | summarized | water | |
| apricot | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 |
| pineapple | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 |
| digital | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 |
| information | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 |
Now that we have some intuitions, let's move on to examine the details of these measures. Specifying a distributional similarity measure requires that we specify three parameters: (1) how the co-occurrence terms are defined (i.e. what counts as a neighbor), (2) how these terms are weighted (binary? frequency? mutual information?) and (3) what vector distance metric we use (cosine? Euclidean distance?). Let's look at each of these requirements in the next three subsections.