20.7.2 Measures of Association with Context

Now that we have a definition for the features or dimensions of a word's context vector, we are ready to discuss the values that should be associated with those features. These values are typically thought of as weights or measures of association between each target word w and a given feature f. In the example in Fig. 20.9, our association measure was a binary value for each feature, 1 if the relevant word had occurred in the context, 0 if not. In the example in Fig. 20.10, we used a richer association measure, the relative frequency with which the particular context feature had co-occurred with the target word.

Frequency, or probability, is certainly a better measure of association than just a binary value; features that occur often with a target word are more likely to be good indicators of the word's meaning. Let's define some terminology for implementing a probabilistic measure of association. For a target word w, each element of its co-occurrence vector is a feature f, consisting of a relation r and a related word $ w' $; we can say $ f = (r, w') $. For example, one of the features of the word cell in Fig. 20.10 is $ f = (r, w') = \text{obj-of, attack} $. The probability of a feature f given a target word w is $ P(f|w) $, for which the maximum likelihood estimate is:

$$ P(f|w)=\frac{count(f,w)}{count(w)} $$

Similarly, the maximum likelihood estimate for the joint probability $ P(f, w) $ is:

$$ \begin{array}{c} P(f,w)~=~\frac{\mathrm{count}(f,w)}{\sum_{w^{\prime}}\mathrm{count}(w^{\prime}))}\end{array} $$

$ P(w) $ and $ P(f) $ are computed similarly.

Thus if we were to define simple probability as a measure of association it would look as follows:

$$ \mathrm{assoc}_{\mathrm{prob}}(w,f)=P(f|w) $$

It turns out, however, that simple probability doesn't work as well as more sophisticated association schemes for word similarity.

Why isn't frequency or probability a good measure of association between a word and a context feature? Intuitively, if we want to know what kinds of contexts are shared by apricot and pineapple but not by digital and information, we're not going to get good discrimination from words like the, it, or they, which occur frequently with all sorts of words, and aren't informative about any particular word. We'd like context words which are particularly informative about the target word. We, therefore, need a weighting or measure of association which asks how much more often than chance that the feature co-occurs with the target word. As Curran (2003) points out, such a weighting is what we also want for finding good collocations, and so the measures of association used for weighting context words for semantic similarity are exactly the same measure used for finding a word's collocations.

One of the most important measures of association was first proposed by Church and Hanks (1989, 1990) and is based on the notion of mutual information. The mutual information between two random variables X and Y is

$$ I(X,Y)=\sum_{x}\sum_{y}P(x,y)\log_{2}\frac{P(x,y)}{P(x)P(y)} $$

The pointwise mutual information (Fano, 1961) $ ^{3} $ is a measure of how often two events x and y occur, compared with what we would expect if they were independent:

$$ I(x,y)=\log_{2}\frac{P(x,y)}{P(x)P(y)} $$

We can apply this intuition to co-occurrence vectors, by defining the pointwise mutual information association between a target word w and a feature f as:

$$ \mathrm{assoc}_{\mathrm{PMI}}(w,f)=\log_{2}\frac{P(w,f)}{P(w)P(f)} $$

The intuition of the PMI measure is that the numerator tells us how often we observed the two words together (assuming we compute probability using MLE as above). The denominator tells us how often we would expect the two words to co-occur assuming they each occurred independently, so their probabilities could just be multiplied. Thus the ratio gives us an estimate of how much more the target and feature co-occur than we expect by chance.

Since $f$ is itself composed of two variables $r$ and $w^{\prime}$, there is a slight variant on this model, due to Lin (1998a), that breaks down the expected value for $P(f)$ slightly differently; we'll call it the Lin association measure \textit{assoc\_Lin}, not to be confused with the WordNet measure \textit{sim\_Lin} that we discussed in the previous section:

$$ \mathrm{assoc}_{\mathrm{Lin}}(w,f)=\log_{2}\frac{P(w,f)}{P(w)P(r|w)P(w^{\prime}|w)} $$

For both assoc $ _{PMI} $ and assoc $ _{Lin} $, we generally only use the feature f for a word w if the assoc value is positive, since negative PMI values (implying things are co-

occurring less often than we would expect by chance) tend to be unreliable unless the training corpora are enormous (Dagan et al., 1993; Lin, 1998a). In addition, when we are using the assoc-weighted features to compare two target words, we only use features that co-occur with both target words.

Fig. 20.11 from Hindle (1990) shows the difference between raw frequency counts and PMI-style association, for some direct objects of the verb drink.

One of the most successful association measures for word similarity attempts to capture the same intuition as mutual information, but uses the t-test statistic to measure how much more frequent the association is than chance. This measure was proposed for collocation-detection by Manning and Schütze (1999, Chapter 5) and then applied to word similarity by Curran and Moens (2002), Curran (2003).

The t-test statistic computes the difference between observed and expected means, normalized by the variance. The higher the value of t, the more likely we can reject the null hypothesis that the observed and expected means are the same.

$$ t=\frac{\bar{x}-\mu}{\sqrt{\frac{s^{2}}{N}}} $$

When applied to association between words, the null hypothesis is that the two words are independent, and hence $ P(f, w) = P(f)P(w) $ correctly models the relationship between the two words. We want to know how different the actual MLE probability $ P(f, w) $ is from this null hypothesis value, normalized by the variance. Note the similarity to the comparison with the product model in the PMI measure above. The variance $ s^{2} $ can be approximated by the expected probability $ P(f)P(w) $ (see Manning and Schütze (1999)). Ignoring N (since it is constant), the resulting t-test association measure from Curran (2003) is thus:

$$ \mathrm{assoc_{t-test}}(w,f)=\frac{P(w,f)-P(w)P(f)}{\sqrt{P(f)P(w)}} $$

See the history section for a summary of various other weighting factors that have been tested on word similarity.