20.7.1 Defining a Word's Co-occurrence Vectors

In our example feature vector, we used the feature $ \underline{\text{w occurs in the neighborhood of word v.}} $. That is, for a vocabulary size N, each word w had N features, specifying

whether vocabulary element $ \nu_j $ occurred in the neighborhood. Neighborhoods range from a small window of words (as few as one or two words on either side) to very large windows of $ \pm 500 $ words. In a minimal window, for example, we might have two features for each word $ \nu_j $ in the vocabulary, $ \underline{\text{word}} $ $ \nu_k $ occurs immediately before word $ \underline{\text{w}} $ and word $ \nu_k $ occurs immediately after word $ \underline{\text{w}} $.

To keep these contexts efficient, we often ignore very frequent words which tend not to be very discriminative, e.g., function words such as a, am, the, of, 1, 2, and so on. These removed words are called stopwords or the stoplist.

Even with the removal of the stopwords, when used on very large corpora these co-occurrence vectors tend to be very large. Instead of using every word in the neighborhood, Hindle (1990) suggested choosing words that occur in some sort of grammatical relation or dependency to the target words. Hindle suggested that nouns which bear the same grammatical relation to the same verb might be similar. For example, the words tea, water, and beer are all frequent direct objects of the verb drink. The words senate, congress, panel, and legislature all tend to be subjects of the verbs consider, vote, and approve.

Hindle's intuition follows from the early work of Harris (1968), who suggested that:

The meaning of entities, and the meaning of grammatical relations among them, is related to the restriction of combinations of these entities relative to other entities.

There have been a wide variety of realizations of Hindle's idea since then. In general, in these methods each sentence in a large corpus is parsed and a dependency parse is extracted. We saw in Ch. 12 lists of grammatical relations produced by dependency parsers, including noun-verb relations like subject, object, indirect object, and noun-noun relations like genitive, ncomp, and so on. A sentence like the following would result in the set of dependencies shown here:

(20.32) I discovered dried tangerines:

discover (subject I) I (subj-of discover)

tangerine (obj-of discover) tangerine (adj-mod dried)

dried (adj-mod-of tangerine)

Since each word can be in a variety of different dependency relations with other words, we’ll need to augment the feature space. Each feature is now a pairing of a word and a relation, so instead of a vector of $N$ features, we have a vector of $N \times R$ features, where $R$ is the number of possible relations. Fig. 20.10 shows a schematic example of such a vector, taken from Lin (1998a), for the word cell. As the value of each attribute we have shown the frequency of the feature co-occurring with cell; the next section will discuss the use of what values and weights to use for each attribute.

Since full parsing is very expensive, it is common to use a chunker or shallow parser of the type defined in Sec. ??, with the goal of extracting only a smaller set of relations like subject, direct object, and prepositional object of a particular preposition (Curran, 2003).