23.1.3 Term Selection and Creation

Thus far, we have been assuming that it is precisely the words that occur in a collection that are used to index the documents in the collection. Two common variations on this assumption involve the use of stemming, and a stop list.

Stemming, as we discussed in Ch. 3, is the process of collapsing the morphological variants of a word together. For example, without stemming, the terms process, processing and processed will be treated as distinct items with separate term frequencies in a term-by-document matrix; with stemming they will be conflated to the single term process with a single summed frequency count. The major advantage to using stemming is that it allows a particular query term to match documents containing any of the morphological variants of the term. The Porter stemmer (Porter, 1980) described in Ch. 3 is frequently used for retrieval from collections of English documents.

A problem with this approach is that it throws away useful distinctions. For example, consider the use of the Porter stemmer on documents and queries containing the words stocks and stockings. In this case, the Porter stemmer reduces these surface forms to the single term stock. Of course, the result of this is that queries concerning stock prices will return documents about stockings, and queries about stockings will find documents about stocks. Additionally, we probably don't want to stem, e.g., the word Illustrator to illustrate, since the capitalized form Illustrator tends to refer to the software package Most modern web search engines, therefore need to use more sophisticated methods for stemming.

A second common technique involves the use of stop lists, which address the issue of what words should be allowed into the index. A stop list is simply a list of high frequency words that are eliminated from the representation of both documents and queries. Two motivations are normally given for this strategy: high frequency, closed-class terms are seen as carrying little semantic weight and are thus unlikely to help with retrieval, and eliminating them can save considerable space in the inverted index files used to map from terms to the documents that contain them. The downside of using a stop list is that it makes it difficult to search for phrases that contain words in the

stop list. For example, a common stop list presented in Frakes and Baeza-Yates (1992), would reduce the phrase to be or not to be to the phrase not.