23.1.2 Term Weighting

$$ \boldsymbol{A}=\left(\begin{array}{cc}8&6\\ 2&0\\ 7&0\\ 4&0\end{array}\right) $$

In the examples above, we assumed that the term weights were set as the simple frequency counts of the terms in the documents. This is a simplification of what we do in practice. The method used to assign terms weights in the document and query vectors has an enormous impact on the effectiveness of a retrieval system. Two factors have proven to be critical in deriving effective term weights. We have already seen the first, the term frequency, in its simplest form the raw frequency of a term within a document (Luhn, 1957). This reflects the intuition that terms that occur frequently within a document may reflect its meaning more strongly than terms that occur less frequently and should thus have higher weights.

The second factor is used to give a higher weight to words that only occur in a few documents. Terms that are limited to a few documents are useful for discriminating those documents from the rest of the collection, while terms that occur frequently across the entire collection aren't as helpful. Documents. The inverse document frequency or IDF term weight (Sparck Jones, 1972) is one way of assigning higher weights to these more discriminative words. IDF is defined via the fraction $ N/n_i $, where N is the total number of documents in the collection, and $ n_i $ is the number of documents in which term i occurs. The fewer documents a term occurs in, the higher this weight. The lowest weight of 1 is assigned to terms that occur in all the documents. Due to the large number of documents in many collections, this measure is usually squashed with a log function. The resulting definition for inverse document frequency (IDF) is thus:

$$ idf_{i}=log\left(\frac{N}{n_{i}}\right) $$

Combining term frequency with IDF results in a scheme known as tf-idf weighting:

$$ w_{i,j}=\mathrm{t f}_{i,j}\times\mathrm{i d f}_{i} $$

In tf-idf weighting, the weight of term $i$ in the vector for document $j$ is the product of its overall frequency in $j$ with the log of its inverse document frequency in the collection (sometimes the term frequency is logged as well). Tf-idf thus prefers words which are frequent in the current document $j$ but rare overall in the collection. Let's repeat

the cosine formula for query-document comparison with tf-idf weights added. We'll modify the formula slightly, since as we noted earlier, most values for any query or document vector will be zero. This means that in practice we don't compute the cosine by iterating over all the (mostly zero) dimensions. Instead we only compute over the words that are present, as suggested by the following equation for the tf-idf weighted cosine between a query q and a document d:

$$ sim(\vec{q},\vec{d})=\frac{\sum\limits_{w\in q,d}tf_{w,q}tf_{w,d}(idf_{w})^{2}}{\sqrt{\sum\limits_{q_{i}\in q}(tf_{q_{i},q}idf_{q_{i}})^{2}}\times\sqrt{\sum\limits_{d_{i}\in d}(tf_{d_{i},d}idf_{d_{i}})^{2}}} $$

With some minor variations, this tf-idf weighting scheme is used to assign term weights to documents in nearly all vector space retrieval models. The tf-idf scheme is also used in many other aspects of language processing; we'll see it again when we introduce summarization on page 31.