23.1.1 The Vector Space Model

In the vector space model of information retrieval, documents and queries are represented as vectors of features representing the terms (words) that occur within the collection (Salton, 1971).

The value of each feature is called the term weight and is usually a function of the term's frequency in the document, along with other factors.

For example, in a fried chicken recipe we found on the Web the four terms chicken, fried, oil, and pepper occur with term frequencies 8, 2, 7, and 4, respectively. So if we just used simple term frequency as our weights, and assuming we pretended only these

4 words occurred in the collection and we put the features are in the above order, the vector for this document (call it j) would be:

$$ \vec{d}_{j}=\left(8,2,7,4\right) $$

More generally, we represent a vector for a document $ d_{j} $ a

$$ \vec{d}_{j}=(w_{1,j},w_{2,j},w_{3,j},\cdots,w_{n,j}) $$

where $ \vec{d}_{j} $ denotes a particular document, and the vector contains a weight feature for each of the N terms that occur in the collection as a whole; $ w_{2,j} $ thus refers to the weight that term 2 has in document j.

We can also represent a query in the same way. For example, a query q for fried chicken would have the representation:

$$ \vec{q}=\left(1,1,0,0\right) $$

More generally,

$$ \vec{q}=\left(w_{1,q},w_{2,q},w_{3,q},\cdots,w_{n,q}\right) $$

Note that N, the number of dimensions in the vector, is the total number of terms in the whole collection. This can be hundreds of thousands of words, even if (as is often done) we don't consider some function words in the set of possible terms. But of course a query or even a long document can't contain very many of these hundreds of thousands of terms. Thus most of the values of the query and document vectors will be zero. Thus in practice we don't actually store all the zeros (we use hashes and other sparse representations).

Now consider a different document, a recipe for poached chicken; here the counts are:

$$ \overrightarrow{d}_{k}=(6,0,0,0) $$

Intuitively we’d like the query q fried chicken to match document $ d_j $ (the fried chicken recipe) rather than document $ d_k $ (the poached chicken recipe). A brief glance at the feature suggests that this might be the case; both the query and the fried chicken recipe have the words fried and chicken, while the poached chicken recipe is missing the word fried.

It is useful to view the features used to represent documents and queries in this model as dimensions in a multi-dimensional space, where the feature weights serve to locate documents in that space. When a user's query is translated into a vector it denotes a point in that space. Documents that are located close to the query can then be judged as being more relevant than documents that are farther away.

Fig. 23.3 shows a graphical illustration, plotting the first two dimensions (chicken and fried) for all three vectors. Note that if we measure the similarity between vectors by the angle between the vectors, that $q$ is more similar to $d_{j}$ than to $d_{k}$, because the angle between $q$ and $d_{j}$ is smaller.

In vector-based information retrieval we standardly use the cosine metric that we

introduced in Ch. 20 rather than the actual angle. We measure the distance between two documents by the cosine of the angle between their vectors. When two documents are identical they will receive a cosine of one; when they are orthogonal (share no common terms) they will receive a cosine of zero. The equation for cosine is:

$$ s i m(\vec{q},\vec{d}_{j})=\frac{\sum_{i=1}^{N}w_{i,q}\times w_{i,j}}{\sqrt{\sum_{i=1}^{N}w_{i,q}^{2}}\times\sqrt{\sum_{i=1}^{N}w_{i,j}^{2}}} $$

Recall from Ch. 20 that another way to think of the cosine ias as the normalized dot product. That is, the cosine is the dot product between the two vectors divided by the lengths of each of the two vectors. This is because the numerator of the cosine is the dot product:

$$ dot-product(\overrightarrow{x},\overrightarrow{y})=\overrightarrow{x}\cdot\overrightarrow{y}=\sum_{i=1}^{N}x_{i}\times u_{i} $$

while the denominator of the cosine contains terms for the lengths of the two vectors; recall that vector length is defined as:

$$ |\overrightarrow{x}|=\sqrt{\sum_{i=1}^{N}x_{i}^{2}} $$

This characterization of documents and queries as vectors provides all the basic parts for an ad hoc retrieval system. A document retrieval system can simply accept a user's query, create a vector representation for it, compare it against the vectors representing all known documents, and sort the results. The result is a list of documents rank ordered by their similarity to the query.

A further note on representation; the characterization of documents as vectors of term weights allows us to view the document collection as a whole as a (sparse) matrix of weights, where $ w_{i,j} $ represents the weight of term i in document j. This weight matrix is typically called a term-by-document matrix. Under this view, the columns of the matrix represent the documents in the collection, and the rows represent the terms. The term-by-document matrix for the two recipe documents above (again using only the raw term frequency counts as the term weights) would be: