23.2.4 Evaluation of Factoid Answers

A wide variety of techniques have been employed to evaluate question answering systems. By far the most influential evaluation framework has been provided by the TREC Q/A track first introduced in 1999.

The primary measure used in TREC is an intrinsic or in vitro evaluation metric known as mean reciprocal rank, or MRR. As with the ad hoc information retrieval task described in Sec. 23.1, MRR assumes a test set of questions that have been human-labeled with correct answers. MRR also assumes that systems are returning a short ranked list of answers, or passages containing answers. Each question is then scored based on the reciprocal of the rank of the first correct answer. For example if the system returned 5 answers but the first 3 are wrong and hence the highest-ranked correct answer is ranked 4, the reciprocal rank score for that question would be $ \frac{1}{4} $. Questions with return sets that do not contain any correct answers are assigned a zero. The score of a system is then the average of the score for each question in the set. More formally, for an evaluation of a system returning M ranked answers for test set consisting of N questions, the MRR is defined as:

(23.24)

23.3 SUMMARIZATION

$$ MRR=\frac{\sum_{i=1}^{N}\frac{1}{rank_{i}}}{N} $$

The algorithms we have described so far in this chapter present the user an entire document (information retrieval), or a short factoid answer phrase (factoid question answering). But sometimes the user wants something that lies in between these extremes: something like a $ \underline{\text{summary}} $ of a document or set of documents.

outlines of any document

Text summarization is the process of distilling the most important information from a text to produce an abridged version for a particular task and user (definition adapted from Mani and Maybury (1999)). Important kinds of summaries that are the focus of current research include:

• abstracts of a scientific article

headlines of a news article

snippets summarizing a web page on a search engine results page

• action items or other summaries of a (spoken) business meeting

summaries of email threads

• compressed sentences for producing simplified or compressed text

• answers to complex questions, constructed by summarizing multiple documents

These kinds of summarization goals are often characterized by their position on two dimensions:

• single document versus multiple document summarization

• generic summarization versus query-focused summarization

In single document summarization we are given a single document and produce a summary. Single document summarization is thus used in situations like producing a headline or an outline, where the final goal is to characterize the content of a single document.

In multiple document summarization, the input is a group of documents, and our goal is to produce a condensation of the content of the entire group. We might use multiple document summarization when we are summarizing a series of news stories on the same event, or whenever we have web content on the same topic that we'd like to synthesize and condense.

A generic summary is one in which we don't consider a particular user or a particular information need; the summary simply gives the important information in the document(s). By contrast, in query-focused summarization, also called focused summarization, topic-based summarization and user-focused summarization, the summary is produced in response to a user query. We can think of query-focused summarization as a kind of longer, non-factoid answer to a user question.

In the remainder of this section we give a brief overview of the architecture of automatic text summarization systems; the following sections then give details.

One crucial architectural dimension for text summarizers is whether they are producing an abstract or an extract. The simplest kind of summary, an extract, is formed by selecting (extracting) phrases or sentences from the document to be summarized and pasting them together. By contrast, an abstract uses different words to describe the contents of the document. We’ll illustrate the difference between an extract and an abstract using the well-known Gettysburg address, a famous speech by Abraham Lincoln, shown in Fig. 23.12. $ ^{1} $ Fig. 23.13 shows an extractive summary from the speech followed by an abstract of the speech.

Fourscore and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field as a final resting-place for those who here gave their lives that this nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we cannot dedicate...we cannot consecrate...we cannot hallow... this ground. The brave men, living and dead, who struggled here, have consecrated it far above our poor power to add or detract. The world will little note nor long remember what we say here, but it can never forget what they did here. It is for us, the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us...that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion; that we here highly resolve that these dead shall not have died in vain; that this nation, under God, shall have a new birth of freedom; and that government of the people, by the people, for the people, shall not perish from the earth.

Most current text summarizers are extractive, since extraction is much easier than abstracting; the transition to more sophisticated abstractive summarization is a key goal of recent research.

Text summarization systems and, as it turns out, natural language generation systems as well, are generally described by their solutions to the following three problems:

1. Content Selection: What information to select from the document(s) we are summarizing. We usually make the simplifying assumption that the granularity of extraction is the sentence or clause. Content selection thus mainly consists of choosing which sentences or clauses to extract into the summary.

2. Information Ordering: How to order and structure the extracted units.

3. Sentence Realization: What kind of clean up to perform on the extracted units so they are fluent in their new context.

In the next sections we'll show these components in three summarization tasks: single document summarization, multiple document summarization, and query-focused summarization.