5.7.1 Error Analysis

In order to improve any model we need to understand where it went wrong. Analyzing the error in a classifier like a part-of-speech tagger is done via a confusion matrix, or contingency table. A confusion matrix for an N-way classification task is an N-by-N matrix where the cell $ (x,y) $ contains the number of times an item with correct classification x was classified by the model as y. For example, the following table shows a portion of the confusion matrix from the HMM tagging experiments of Franz (1996). The row labels indicate correct tags, column labels indicate the tagger's hypothesized tags, and each cell indicates percentage of the overall tagging error. Thus 4.4% of the total errors were caused by mistagging a VBD as a VBN. Common errors are boldfaced.

The confusion matrix above, and related error analyses in Franz (1996), Kupiec (1992), and Ratnaparkhi (1996), suggest that some major problems facing current taggers are:

1. NN versus NNP versus JJ: These are hard to distinguish prenominally. Distinguishing proper nouns is especially important for information extraction and machine translation.

2. RP versus RB versus IN: All of these can appear in sequences of satellites immediately following the verb.

3. VBD versus VBN versus JJ: Distinguishing these is important for partial parsing (participles are used to find passives), and for correctly labeling the edges of noun-phrases.

Error analysis like this is a crucial part of any computational linguistic application. Error analysis can help find bugs, find problems in the training data, and, most important, help in developing new kinds of knowledge or algorithms to use in solving problems.

5.8 ADVANCED ISSUES IN PART-OF-SPEECH TAGGING