22.1.3 Evaluating Named Entity Recognition

The familiar metrics of recall, precision and $ F_{1} $ measure introduced in Ch. 13 are used to evaluate NER systems. Recall that recall is the ratio of the number of correctly labeled responses to the total that should have been labeled; precision is the ratio of the number of correctly labeled responses to the total labeled. The

F-measure (van Rijsbergen, 1975) provides a way to combine these two measures into a single metric. The F-measure is defined as:

$$ F_{\beta}=\frac{(\beta^{2}+1)P R}{\beta^{2}P+R} $$

The $ \beta $ parameter is used to differentially weight the importance of recall and precision, based perhaps on the needs of an application. Values of $ \beta > 1 $ favor recall,

while values of $ \beta < 1 $ favor precision. When $ \beta = 1 $, precision and recall are equally balanced; this is sometimes called $ F_{\beta=1} $ or just $ F_1 $:

$$ F_{1}=\frac{2PR}{P+R} $$

As with syntactic chunking, it is important to distinguish the metrics used to measure performance at the application level from those used during training. At the application level, recall and precision are measured with respect to the actual named entities detected. On the other hand, with an IOB encoding scheme the learning algorithms are attempting to optimize performance at the tag level. Performance at these two levels can be quite different; since the vast majority of tags in any given text are outside any entity, simply emitting an O tag for every token gives fairly high tag-level performance.

High-performing systems at recent standardized evaluations have entity level F-measures around .92 for PERSONS and LOCATIONS, and around .84 for ORGANIZATIONS (Sang and De Meulder, 2003).