8.1.1 Sentence Tokenization
We saw two examples above where sentence tokenization is difficult because sentence boundaries are not always indicated by periods, and can sometimes be indicated by punctuation like colons. An additional problem occurs when an abbreviation ends a sentence, in which case the abbreviation-final period is playing a dual role:
(8.2) He said the increase in credit limits helped B.C. Hydro achieve record net income of about $1 billion during the year ending March 31.
(8.3) Cousins, however, was insistent that all debts will be collected: “We continue to pursue monies owing and we expect to be paid for electricity we have sold.”
(8.4) The group included Dr. J. M. Freeman and T. Boone Pickens Jr.
A key part of sentence tokenization is thus period disambiguation; we've seen a simple perl script for period disambiguation in Ch. 3. Most sentence tokenization algorithms are slightly more complex than this deterministic algorithm, and in particular are trained by machine learning methods rather than being hand-built. We do this by hand-labeling a training set with sentence boundaries, and then using any supervised machine learning method (decision trees, logistic regression, SVM, etc) to train a classifier to mark the sentence boundary decisions.
More specifically, we could start by tokenizing the input text into tokens separated by whitespace, and then select any token containing one of the three characters !, . or ? (or possibly also :). After hand-labeling a corpus of such tokens, then we train a classifier to make a binary decision (EOS (end-of-sentence) versus not-EOS) on these potential sentence boundary characters inside these tokens.
The success of such a classifier depends on the features that are extracted for the classification. Let's consider some feature templates we might use to disambiguate these candidate sentence boundary characters, assuming we have a small amount of training data, labeled for sentence boundaries:
- the prefix (the portion of the candidate token preceding the candidate)
- the suffix (the portion of the candidate token following the candidate)
• whether the prefix or suffix is an abbreviation (from a list)
the word preceding the candidate
the word following the candidate
• whether the word preceding the candidate is an abbreviation
• whether the word following the candidate is an abbreviation
Consider the following example:
(8.5) ANLP Corp. chairman Dr. Smith resigned.
Given these feature templates, the feature values for the period . in the word Corp. in (8.5) would be:
$$ \begin{aligned}&PreviousWord=ANLP\quad&NextWord=chairman\\&Prefix=Corp\quad&Suffix=NULL\\&PreviousWord Abbreviation=1&NextWord Abbreviation=0\end{aligned} $$
If our training set is large enough, we can also look for lexical cues about sentence boundaries. For example, certain words may tend to occur sentence-initially, or sentence-finally. We can thus add the following features:
- Probability[candidate occurs at end of sentence]
- Probability[word following candidate occurs at beginning of sentence]
Finally, while most of the above features are relatively language-independent, we can use language-specific features. For example, in English, sentences usually begin with capital letters, suggesting features like the following:
- case of candidate: Upper, Lower, AllCap, Numbers
- case of word following candidate: Upper, Lower, AllCap, Numbers
Similar, we can have specific subclasses of abbreviations, such as honorifics or titles (e.g., Dr., Mr., Gen.), corporate designators (e.g., Corp., Inc.), or month-names (e.g., Jan., Feb.).
Any machine learning method can be applied to train EOS classifiers. Logistic regression and decision trees are two very common methods; logistic regression may have somewhat higher accuracy, although we have instead shown an example of a decision tree in Fig. 8.3 because it is easier for the reader to see how the features are used.
