8.1.2 Non-Standard Words

The second step in text normalization is normalizing non-standard words. Non-standard words are tokens like numbers or abbreviations, which need to be expanded into sequences of English words before they can be pronounced.

What is difficult about these non-standard words is that they are often very ambiguous. For example, the number 1750 can be spoken in at least three different ways, depending on the context:

原书第 279 页

seventeen fifty: (in ‘The European economy in 1750’)

one seven five zero: (in 'The password is 1750')

seventeen hundred and fifty: (in '1750 dollars')

one thousand, seven hundred, and fifty: (in '1750 dollars')

Similar ambiguities occur for Roman numerals like IV, (which can be pronounced four, fourth, or as the letters I V (meaning ‘intravenous’)), or 2/3, which can be two thirds or February third or two slash three.

in addition to numbers, various non-standard words are composed or letters. In three types non-standard words include abbreviations, letter sequences, and acronyms. Abbreviations are generally pronounced by expanding them; thus Jan 1 is pronounced January first, and Wed is pronounced Wednesday. Letter sequences like UN, DVD, PC, and IBM are pronounced by pronouncing each letter in a sequence (IBM is thus pronounced ay b iy eh m). Acronyms like IKEA, MoMA, NASA, and UNICEF are pronounced as if they were words; MoMA is pronounced m ow m ax. Ambiguity occurs here as well; should Jan be read as a word (the name Jan) or expanded as the month January?

These different types of numeric and alphabetic non-standard words can be summarized in Fig. 8.4. Each of the types has a particular realization (or realizations). For example, a year NYER is generally read in the paired method, in which each pair of digits is pronounced as an integer (e.g., seventeen fifty for 1750), while a U.S. zip code NZIP is generally read in the serial method, as a sequence of single digits (e.g., nine four one one zero for 94110). The type BMONEY deals with the idiosyncracies of expressions like $3.2 billion, which must be read out with the word dollars at the end, as three point two billion dollars.

For the alphabetic NSWs, we have the class EXPN for abbreviations like N.Y. which are expanded, LSEQ for acronyms pronounced as letter sequences, and ASWD for acronyms pronounced as if they were words.

Dealing with non-standard words requires at least three steps: tokenization to separate out and identify potential non-standard words, classification to label them with a type from Fig. 8.4, and expansion to convert each type into a string of standard words.

In the tokenization step, we can tokenize the input by whitespace, and then assume that any word which is not in the pronunciation dictionary is a non-standard word. More sophisticated tokenization algorithms would also deal with the fact that some dictionaries already contain some abbreviations. The CMU dictionary, for example, contains abbreviated (and hence incorrect) pronunciations for st, mr, mrs, as well as day and month abbreviations like mon, tues, nov, dec, etc. Thus in addition to unseen words, we also need to label any of these acronyms and also single-character token as potential non-standard words. Tokenization algorithms also need to split words which are combinations of two tokens, like 2-car or RVing. Words can be split by simple heuristics, such as splitting at dashes, or at changes from lower-case to upper-case.

The next step is assigning a NSW type; many types can be detected with simple regular expressions. For example, NYER could be detected by the following regular expression:

/(1[89][0-9][0-9])|(20[0-9][0-9]/

Other classes might be harder to write rules for, and so a more powerful option is

原书第 280 页

| --- | --- | --- | --- |

| NUMBERS | NUM | number (cardinal) | 12, 45, 1/2, 0.6 |

Figure 8.4 Some types of non-standard words in text normalization, selected from Table 1 of Sproat et al. (2001); not listed are types for URLs, emails, and some complex uses of punctuation.

to use a machine learning classifier with many features.

To distinguish between the alphabetic ASWD, LSEQ and EXPN classes, for example we might want features over the component letters. Thus short, all-capital words (IBM, US) might be LSEQ, longer all-lowercase words with a single-quote (gov't, cap'n) might be EXPN, and all-capital words with multiple vowels (NASA, IKEA) might be more likely to be ASWD.

Another very useful features is the identity of neighboring words. Consider ambiguous strings like 3/4, which can be an NDATE march third or a num three-four NDATE might be preceded by the word on, followed by the word of, or have the word Monday somewhere in the surrounding words. By contrast, NUM examples might be preceded by another number, or followed by words like mile and inch. Similarly, Roman numerals like VII tend to be NORD (seven) when preceded by Chapter, part, or Act, but NUM (seventh) when the words king or Pope occur in the neighborhood. These context words can be chosen as features by hand, or can be learned by machine learning techniques like the decision list algorithm of Ch. 8.

We can achieve the most power by building a single machine learning classifier which combines all of the above ideas. For example, the NSW classifier of (Sproat et al., 2001) uses 136 features, including letter-based features like 'all-upper-case', 'has-two-vowels', 'contains-slash', and 'token-length', as well as binary features for the presence of certain words like Chapter, on, or king in the surrounding context. Sproat et al. (2001) also included a rough-draft rule-based classifier, which used handwritten regular expression to classify many of the number NSWs. The output of this rough-draft classifier was used as just another feature in the main classifier.

In order to build such a main classifier, we need a hand-labeled training set, in which each token has been labeled with its NSW category; one such hand-labeled data-base was produced by Sproat et al. (2001). Given such a labeled training set, we

原书第 281 页

can use any supervised machine learning algorithm to build the classifier.

Formally, we can model this task as the goal of producing the tag sequence T which is most probable given the observation sequence:

$$ T^{*}\ =\ \underset{T}{\operatorname{argmax}}P(T|O) $$

One way to estimate this probability is via decision trees. For example, for each observed token $ o_i $, and for each possible NSW tag $ t_j $, the decision tree produces the posterior probability $ P(t_j | o_i) $. If we make the incorrect but simplifying assumption that each tagging decision is independent of its neighbors, we can predict the best tag sequence $ \hat{T} = \arg\max_T P(T | O) $ using the tree:

$$ \begin{align*}\hat T~=~\underset{T}{\argmax}P(T|O)\\\approx\underset{i=1}{\overset{m}{\prod}}\underset{t}{\argmax}P(t|o_i)\end{align*} $$

The third step in dealing with NSWs is expansion into ordinary words. One NSW type, EXPN, is quite difficult to expand. These are the abbreviations and acronyms like NY. Generally these must be expanded by using an abbreviation dictionary, with any ambiguities dealt with by the homonym disambiguation algorithms discussed in the next section.

Expansion of the other NSW types is generally deterministic. Many expansions are trivial; for example, LSEQ expands to a sequence of words, one for each letter, ASWD expands to itself, NUM expands to a sequence of words representing the cardinal number, NORD expands to a sequence of words representing the ordinal number, and NDIG and NZIP both expand to a sequence of words, one for each digit.

Other types are slightly more complex; NYER expands to two pairs of digits, unless the year ends in 00, in which case the four years are pronounced as a cardinal number (2000 as two thousand) or in the hundreds method (e.g., 1800 as eighteen hundred). NTEL can be expanded just as a sequence of digits; alternatively, the last four digits can be read as paired digits, in which each pair is read as an integer. It is also possible to read them in a form known as trailing unit, in which the digits are read serially until the last nonzero digit, which is pronounced followed by the appropriate unit (e.g., 876-5000 as eight seven six five thousand). The expansion of NDATE, MONEY, and NTIME is left as exercises (8.1)-(8.4) for the reader.

Of course many of these expansions are dialect-specific. In Australian English, the sequence 33 in a telephone number is generally read double three. Other languages also present additional difficulties in non-standard word normalization. In French or German, for example, in addition to the above issues, normalization may depend on morphological properties. In French, the phrase 1 fille ('one girl') is normalized to une fille, but 1 garçon ('one boy') is normalized to un garçon. Similarly, in German Heinrich IV ('Henry IV') can be normalized to Heinrich der Vierte, Heinrich des Vierten, Heinrich dem Vierten, or Heinrich den Vierten depending on the grammatical case of the noun (Demberg, 2006).

原书第 282 页