8.2.3 Grapheme-to-Phoneme

Once we have expanded non-standard words and looked them all up in a pronunciation dictionary, we need to pronounce the remaining, unknown words. The process of converting a sequence of letters into a sequence of phones is called grapheme-to-phoneme conversion, sometimes shortened g2p. The job of a grapheme-to-phoneme algorithm is thus to convert a letter string like cake into a phone string like [K EY K].

The earliest algorithms for grapheme-to-phoneme conversion were rules written by hand using the Chomsky-Halle phonological rewrite rule format of Eq. ?? in Ch. 7. These are often called letter-to-sound or LTS rules, and they are still used in some systems. LTS rules are applied in order, with later (default) rules only applying if the context for earlier rules are not applicable. A simple pair of rules for pronouncing the letter c might be as follows:

$$ \begin{array}{l}c\rightarrow[k]/{\mathrm{\bigsqseq\mathrm{\{a,o\}V}}};{\mathrm{context-dependent}}\\c\rightarrow[s];{\mathrm{context-independent}}\end{array} $$

Actual rules must be much more complicated (for example c can also be pronounced [ch] in cello or concerto). Even more complex are rules for assigning stress, which are famously difficult for English. Consider just one of the many stress rules from Allen et al. (1987), where the symbol X represents all possible syllable onsets:

$$ V\to[+\mathrm{stress}]/X\triangleq\mathrm{C}^{*}\setminus\{\mathrm{V}_{\mathrm{short}}\mathrm{C}\mathrm{C?}|\mathrm{V}\}\setminus\{\mathrm{V}_{\mathrm{short}}\mathrm{C}^{*}|\mathrm{V}\} $$

This rule represents the following two situations:

1. Assign 1-stress to the vowel in a syllable preceding a weak syllable followed by a morpheme-final syllable containing a short vowel and 0 or more consonants (e.g. difficult)

2. Assign 1-stress to the vowel in a syllable preceding a weak syllable followed by a morpheme-final vowel (e.g. oregano)

While some modern systems still use such complex hand-written rules, most systems achieve higher accuracy by relying instead on automatic or semi-automatic methods based on machine learning. This modern probabilistic grapheme-to-phoneme problem was first formalized by Lucassen and Mercer (1984). Given a letter sequence L, we are searching for the most probable phone sequence P:

$$ \hat{P}=\underset{P}{\operatorname{a r g m a x}}P(P|L) $$

The probabilistic method assumes a training set and a test set; both sets are lists of words from a dictionary, with a spelling and a pronunciation for each word. The next subsections show how the popular decision tree model for estimating this probability $ P(P|L) $ can be trained and applied to produce the pronunciation for an unseen word.

Finding a letter-to-phone alignment for the training set

Most letter-to-phone algorithms assume that we have an alignment, which tells us which phones align with each letter. We’ll need this alignment for each word in the training set. Some letters might align to multiple phones (e.g., x often aligns to k s), while other letters might align with no phones at all, like the final letter of cake in the following alignment:

L: c a k e

| | |

P: K EY K ε

One method for finding such a letter-to-phone alignment is the semi-automatic method of (Black et al., 1998). Their algorithm is semi-automatic because it relies on a hand-written list of the allowable phones that can realize each letter. Here are allowables lists for the letters c and e:

c: k ch s sh t-s ε

e: ih iy er ax ah eh ey uw ay ow y-uw oy aa ε

In order to produce an alignment for each word in the training set, we take this allowables list for all the letters, and for each word in the training set, we find all alignments between the pronunciation and the spelling that conform to the allowables list. From this large list of alignments, we compute, by summing over all alignments for all words, the total count for each letter being aligned to each phone (or multiphone or ε). From these counts we can normalize to get for each phone $p_i$ and letter $l_j$ a probability $P(p_i|l_j)$:

$$ P(p_{i}|l_{j})=\frac{\mathrm{count}(p_{i},l_{j})}{\mathrm{count}(l_{j})} $$

We can now take these probabilities and realign the letters to the phones, using the Viterbi algorithm to produce the best (Viterbi) alignment for each word, where the probability of each alignment is just the product of all the individual phone/letter alignments.

In this way we can produce a single good alignment A for each particular pair $ (P,L) $ in our training set.

Choosing the best phone string for the test set

Given a new word w, we now need to map its letters into a phone string. To do this, we'll first train a machine learning classifier, like a decision tree, on the aligned training set. The job of the classifier will be to look at a letter of the word and generate the most probable phone.

What features should we use in this decision tree besides the aligned letter $l_{i}$ itself? Obviously we can do a better job of predicting the phone if we look at a window of surrounding letters; for example consider the letter $a$. In the word cat, the $a$ is pronounce AE. But in our word cake, $a$ is pronounced EY, because cake has a final $e$; thus knowing whether there is a final $e$ is a useful feature. Typically we look at the $k$ previous letters and the $k$ following letters.

Another useful feature would be the correct identity of the previous phone. Knowing this would allow us to get some phonotactic information into our probability model.

Of course, we can't know the true identity of the previous phone, but we can approximate this by looking at the previous phone that was predicted by our model. In order to do this, we'll need to run our decision tree left to right, generating phones one by one.

In summary, in the most common decision tree model, the probability of each phone $ p_{i} $ is estimated from a window of k previous and k following letters, as well as the most recent k phones that were previously produced.

Fig. 8.7 shows a sketch of this left-to-right process, indicating the features that a decision tree would use to decide the letter corresponding to the letter s in the word Jurafsky. As this figure indicates, we can integrate stress prediction into phone prediction by augmenting our set of phones with stress information. We can do this by having two copies of each vowel (e.g., AE and AE1), or possibly even the three levels of stress AE0, AE1, and AE2, that we saw in the CMU lexicon. We'll also want to add other features into the decision tree, including the part-of-speech tag of the word (most part-of-speech taggers provide an estimate of the part-of-speech tag even for unknown words) and facts such as whether the previous vowel was stressed.

In addition, grapheme-to-phoneme decision trees can also include other more sophisticated features. For example, we can use classes of letters (corresponding roughly to consonants, vowels, liquids, and so on). In addition, for some languages, we need to know features about the following word. For example French has a phenomenon called liaison, in which the realization of the final phone of some words depends on whether there is a next word, and whether it starts with a consonant or a vowel. For example the French word six can be pronounced [sis] (in j'en veux six 'I want six'), [siz] (six enfants 'six children'), [si] (six filles 'six girls').

Finally, most synthesis systems build two separate grapheme-to-phoneme decision trees, one for unknown personal names and one for other unknown words. For pronouncing personal names it turns out to be helpful to use additional features that indicate which foreign language the names originally come from. Such features could be the output of a foreign-language classifier based on letter sequences (different languages have characteristic letter N-gram sequences).

The decision tree is a conditional classifier, computing the phoneme string that has the highest conditional probability given the grapheme sequence. More recent grapheme-to-phoneme conversion makes use of a joint classifier, in which the hidden state is a combination of phone and grapheme called a graphone; see the end of the chapter for references.

8.3 PROSODIC ANALYSIS

PROSODY

The final stage of linguistic analysis is prosodic analysis. In poetry, the word prosody refers to the study of the metrical structure of verse. In linguistics and language processing, however, we use the term prosody to mean the study of the intonational and rhythmic aspects of language. More technically, prosody has been defined by Ladd (1996) as the ‘use of suprasegmental features to convey sentence-level pragmatic meanings’. The term suprasegmental means above and beyond the level of the segment or phone, and refers especially to the uses of acoustic features like F0 duration, and

energy independently of the phone string.

By sentence-level pragmatic meaning, Ladd is referring to a number of kinds of meaning that have to do with the relation between a sentence and its discourse or external context. For example, prosody can be used to mark discourse structure or function, like the difference between statements and questions, or the way that a conversation is structured into segments or subdialogs. Prosody is also used to mark saliency, such as indicating that a particular word or phrase is important or salient. Finally, prosody is heavily used for affective and emotional meaning, such as expressing happiness, surprise, or anger.

In the next sections we will introduce the three aspects of prosody, each of which is important for speech synthesis: prosodic prominence, prosodic structure and tune. Prosodic analysis generally proceeds in two parts. First, we compute an abstract representation of the prosodic prominence, structure and tune of the text. For unit selection synthesis, this is all we need to do in the text analysis component. For diphone and HMM synthesis, we have one further step, which is to predict duration and F0 values from these prosodic structures.