8.4.1 Building a diphone database

There are six steps in building a diphone database:

1. Create a diphone inventory

2. Recruit a speaker

3. Create a text for the speaker to read for each diphone

4. Record the speaker reading each diphone

5. Segment, label, and pitch-mark the diphones

6. Excise the diphones

What is the inventory of diphones that we need for a system? If we have 43 phones (like the AT&T diphone system of Olive et al. (1998)), there are $ 43^2 = 1849 $ hypothetically possible diphone combinations. Not all of these diphones can actually occur. For example, the rules of English phonotactics rules out some combinations; phones like [h], [y], and [w] can only occur before vowels. In addition, some diphone systems don’t bother storing diphones if there is no possible coarticulation between the phones, such as across the silence between successive voiceless stops. The 43-phone system of Olive et al. (1998) thus has only 1162 diphones rather than the 1849 hypothetically possible set.

Next we recruit our speaker, often called a voice talent. The database of diphones for this speaker is called a voice; commercial systems often have multiple voices, such as one male and one female voice.

We’ll now create a text for the voice talent to say, and record each diphone. The most important thing in recording diphones is to keep them as consistent as possible; if possible, they should have constant pitch, energy, and duration, so they are easy to paste together without noticeable breaks. We do this by enclosing each diphone to be recorded in a carrier phrase. By putting the diphone in the middle of other phones, we keep utterance-final lengthening or initial phone effects from making any diphone louder or quieter than the others. We’ll need different carrier phrases for consonant-vowel, vowel-consonant, phone-silence, and silence-phone sequences. For example, a consonant vowel sequence like [b aa] or [b ae] could be embedded between the syllables [t aa] and [m aa]:

pause t aa b aa m aa pause

pause t aa b ae m aa pause

pause t aa b eh m aa pause

If we have an earlier synthesizer voice lying around, we usually use that voice to read the prompts out loud, and have our voice talent repeat after the prompts. This is another way to keep the pronunciation of each diphone consistent. It is also very important to use a high quality microphone and a quiet room or, better, a studio sound booth.

Once we have recorded the speech, we need to label and segment the two phones that make up each diphone. This is usually done by running a speech recognizer in forced alignment mode. In forced alignment mode, a speech recognition is told exactly what the phone sequence is; its job is just to find the exact phone boundaries.

in the waveform. Speech recognizers are not completely accurate at finding phone boundaries, and so usually the automatic phone segmentation is hand-corrected.

We now have the two phones (for example [b aa]) with hand-corrected boundaries. There are two ways we can create the /b-aa/ diphone for the database. One method is to use rules to decide how far into the phone to place the diphone boundary. For example, for stops, we put place the diphone boundary 30% of the way into the phone. For most other phones, we place the diphone boundary 50% into the phone.

A more sophisticated way to find diphone boundaries is to store the entire two phones, and wait to excise the diphones until we are know what phone we are about to concatenate with. In this method, known as optimal coupling, we take the two (complete, uncut) diphones we need to concatenate, and we check every possible cutting point for each diphones, choosing the two cutting points that would make the final frame of the first diphone acoustically most similar to the end frame of the next diphone (Taylor and Isard, 1991; Konkie and Isard, 1996). Acoustical similar can be measured by using cepstral similarity, to be defined in Sec. ??.