8.3.7 Final result of text analysis: Internal Representation

The final output of text analysis is what we called the internal representation of the input text sentence. For unit selection synthesis, the internal representation can be as simple as a phone string together with indications of prosodic boundaries and prominent syllables, as shown in Fig. 8.1. For diphone synthesis as well as non-concatenative synthesis algorithms the internal representation must also include a duration and an F0 value for each phone.

Fig. 8.13 shows some sample TTS output from the FESTIVAL (Black et al., 1999) diphone speech synthesis system for the sentence Do you really want to see all of it?. This output, together with the F0 values shown in Fig. 8.14 would be the input

| do | you | H $ ^{{*}} $ | want | to | see | L $ ^{{*}} $ | L-H% | | | | | | | | | | | |

| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |

| d | uw | y | uw | r | ih | l | iy | w | aa | n | t | t | ax | s | all | of | it | |

| 110 | 110 | 50 | 50 | 75 | 64 | 57 | 82 | 57 | 50 | 72 | 41 | 43 | 47 | 54 | 130 | 1 | ah | vh |

As was suggested above, determining the proper prosodic pattern for a sentence is difficult, as real-world knowledge and semantic information is needed to know which syllables to accent, and which tune to apply. This sort of information is difficult to extract from the text and hence prosody modules often aim to produce a “neutral declarative” version of the input text, which assume the sentence should be spoken in a default way with no reference to discourse history or real-world events. This is one of the main reasons why intonation in TTS often sounds “wooden”.

8.4 DIPHONE WAVEFORM SYNTHESIS

We are now ready to see how the internal representation can be turned into a waveform. We will present two kinds of concatenative synthesis: diphone synthesis in this section, and unit selection synthesis in the next section.

Recall that for diphone synthesis, our internal representation is as shown in Fig. 8.13 and Fig. 8.14, consisting of a list of phones, each phone associated with a duration and a set of F0 targets.

The diphone concatenative synthesis model generates a waveform from a sequence of phones by selecting and concatenating units from a prerecorded database of diphones. A diphone is a phone-like unit going from roughly the middle of one phone to

the middle of the following phone. Diphone concatenative synthesis can be characterized by the following steps:

Training:

1. Record a single speaker saying an example of each diphone.

2. Cut each diphone out from the speech and store all diphones in a diphone database.

Synthesis:

1. Take from the database a sequence of diphones that corresponds to the desired phone sequence.

2. Concatenate the diphones, doing some slight signal processing at the boundaries

3. Use signal processing to change the prosody (f0, duration) of the diphone sequence to the desired prosody.

We tend to use diphones rather than phones for concatenative synthesis because of the phenomenon of coarticulation. In Ch. 7 we defined coarticulation as the movement of articulators to anticipate the next sound, or perseverating movement from the last sound. Because of coarticulation, each phone differs slightly depending on the previous and following phone. This if we just concatenated phones together, we would have very large discontinuities at the boundaries.

In a diphone, we model this coarticulation by including the transition to the next phone inside the unit. The diphone [w-eh], for example, includes the transition from the [w] phone to the [eh] phone. Because a diphone is defined from the middle of one phone to the middle of the next, when we concatenate the diphones, we are concatenating the middle of phones, and the middle of phones tend to be less influenced by the context. Fig. ?? shows the intuition that the beginning and end of the vowel [eh] have much more movement than the center.