12.8.1 Disfluencies and Repair

Perhaps the most salient syntactic feature that distinguishes spoken and written language is the class of phenomena known individual as disfluencies and collectively as the phenomenon of repair.

Disfluencies include the use of the words uh and um, word repetitions, restarts, and word fragments. The ATIS sentence in Fig. 12.18 shows examples of a restart and the use of uh. The restart here occurs when the speaker starts by asking for one-way flights, and then stops and corrects herself, restarting and asking about one-way fares.

The segment one-way flights is referred to as the reparandum, and the replacing sequence one-way fares is referred to as the repair. The repair is also called the fluent region. The interruption point, where the speaker breaks off the original word sequence, here occurs right after the word flights. In the editing phase we see what are often called edit terms, such as you know, I mean, uh, and um.

The words uh and um (sometimes called filled pauses or fillers) are generally treated like regular words in speech recognition lexicons and grammars.

Incomplete words like wha- and betwee- in Fig. 12.17 are known as fragments. Fragments are extremely problematic for speech recognition systems, since they are often incorrectly attached to previous or following words, resulting in word missegmentation.

Disfluencies are very common. One count in the Switchboard Treebank corpus found that 37% of the sentences with more than two words were disfluent in some way. Indeed, the word uh is one of the most frequent words in Switchboard.

For applications like speech understanding, where our goal is to build a meaning for the input sentence, it may be useful to detect these restarts in order to edit out what the speaker probably considered the “corrected” words. For example in the sentence above, if we could detect that there was a restart, we could just delete the reparandum, and parse the remaining parts of the sentence:

Does American airlines offer any $ \underline{\text{one-way flights uh}} $ one-way fares for 160 dollars?

How do disfluencies interact with the constituent structure of the sentence? Hindle (1983) showed that the repair often has the same structure as the constituent just before the interruption point. Thus in the example above, the repair is an NP, as is the reparam-dum. This means that if it is possible to automatically find the interruption point, it is also often possible to automatically detect the boundaries of the reparam-dum.

There are other interactions between disfluencies and syntactic structure. For example when there is a disfluency immediately after a subject NP, the repair always repeats the subject but not the preceding discourse marker. If the repair happens after an auxiliary or main verb, the verb and subject are (almost) always recycled together (Fox and Jasperson, 1995).