4.3.1 N-gram Sensitivity to the Training Corpus
The N-gram model, like many statistical models, is very dependent on the training corpus. One implication of this is that the probabilities often encode very specific facts about a given training corpus. Another implication is that N-grams do a better and better job of modeling the training corpus as we increase the value of N.
We can visualize both of these facts by borrowing the technique of Shannon (1951) and Miller and Selfridge (1950), of generating random sentences from different N-gram models. It's simplest to visualize how this works for the unigram case. Imagine all the words of English covering the probability space between 0 and 1, each word covering an interval equal to its frequency. We choose a random value between 0 and 1, and print out the word whose interval includes the real value we have chosen. We continue choosing random numbers and generating words until we randomly generate the sentence-final token . The same technique can be used to generate bigrams by first generating a random bigram that starts with (according to its bigram probability), then choosing a random bigram to follow it (again, according to its conditional probability), and so on.
To give an intuition for the increasing power of higher-order N-grams, Fig. 4.3 shows random sentences generated from unigram, bigram, trigram, and quadrigram models trained on Shakespeare's works.
| Unigram | ● To him swallowed confess hear both. Which. Of save on trail for are ay device and rote life have\n● Every enter now severally so, let\n● Hill he late speaks; or! a more to leg less first you enter\n● Are where exeunt and sighs have rise excellency took of.. Sleep knave we. near; vile like |
| Bigram | ● What means, sir. I confess she? then all sorts, he is trim, captain.\n● Why dost stand forth thy canopy, forsooth; he is this palpable hit the King Henry. Live king. Follow.\n● What we, hath got so she that I rest and sent to scold and nature bankrupt, nor the first gentleman?\n● Enter Menenius, if it so many good direction found'st thou art a strong upon command of fear not a liberal largess given away, Falstaff! Exeunt |
| Trigram | ● Sweet prince, Falstaff shall die. Harry of Monmouth's grave.\n● This shall forbid it should be branded, if renown made it empty.\n● Indeed the duke; and had a very good friend.\n● Fly, and will rid me these news of price. Therefore the sadness of parting, as they say, 'tis done. |
| Quadrigram | ● King Henry. What! I will go seek the traitor Gloucester. Exeunt some of the watch. A great banquet serv'd in;\n● Will you not tell me who I am?\n● It cannot be but so.\n● Indeed the short and the long. Marry, 'tis a noble Lepidus. |
The longer the context on which we train the model, the more coherent the sentences. In the unigram sentences, there is no coherent relation between words, nor any sentence-final punctuation. The bigram sentences have some very local word-to-word coherence (especially if we consider that punctuation counts as a word). The trigram
and quadrigram sentences are beginning to look a lot like Shakespeare. Indeed a careful investigation of the quadrigram sentences shows that they look a little too much like Shakespeare. The words It cannot be but so are directly from King John. This is because, not to put the knock on Shakespeare, his oeuvre is not very large as corpora go (N = 884,647,V = 29,066), and our N-gram probability matrices are ridiculously sparse. There are $ V^2 = 844,000,000 $ possible bigrams alone, and the number of possible quadrigrams is $ V^4 = 7 \times 10^{17} $. Thus once the generator has chosen the first quadrigram (It cannot be but), there are only five possible continuations (that, I, he, thou, and so); indeed for many quadrigrams there is only one continuation.
To get an idea of the dependence of a grammar on its training set, let's look at an N-gram grammar trained on a completely different corpus: the Wall Street Journal (WSJ) newspaper. Shakespeare and the Wall Street Journal are both English, so we might expect some overlap between our N-grams for the two genres. In order to check whether this is true, Fig. 4.4 shows sentences generated by unigram, bigram, and trigram grammars trained on 40 million words from WSJ.
| unigram: Months the my and issue of year foreign new exchange's september were recession exchange new endorsed a acquire to six executives |
| bigram: Last December through the way to preserve the Hudson corporation N. B. E. C. Taylor would seem to complete the major central planners one point five percent of U. S. E. has already old M. X. corporation of living on information such as more frequently fishing to keep her |
| trigram: They also point to ninety nine point six billion dollars from two hundred four oh six three percent of the rates of interest stores as Mexico and Brazil on market conditions |
Compare these examples to the pseudo-Shakespeare in Fig. 4.3. While superficially they both seem to model “English-like sentences” there is obviously no overlap whatsoever in possible sentences, and little if any overlap even in small phrases. This stark difference tells us that statistical models are likely to be pretty useless as predictors if the training sets and the test sets are as different as Shakespeare and WSJ.
How should we deal with this problem when we build N-gram models? In general we need to be sure to use a training corpus that looks like our test corpus. We especially wouldn't choose training and tests from different genres of text like newspaper text, early English fiction, telephone conversations, and web pages. Sometimes finding appropriate training text for a specific new task can be difficult; to build N-grams for text prediction in SMS (Short Message Service), we need a training corpus of SMS data. To build N-grams on business meetings, we would need to have corpora of transcribed business meetings.
For general research where we know we want written English but don't have a domain in mind, we can use a balanced training corpus that includes cross sections from different genres, such as the 1-million-word Brown corpus of English (Francis and
Kučera, 1982) or the 100-million-word British National Corpus (Leech et al., 1994).
Recent research has also studied ways to dynamically adapt language models to different genres; see Sec. 4.9.4.