9.4.2 Gaussian PDFs

Vector quantization has the advantage of being extremely easy to compute and requires very little storage. Despite these advantages, vector quantization turns out not to be a good model of speech. A small number of codewords is insufficient to capture the wide variability in the speech signal. Speech is simply not a categorical, symbolic process.

Modern speech recognition algorithms therefore do not use vector quantization to compute acoustic likelihoods. Instead, they are based on computing observation probabilities directly on the real-valued, continuous input feature vector. These acoustic models are based on computing a probability density function or pdf over a continuous space. By far the most common method for computing acoustic likelihoods is the Gaussian Mixture Model (GMM) pdfs, although neural networks, support vector machines (SVMs) and conditional random fields (CRFs) are also used.

Let's begin with the simplest use of Gaussian probability estimators, slowly building up the more sophisticated models that are used.

Univariate Gaussians

The Gaussian distribution, also known as the normal distribution, is the bell-curve function familiar from basic statistics. A Gaussian distribution is a function parameterized by a mean, or average value, and a variance, which characterizes the average spread or dispersal from the mean. We will use $ \mu $ to indicate the mean, and $ \sigma^{2} $ to indicate the variance, giving the following formula for a Gaussian function:

$$ f(x|\mu,\sigma)=\frac{1}{\sqrt{2\pi\sigma^{2}}}exp(-\frac{(x-\mu)^{2}}{2\sigma^{2}}) $$

Recall from basic statistics that the mean of a random variable X is the expected value of X. For a discrete variable X, this is the weighted sum over the values of X (for a continuous variable, it is the integral):

$$ \mu=E(X)=\sum_{i=1}^{N}p(X_{i})X_{i} $$

The variance of a random variable X is the weighted squared average deviation from the mean:

$$ \sigma^{2}=E(X_{i}-E(X))^{2}=\sum_{i=1}^{N}p(X_{i})(X_{i}-E(X))^{2} $$

When a Gaussian function is used as a probability density function, the area under the curve is constrained to be equal to one. Then the probability that a random variable

takes on any particular range of values can be computed by summing the area under the curve for that range of values. Fig. 9.18 shows the probability expressed by the area under an interval of a Gaussian.

We can use a univariate Gaussian pdf to estimate the probability that a particular HMM state $j$ generates the value of a single dimension of a feature vector by assuming that the possible values of (this one dimension of the) observation feature vector $o_t$ are normally distributed. In other words we represent the observation likelihood function $b_j(o_t)$ for one dimension of the acoustic vector as a Gaussian. Taking, for the moment, our observation as a single real valued number (a single cepstral feature), and assuming that each HMM state $j$ has associated with it a mean value $\mu_j$ and variance $\sigma_j^2$, we compute the likelihood $b_j(o_t)$ via the equation for a Gaussian pdf:

$$ b_{j}(o_{t})=\frac{1}{\sqrt{2\pi\sigma_{j}^{2}}}exp\left(-\frac{(o_{t}-\mu_{j})^{2}}{2\sigma_{j}^{2}}\right) $$

Equation (9.26) shows us how to compute $ b_j(o_t) $, the likelihood of an individual acoustic observation given a single univariate Gaussian from state j with its mean and variance. We can now use this probability in HMM decoding.

But first we need to solve the training problem; how do we compute this mean and variance of the Gaussian for each HMM state $ q_i $? Let's start by imagining the simpler situation of a completely labeled training set, in which each acoustic observation was labeled with the HMM state that produced it. In such a training set, we could compute the mean of each state by just taking the average of the values for each $ o_t $ that corresponded to state i, as shown in (9.27). The variance could just be computed from the sum-squared error between each observation and the mean, as shown in (9.28).

$$ \begin{aligned}\hat{\mu}_{i}&=\frac{1}{T}\sum_{t=1}^{T}o_{t} s.t.q_{t} is state i\\\hat{\sigma}_{j}^{2}&=\frac{1}{T}\sum_{t=1}^{T}(o_{t}-\mu_{i})^{2}s.t.q_{t} is state i\end{aligned} $$

But since states are hidden in an HMM, we don’t know exactly which observation vector $ o_t $ was produced by which state. What we would like to do is assign each observation vector $ o_t $ to every possible state $ i $, prorated by the probability that the HMM was in state $ i $ at time $ t $. Luckily, we already know how to do this prorating; the probability of being in state $ i $ at time $ t $ was defined in Ch. 6 as $ \xi_t(i) $, and we saw how to compute $ \xi_t(i) $ as part of the Baum-Welch algorithm using the forward and backward probabilities. Baum-Welch is an iterative algorithm, and we will need to do the probability computation of $ \xi_t(i) $ iteratively since getting a better observation probability $ b $ will also help us be more sure of the probability $ \xi $ of being in a state at a certain time. Thus we give equations for computing an updated mean and variance $ \hat{\mu} $ and $ \hat{\sigma}^2 $:

$$ \hat{\mu}_{i}~=~\frac{\sum_{t=1}^{T}\xi_{t}(i)o_{t}}{\sum_{t=1}^{T}\xi_{t}(i)} $$

$$ \hat{\sigma}_{i}^{2}~=~\frac{\sum_{t=1}^{T}\xi_{t}(i)(o_{t}-\mu_{i})^{2}}{\sum_{t=1}^{T}\xi_{t}(i)} $$

Equations (9.29) and (9.30) are then used in the forward-backward (Baum-Welch) training

of the HMM. As we will see, the values of $ \mu_i $ and $ \sigma_i $ are first set to some initial estimate, which is then re-estimated until the numbers converge.

Multivariate Gaussians

Equation (9.26) shows how to use a Gaussian to compute an acoustic likelihood for a single cepstral feature. Since an acoustic observation is a vector of 39 features, we’ll need to use a multivariate Gaussian, which allows us to assign a probability to a 39-valued vector. Where a univariate Gaussian is defined by a mean $ \mu $ and a variance $ \sigma^2 $, a multivariate Gaussian is defined by a mean vector $ \bar{\mu} $ of dimensionality $ D $ and a covariance matrix $ \Sigma $, defined below. As we discussed in the previous section, for a typical cepstral feature vector in LVCSR, $ D $ is 39:

$$ f(\overrightarrow{x}|\overrightarrow{\mu},\Sigma)=\frac{1}{(2\pi)^{\frac{D}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}(x-\mu)^{\mathrm{T}}\Sigma^{-1}(x-\mu)\right) $$

The covariance matrix $ \Sigma $ captures the variance of each dimension as well as the covariance between any two dimensions.

Recall again from basic statistics that the covariance of two random variables X and Y is the expected value of the product of their average deviations from the mean:

$$ \Sigma=E[(X-E(X))(Y-E(Y)])=\sum_{i=1}^{N}p(X_{i}Y_{i})(X_{i}-E(X))(Y_{i}-E(Y)) $$

Thus for a given HMM state with mean vector $ \mu_j $ and covariance matrix $ \Sigma_j $, and a given observation vector $ o_t $, the multivariate Gaussian probability estimate is:

$$ b_{j}(o_{t})=\frac{1}{(2\pi)^{\frac{D}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}(o_{t}-\mu_{j})^{T}\Sigma_{j}^{-1}(o_{t}-\mu_{j})\right) $$

The covariance matrix $ \Sigma_j $ expresses the variance between each pair of feature dimensions. Suppose we made the simplifying assumption that features in different dimensions did not covary, i.e., that there was no correlation between the variances of different dimensions of the feature vector. In this case, we could simply keep a distinct variance for each feature dimension. It turns out that keeping a separate variance for each dimension is equivalent to having a covariance matrix that is diagonal, i.e. non-zero elements only appear along the main diagonal of the matrix. The main diagonal of such a diagonal covariance matrix contains the variances of each dimension, $ \sigma_1^2, \sigma_2^2, \ldots \sigma_D^2 $;

Let's look at some illustrations of multivariate Gaussians, focusing on the role of the full versus diagonal covariance matrix. We'll explore a simple multivariate Gaussian with only 2 dimensions, rather than the 39 that are typical in ASR. Fig. 9.19 shows three different multivariate Gaussians in two dimensions. The leftmost figure shows a Gaussian with a diagonal covariance matrix, in which the variances of the two dimensions are equal. Fig. 9.20 shows 3 contour plots corresponding to the Gaussians in Fig. 9.19; each is a slice through the Gaussian. The leftmost graph in Fig. 9.20 shows a slice through the diagonal equal-variance Gaussian. The slice is circular, since the variances are equal in both the X and Y directions.

The middle figure in Fig. 9.19 shows a Gaussian with a diagonal covariance matrix, but where the variances are not equal. It is clear from this figure, and especially from the contour slice show in Fig. 9.20, that the variance is more than 3 times greater in one dimension than the other.

The rightmost graph in Fig. 9.19 and Fig. 9.20 shows a Gaussian with a non-diagonal covariance matrix. Notice in the contour plot in Fig. 9.20 that the contour is not lined up with the two axes, as it is in the other two plots. Because of this, knowing the value in one dimension can help in predicting the value in the other dimension. Thus having a non-diagonal covariance matrix allows us to model correlations between the values of the features in multiple dimensions.

A Gaussian with a full covariance matrix is thus a more powerful model of acoustic likelihood than one with a diagonal covariance matrix. And indeed, speech recognition performance is better using full-covariance Gaussians than diagonal-covariance Gaussians. But there are two problems with full-covariance Gaussians that makes them

difficult to use in practice. First, they are slow to compute. A full covariance matrix has $ D^2 $ parameters, where a diagonal covariance matrix has only $ D $. This turns out to make a large difference in speed in real ASR systems. Second, a full covariance matrix has many more parameters and hence requires much more data to train than a diagonal covariance matrix. Using a diagonal covariance model means we can save room for using our parameters for other things like triphones (context-dependent phones) to be introduced in Sec. ??.

For this reason, in practice most ASR systems use diagonal covariance. We will assume diagonal covariance for the remainder of this section.

Equation (9.33) can thus be simplified to the version in (9.34) in which instead of a covariance matrix, we simply keep a mean and variance for each dimension. Equation (9.34) thus describes how to estimate the likelihood $ b_j(o_t) $ of a $ D $-dimensional feature vector $ o_t $ given HMM state $ j $, using a diagonal-covariance multivariate Gaussian.

$$ b_{j}(o_{t})=\prod_{d=1}^{D}\frac{1}{\sqrt{2\pi\sigma_{jd}^{2}}}exp\left(-\frac{1}{2}[\frac{(o_{td}-\mu_{jd})^{2}}{\sigma_{jd}^{2}}]\right) $$

Training a diagonal-covariance multivariate Gaussian is a simple generalization of training univariate Gaussians. We'll do the same Baum-Welch training, where we use the value of $ \tilde{\xi}_t(i) $ to tell us the likelihood of being in state $ i $ at time $ t $. Indeed, we'll use exactly the same equation as in (9.30), except that now we are dealing with vectors instead of scalars; the observation $ o_t $ is a vector of cepstral features, the mean vector $ \tilde{\mu} $ is a vector of cepstral means, and the variance vector $ \tilde{\sigma}_t^2 $ is a vector of cepstral variances.

$$ \hat{\mu}_{i}=\frac{\sum_{t=1}^{T}\xi_{t}(i)o_{t}}{\sum_{t=1}^{T}\xi_{t}(i)} $$

$$ \hat{\sigma}_{i}^{2}=\frac{\sum_{t=1}^{T}\xi_{t}(i)(o_{t}-\mu_{i})(o_{t}-\mu_{i})^{T}}{\sum_{t=1}^{T}\xi_{t}(i)} $$

Gaussian Mixture Models

The previous subsection showed that we can use a multivariate Gaussian model to assign a likelihood score to an acoustic feature vector observation. This models each dimension of the feature vector as a normal distribution. But a particular cepstral feature might have a very non-normal distribution; the assumption of a normal distribution may be too strong an assumption. For this reason, we often model the observation likelihood not with a single multivariate Gaussian, but with a weighted mixture of multivariate Gaussians. Such a model is called a Gaussian Mixture Model or GMM. Equation (9.37) shows the equation for the GMM function; the resulting function is the sum of M Gaussians. Fig. 9.21 shows an intuition of how a mixture of Gaussians can model arbitrary functions.

$$ f(x|\boldsymbol{\mu},\boldsymbol{\Sigma})=\sum_{k=1}^{M}c_{k}\frac{1}{\sqrt{2\pi|\boldsymbol{\Sigma}_{k}|}}exp[(\boldsymbol{x}-\boldsymbol{\mu}_{k})^{T}\boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu}_{k})] $$

Equation (9.38) shows the definition of the output likelihood function $ b_{j}(o_{t}) $

$$ b_{j}(o_{t})=\sum_{m=1}^{M}c_{jm}\frac{1}{\sqrt{2\pi|\Sigma_{jm}|}}exp[(x-\mu_{jm})^{T}\Sigma_{jm}^{-1}(o_{t}-\mu_{jm})] $$

Let's turn to training the GMM likelihood function. This may seem hard to do; how can we train a GMM model if we don't know in advance which mixture is supposed to account for which part of each distribution? Recall that a single multivariate Gaussian could be trained even if we didn't know which state accounted for each output, simply by using the Baum-Welch algorithm to tell us the likelihood of being in each state j at time t. It turns out the same trick will work for GMMs; we can use Baum-Welch to tell us the probability of a certain mixture accounting for the observation, and iteratively update this probability.

We used the $ \xi $ function above to help us compute the state probability. By analogy with this function, let's define $ \xi_{tm}(j) $ to mean the probability of being in state $ j $ at time $ t $ with the $ m $th mixture component accounting for the output observation $ o_t $. We can compute $ \xi_{tm}(j) $ as follows:

$$ \xi_{tm}(j)=\frac{\sum_{i=1}^{}N\alpha_{t-1}(j)a_{ij}c_{jm}b_{jm}(o_{t})\beta_{t}(j)}{\alpha_{T}(F)} $$

Now if we had the values of $\xi$ from a previous iteration of Baum-Welch, we can use $\xi_{tm}(j)$ to recompute the mean, mixture weight, and covariance using the following equations:

$$ \begin{array}{c c}{\hat{\mu}_{i m}\;=\;\frac{\sum_{t=1}^{T}\xi_{t m}(i)o_{t}}{\sum_{t=1}^{T}\sum_{m=1}^{M}\xi_{t m}(i)}}&{}\end{array} $$

$$ \begin{array}{c} \hat{c}_{im}~=~\frac{\sum_{t=1}^{T}\xi_{tm}(i)}{\sum_{t=1}^{T}\sum_{k=1}^{M}\xi_{tk}(i)}\end{array} $$

$$ \begin{array}{c c}{\hat{\Sigma}_{i m}\;=\;\frac{\sum_{t=1}^{T}\xi_{t}(i)(o_{t}-\mu_{i m})(o_{t}-\mu_{i m})^{T}}{\sum_{t=1}^{T}\sum_{k=1}^{M}\xi_{t m}(i)}}&{}\end{array} $$