6.6.2 Logistic regression

Linear regression is what we want when we are predicting a real-valued outcome. But somewhat more commonly in speech and language processing we are doing classification, in which the output y we are trying to predict takes on one from a small set of discrete values.

Consider the simplest case of binary classification, where we want to classify whether some observation x is in the class (true) or not in the class (false). In other words, y can only take on the values 1 (true) or 0 (false), and we'd like a classifier that can take features of x and return true or false. Furthermore, instead of just returning the 0 or 1 value, we'd like a model that can give us the probability that a particular observation is in class 0 or 1. This is important because in most real-world tasks we're passing the results of this classifier onto some further classifier to accomplish some task. Since we are rarely completely certain about which class an observation falls in, we'd prefer not to make a hard decision at this stage, ruling out all other classes. Instead, we'd like to pass on to the later classifier as much information as possible: the entire set of classes, with the probability value that we assign to each class.

Could we modify our linear regression model to use it for this kind of probabilistic classification? Suppose we just tried to train a linear model to predict a probability as follows:

$$ \begin{aligned}P(y=true|x)&=\sum_{i=0}^{N}w_{i}\times f_{i}\\&=w\cdot f\end{aligned} $$

We could train such a model by assigning each training observation the target value y = 1 if it was in the class (true) and the target value y = 0 if it was not (false). Each observation x would have a feature vector f, and we would train the weight vector w to minimize the predictive error from 1 (for observations in the class) or 0 (for observations not in the class). After training, we would compute the probability of a class given an observation by just taking the dot product of the weight vector with the features for that observation.

The problem with this model is that there is nothing to force the output to be a legal probability, i.e. to lie between zero and 1. The expression $ \sum_{i=0}^{N} w_i \times f_i $ produces values from $ -\infty $ to $ \infty $. How can we fix this problem? Suppose that we keep our linear predictor $ w \cdot f $, but instead of having it predict a probability, we have it predict a ratio of two probabilities. Specifically, suppose we predict the ratio of the probability of being in the class to the probability of not being in the class. This ratio is called the odds. If an event has probability .75 of occurring and probability .25 of not occurring, we say

the odds of occurring is .75/.25 = 3. We could use the linear model to predict the odds of y being true:

LOGISTIC REGRESSION

$$ \frac{p(y=true)|x}{1-p(y=true|x)}=w\cdot f $$

This last model is close: a ratio of probabilities can lie between 0 and $ \infty $. But we need the left-hand side of the equation to lie between $ -\infty $ and $ \infty $. We can achieve this by taking the natural log of this probability:

$$ \ln\left(\frac{p(y=t r u e|x)}{1-p(y=t r u e|x)}\right)=w\cdot f $$

Now both the left and right hand lie between $ -\infty $ and $ \infty $. This function on the left (the log of the odds) is known as the logit function:

$$ \mathrm{logit}\left(p(x)\right)=\ln\left(\frac{p(x)}{1-p(x)}\right) $$

The model of regression in which we use a linear function to estimate, not the probability, but the logit of the probability, is known as logistic regression. If the linear function is estimating the logit, what is the actual formula in logistic regression for the probability $ P(y = true) $? You should stop here and take Equation (6.58) and apply some simple algebra to solve for the probability $ P(y = true) $.

Hopefully when you solved for $ P(y = true) $ you came up with a derivation something like the following:

$$ \begin{aligned}&\ln\left(\frac{p(y=true|x)}{1-p(y=true|x)}\right)=w\cdot f\\&\frac{p(y=true|x)}{1-p(y=true|x)}=e^{w\cdot f}\\&p(y=true|x)=(1-p(y=true|x))e^{w\cdot f}\\&p(y=true|x)=e^{w\cdot f}-p(y=true|x)e^{w\cdot f}\\&p(y=true|x)+p(y=true|x)e^{w\cdot f}=e^{w\cdot f}\\&p(y=true|x)(1+e^{w\cdot f})=e^{w\cdot f}\end{aligned} $$

$$ p(y=true|x)=\frac{e^{w\cdot f}}{1+e^{w\cdot f}} $$

Once we have this probability, we can easily state the probability of the observation not belonging to the class, $ p(y = \text{false}|x) $, as the two must sum to 1:

$$ p(y=f a l s e|x)=\frac{1}{1+e^{w\cdot f}} $$

Here are the equations again using explicit summation notation:

$$ p(y=\mathrm{true}|x)~=~\frac{\exp(\sum_{i=0}^{N}w_{i}f_{i})}{1+\exp(\sum_{i=0}^{N}w_{i}f_{i})} $$

LOGISTIC FUNCTION

$$ p(y=\mathrm{f a l s e}|x)~=~\frac{1}{1+\exp(\sum_{i=0}^{N}w_{i}f_{i})} $$

We can express the probability $ P(y = true | x) $ in a slightly different way, by dividing the numerator and denominator in (6.61) by $ e^{-w \cdot f} $:

$$ \begin{array}{l}p(y=true|x)=\frac{e^{w\cdot f}}{1+e^{w\cdot f}}\\ =\frac{1}{1+e^{-w\cdot f}}\end{array} $$

These last equation is now in the form of what is called the logistic function, (the function that gives logistic regression its name). The general form of the logistic function is:

$$ \frac{1}{1+e^{-x}} $$

The logistic function maps values from $-\infty$ and $\infty$ to lie between 0 and 1

Again, we can express $P(y=\text{false}|x)$ so as to make the probabilities sum to one:

$$ p(y=false|x)~=~\frac{e^{-w\cdot f}}{1+e^{-w\cdot f}} $$