6.6.3 Logistic regression: Classification

Given a particular observation, how do we decide which of the two classes ('true' or 'false') it belongs to? This is the task of classification, also called inference. Clearly the correct class is the one with the higher probability. Thus we can safely say that our observation should be labeled 'true' if:

$$ \begin{aligned}&p(y=true|x)>p(y=false|x)\\&\frac{p(y=true|x)}{p(y=false|x)}>1\\&\frac{p(y=true|x)}{1-p(y=true|x)}>1\\ \end{aligned} $$

and substituting from Eq. 6.60 for the odds ratio:

$$ \begin{aligned}&e^{w\cdot f}>1\\&w\cdot f>0\\ \end{aligned} $$

or with the explicit sum notation:

$$ \sum_{i=0}^{N}w_{i}f_{i}>0 $$

Thus in order to decide if an observation is a member of the class we just need to compute the linear function, and see if its value is positive; if so, the observation is in the class.

A more advanced point: the equation $ \sum_{i=0}^{N} w_i f_i = 0 $ is the equation of a hyperplane (a generalization of a line to $ N $ dimensions). The equation $ \sum_{i=0}^{N} w_i f_i > 0 $ is thus the part of $ N $-dimensional space above this hyperplane. Thus we can see the logistic regression function as learning a hyperplane which separates points in space which are in the class ('true') from points which are not in the class.