# Precision, Recall and all that

Let us now consider a binary classification task. Our training data is denoted as the set ${\bf T}$, where

Our data vector ${\bf y}$ can have two outcomes, $y_k= 0 \text{ or } 1$. Let us call the outcome with a +1 value, be the positive value while the 0 is called the negative value.

## Definitions

For our binary classifier model trained on $T$ is denoted as $\hat{g}_T.$ The number of positive samples in $T$ is $N_p$, while the number of negative samples is $N_n$. The total number of samples is $N$,

Furthermore, we denote the estimates of $N_p$ and $N_n$ given by our classifier as $\hat{N}_p$ and $\hat{N}_n$, $\hat{N}_p = \text{ Number of positive predictions by classifier}, \\ \hat{N}_n = \text{ Number of negative predictions by classifier}, \\ \hat{N}_p+\hat{N}_n = N.$

With these definitions we have that the confusion matrix is defined as

when we divide $C$ by the number of samples, then we can give the confusion matrix a probabilistic interpretation

In the special case that our classifier doesn’t make any mistakes, the false negatives and positives are zero, therefore

In the other extreme where the classifier doesn’t make any correct classifications we have

And in the case where the classifier is totally random,

## Accuracy

How well did the classifier get the correct labels.

• a perfect classifier would have $a=1$.
• a terrible classifier has $a=0$
• a random classifier has $a=\frac{1}{2}$

## Precision

Of the samples, $\hat{N}_p$, that the classifier thought were positive, how many are actually correct ?

• a perfect classifier would have $p=1$.
• a terrible classifier has $p=0$
• a random classifier has $p=\frac{1}{2}$

## Recall

Recall, is the metric that measures the fraction of positively identified samples,

• a perfect classifier would have $r=1$.
• a terrible classifier has $r=0$
• a random classifier has $r=\frac{1}{2}$

## The F Metric

The $F_\beta$ metric is the following function of precision and recall.

clearly

In the case where $\beta$=1, then $F_1$ is the harmonic mean of the precision and recall

• a perfect classifier would have $F_\beta=1$,
• a terrible classifier has $F_\beta=0$,
• a random classifier has $F_\beta=\frac{1}{2}$.

## ROC curve

The Receiver operator curve is the plot generated when one plots the True positive rate $T_p$ vs the False positive rate $F_p$ for a classifier that depends on a parameter $\Lambda$.

Figure 1 : A sample ROC.

The area $\mathcal{A}$ of the ROC curve can be interpreted as the probability that a random sample $x$ such that $x \in P$ will be classified as a True positive, compared to a False Positive. The area also satisfies the following properties,

• a perfect classifier would have $\mathcal{A}=1$,
• a terrible classifier has $\mathcal{A}=0$,
• a random classifier has $\mathcal{A}=\frac{1}{2}$.

### Mathematical Details

Let us suppose that we have a classifier $g$ such that,

for a given $x$ and $\Lambda$. In addition, there exists distributions $\rho_p(x)$ and $\rho_n(x)$ that represent the true positive distribution and true negative distributions, respectively, that we are trying to distinguish with our classifier $g(x,\Lambda)$. With this classifier, we have the following values for the confusion matrix $C$,

In Fig 2, the solid blue area represents the $T_p(\Lambda)$ value, while the red area is the $F_p(\Lambda)$ values.

Figure 2: A schematic of the true negative and true positive distributions.

With these definitions, let us now compute the area of the ROC, $\mathcal{A}$. Note that since as $\Lambda \rightarrow \infty$, then $T_p \rightarrow 0$, and so we compute the area with the limits ranging from $% $,

# Connection between Accuracy, Recall and Precision

Having written down Recall and precision as conditional probabilities, it is much

By Bayes theorem we have that

In otherwords, we find that

Stack-Overflow