StatQuest(ML) ROC and AUC

Published: by Creative Commons Licence

Before start, you have to know the confusion matrix and sensitivity and specificity, if you're not already down, click here and learn them.

For example, if we want to classify the mice are obese or not. When we use Logistic Regression, we can get the probability that a mouse is obese based on the weight. However, if we want to classify the mice as obese or not obese. then we need a way to turn probabilities into classifications. One way to classify mice is to set a threshold at 0.5, we can use a formula to describe the process:

\[Classification = \begin{cases} obese,\ if\ probability>0.5\\ not\ obese,\ if\ probability\ge 0.5 \end{cases}\]

Also, the threshold can be modify to get better performance when testing.

|

But it's not a good idea to make one confusion matrix for each threshold that mattered, because it would result in a confusingly large number of confusion matrices. So instead of being overwhelmed with confusion matrices, Receiver Operator Characteristic (ROC) graphs provide a simple way to summarize all of the information.

ROC(Receiver Operator Characteristic)

The y-axis shows the True Positive Rate, which is the same thing as Sensitivity, and it tells us what proportion of obese samples were correctly classified.

if you want to know about Sensitivity, click here.

\[True\ Positive\ Rate=Sensitivity= \frac{True\ Positives}{True\ Positives\ +\ False\ Negatives}\]

True Positives: the samples that were correctly classified as Positives;

False Negatives: the samples that were incorrectly classified as Negatives.

The x-axis shows the False Positive Rate, which is the same things as 1-Specificity.

if you want to know about Specificity, click here

\[False\ Positive\ Rate = (1\ -\ Specificity) = \frac{False\ Positives}{False\ Positives\ +\ True\ Negatives}\]

The ROC graph summarizes all of the confusion matrices that each threshold produced.

AUC(Area Under the Curve)

AUC is the area under the curve connecting ROC. The larger the AUC is, the better the method is.

Although ROC graphs are drawn using True Positive Rates and False Positives Rates to summarize confusion matrices, there are other metrics that attempt to do the same thing.

For example, people often replace the False Positive Rate (x-axis) with Precision.

\[Precision=\frac{True\ Positives}{True\ Positives\ +\ False\ Positives}\]

Precision is the proportion of positive results that were correctly classified.

If there were lots of samples that were not obese relative to the number of obese samples, then Precision might be more useful than the False Position. This is because Precision does not include the number of True Negatives in its calculation, and is not effected by the imbalance. In practice, this sort of imbalance occurs when studying a rare disease. In this case, the study will contain many more people without the disease than with the disease.

  • ROC curves make it easy to identify the best threshold for making decision.
  • AUC can help you to decide which categorization method is better.