Evaluating Classifiers

15. Evaluating Classifiers#

This chapter takes the classifiers of Chapter 14 into practice. You will build nearest neighbor classifiers with scikit-learn, look past accuracy to the kinds of mistakes a classifier makes, price those mistakes in dollars, and fit a logistic regression whose decision threshold can be moved to match the business costs. The chapter uses the same real online-shopping sessions as Chapter 14.

  • Classifiers in scikit-learn: KNeighborsClassifier, StandardScaler, and pipelines carry out the nearest neighbor workflow with fit and predict, and a validation set helps choose \(k\).

  • Confusion Matrix: Counting true and false positives and negatives shows which mistakes a classifier makes, which accuracy alone hides.

  • Precision and Recall: Precision is the share of flagged cases that are real; recall is the share of real cases that get flagged.

  • Cost of Errors: When a missed case and a false alarm cost different amounts, the best classifier is the one with the lowest total cost, not always the most accurate one.

  • Logistic Regression: A classifier that predicts probabilities, so the decision threshold can be moved to match the business costs.

Chapter Video

Video credit: StatQuest with Josh Starmer. (2018, October 29). Machine Learning Fundamentals: The Confusion Matrix. YouTube. Watch on YouTube (7:13).

This embedded video is provided for educational reference only. All rights are retained by the respective copyright holder.

Learning Objectives

By the end of this chapter, you will be able to:

  1. Build a k-nearest neighbor classifier with scikit-learn, scaling features with training-set statistics only.

  2. Combine scaling and a classifier in a pipeline and choose k with a validation set.

  3. Evaluate a classifier with a baseline, a confusion matrix, precision, recall, and F1.

  4. Compare classifiers by the business cost of their errors.

  5. Fit a logistic regression, interpret its predicted probabilities and coefficient signs, and choose a decision threshold by cost.

Chapter Flow

Glossary

#

Term

Explanation

1

Scaling (StandardScaler)

Putting features on a common scale; scikit-learn’s StandardScaler converts features to standard units using the training set’s means and SDs.

2

Pipeline

A scikit-learn object that chains steps, such as scaling and a classifier, so they are fitted and applied together.

3

Hyperparameter

A setting chosen by the analyst rather than learned by fit, such as k in k-nearest neighbors.

4

Imbalanced classes

A data set in which one class is much more common than the other.

5

Baseline classifier

A classifier that ignores the features and always predicts the majority class; the accuracy any real classifier must beat.

6

Majority class

The most common label in the training set.

7

Positive class

The class a classifier is trying to detect, such as churn, default, or fraud.

8

Confusion matrix

A table comparing predicted labels with true labels.

9

True positive

A positive case correctly predicted as positive.

10

True negative

A negative case correctly predicted as negative.

11

False positive

A case incorrectly predicted as belonging to a class.

12

False negative

A case incorrectly predicted as not belonging to a class.

13

Precision

Of the cases predicted positive, the fraction that really are positive: TP / (TP + FP).

14

Recall

Of the cases that really are positive, the fraction predicted positive: TP / (TP + FN).

15

F1 score

The harmonic mean of precision and recall.

16

Cost of errors

The business cost assigned to each false positive and false negative, used to compare classifiers by total cost.

17

Logistic regression

A classifier that predicts the probability of the positive class from a weighted sum of the features.

18

Logistic (sigmoid) function

The S-shaped curve 1 / (1 + e^(-z)) that turns any number into a probability between 0 and 1.

19

Predicted probability

A classifier’s estimate of how likely a case is to belong to the positive class.

20

Decision threshold

The probability cutoff at or above which a case is predicted positive; 0.5 by default.

Lecture Slides