{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "acc7afa6",
   "metadata": {},
   "source": [
    "# Evaluating Classifiers\n",
    "\n",
    "This chapter takes the classifiers of Chapter 14 into practice. You will build nearest neighbor classifiers with scikit-learn, look past accuracy to the kinds of mistakes a classifier makes, price those mistakes in dollars, and fit a logistic regression whose decision threshold can be moved to match the business costs. The chapter uses the same real online-shopping sessions as Chapter 14.\n",
    "\n",
    "- **Classifiers in scikit-learn**: `KNeighborsClassifier`, `StandardScaler`, and pipelines carry out the nearest neighbor workflow with `fit` and `predict`, and a validation set helps choose $k$.\n",
    "- **Confusion Matrix**: Counting true and false positives and negatives shows which mistakes a classifier makes, which accuracy alone hides.\n",
    "- **Precision and Recall**: Precision is the share of flagged cases that are real; recall is the share of real cases that get flagged.\n",
    "- **Cost of Errors**: When a missed case and a false alarm cost different amounts, the best classifier is the one with the lowest total cost, not always the most accurate one.\n",
    "- **Logistic Regression**: A classifier that predicts probabilities, so the decision threshold can be moved to match the business costs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2969cbac",
   "metadata": {},
   "source": [
    "<h2>Chapter Video</h2>\n",
    "\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube-nocookie.com/embed/Kdsp6soqA7o\" title=\"Machine Learning Fundamentals: The Confusion Matrix\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen referrerpolicy=\"strict-origin-when-cross-origin\"></iframe>\n",
    "\n",
    "<div class=\"video-credit\">\n",
    "<p><strong>Video credit:</strong> StatQuest with Josh Starmer. (2018, October 29). <em>Machine Learning Fundamentals: The Confusion Matrix</em>. YouTube. <a href=\"https://youtube.com/watch?v=Kdsp6soqA7o\">Watch on YouTube</a> (7:13).</p>\n",
    "<p class=\"video-rights\">This embedded video is provided for educational reference only. All rights are retained by the respective copyright holder.</p>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970d1def",
   "metadata": {},
   "source": [
    "<h2>Learning Objectives</h2>\n",
    "\n",
    "By the end of this chapter, you will be able to:\n",
    "\n",
    "1. Build a k-nearest neighbor classifier with scikit-learn, scaling features with training-set statistics only.\n",
    "2. Combine scaling and a classifier in a pipeline and choose k with a validation set.\n",
    "3. Evaluate a classifier with a baseline, a confusion matrix, precision, recall, and F1.\n",
    "4. Compare classifiers by the business cost of their errors.\n",
    "5. Fit a logistic regression, interpret its predicted probabilities and coefficient signs, and choose a decision threshold by cost."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9513cfd2",
   "metadata": {},
   "source": [
    "<h2>Chapter Flow</h2>\n",
    "\n",
    "```{div} chapter-flow\n",
    "{doc}`Nearest Neighbors with scikit-learn <1501-knn-scikit-learn>`\n",
    "\n",
    "{doc}`Evaluating Classifiers <1502-evaluating-classifiers>`\n",
    "\n",
    "{doc}`Logistic Regression <1503-logistic-regression>`\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d293ca18",
   "metadata": {},
   "source": [
    "<h2>Glossary</h2>\n",
    "\n",
    "| # | Term | Explanation |\n",
    "|---:|---|---|\n",
    "| 1 | Scaling (StandardScaler) | Putting features on a common scale; scikit-learn's `StandardScaler` converts features to standard units using the training set's means and SDs. |\n",
    "| 2 | Pipeline | A scikit-learn object that chains steps, such as scaling and a classifier, so they are fitted and applied together. |\n",
    "| 3 | Hyperparameter | A setting chosen by the analyst rather than learned by `fit`, such as k in k-nearest neighbors. |\n",
    "| 4 | Imbalanced classes | A data set in which one class is much more common than the other. |\n",
    "| 5 | Baseline classifier | A classifier that ignores the features and always predicts the majority class; the accuracy any real classifier must beat. |\n",
    "| 6 | Majority class | The most common label in the training set. |\n",
    "| 7 | Positive class | The class a classifier is trying to detect, such as churn, default, or fraud. |\n",
    "| 8 | Confusion matrix | A table comparing predicted labels with true labels. |\n",
    "| 9 | True positive | A positive case correctly predicted as positive. |\n",
    "| 10 | True negative | A negative case correctly predicted as negative. |\n",
    "| 11 | False positive | A case incorrectly predicted as belonging to a class. |\n",
    "| 12 | False negative | A case incorrectly predicted as not belonging to a class. |\n",
    "| 13 | Precision | Of the cases predicted positive, the fraction that really are positive: TP / (TP + FP). |\n",
    "| 14 | Recall | Of the cases that really are positive, the fraction predicted positive: TP / (TP + FN). |\n",
    "| 15 | F1 score | The harmonic mean of precision and recall. |\n",
    "| 16 | Cost of errors | The business cost assigned to each false positive and false negative, used to compare classifiers by total cost. |\n",
    "| 17 | Logistic regression | A classifier that predicts the probability of the positive class from a weighted sum of the features. |\n",
    "| 18 | Logistic (sigmoid) function | The S-shaped curve 1 / (1 + e^(-z)) that turns any number into a probability between 0 and 1. |\n",
    "| 19 | Predicted probability | A classifier's estimate of how likely a case is to belong to the positive class. |\n",
    "| 20 | Decision threshold | The probability cutoff at or above which a case is predicted positive; 0.5 by default. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3f1188",
   "metadata": {},
   "source": [
    "<h2><a href=\"overview.html\" target=\"_blank\" rel=\"noopener noreferrer\" style=\"color: var(--pst-color-link, #176de8);\">Lecture Slides</a></h2>\n"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}
