{ "cells": [ { "cell_type": "markdown", "id": "c000", "metadata": {}, "source": [ "# Evaluating Classifiers" ] }, { "cell_type": "code", "execution_count": 1, "id": "c001", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:13.674806Z", "iopub.status.busy": "2026-10-23T03:52:13.674693Z", "iopub.status.idle": "2026-10-23T03:52:15.208634Z", "shell.execute_reply": "2026-10-23T03:52:15.207919Z" }, "tags": [ "hide-input", "thebe-init", "runnable" ] }, "outputs": [], "source": [ "path_data = '../../data/'\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "import matplotlib.pyplot as plt\n", "plt.style.use('fivethirtyeight')\n", "%matplotlib inline\n", "\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.metrics import (accuracy_score, confusion_matrix, ConfusionMatrixDisplay,\n", " precision_score, recall_score, f1_score)\n", "\n", "# The churn data and the train-test split from Section 1404.\n", "churn = pd.read_csv(path_data + 'customer_churn.csv')\n", "features = ['tenure_months', 'monthly_charges', 'support_calls']\n", "X = churn[features]\n", "y = churn['churn']\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, random_state=14, stratify=y)" ] }, { "cell_type": "markdown", "id": "c002", "metadata": {}, "source": [ "```{contents}\n", ":local:\n", ":depth: 2\n", "```" ] }, { "cell_type": "markdown", "id": "c003", "metadata": {}, "source": [ "So far we have judged classifiers by one number, accuracy. Accuracy treats every mistake the same, but in business the two kinds of mistakes rarely cost the same. A bank that misses a fraudulent transaction loses the money; a bank that wrongly freezes a good customer's card loses some goodwill and an hour of an analyst's time. A subscription company that fails to spot a customer about to leave loses that customer's future payments; one that sends a retention discount to a customer who was going to stay anyway loses only the discount.\n", "\n", "This section introduces the standard tools for looking past accuracy: a baseline to compare against, the confusion matrix, precision and recall, and the cost of each kind of error. We continue with the customer churn data and the same training and test sets as Section 1404 (the setup cell above recreates them), using the 11-nearest neighbor classifier chosen there." ] }, { "cell_type": "code", "execution_count": 2, "id": "c004", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.209849Z", "iopub.status.busy": "2026-10-23T03:52:15.209771Z", "iopub.status.idle": "2026-10-23T03:52:15.216971Z", "shell.execute_reply": "2026-10-23T03:52:15.216621Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test accuracy: 0.853\n" ] } ], "source": [ "knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11))\n", "knn.fit(X_train, y_train)\n", "y_pred = knn.predict(X_test)\n", "\n", "print('Test accuracy:', round(accuracy_score(y_test, y_pred), 3))" ] }, { "cell_type": "markdown", "id": "c005", "metadata": {}, "source": [ "## Why Accuracy Is Not Enough\n", "\n", "```{index} baseline classifier\n", "```\n", "\n", "```{index} imbalanced classes\n", "```\n", "\n", "```{index} majority class\n", "```\n", "\n", "```{index} DummyClassifier\n", "```\n", "\n", "When one class is much more common than the other, the classes are *imbalanced*, and accuracy can look good even for a useless classifier. The simplest comparison is a **baseline classifier** that ignores the features and always predicts the *majority class*, the most common label in the training set. scikit-learn provides one as `DummyClassifier`." ] }, { "cell_type": "code", "execution_count": 3, "id": "c006", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.218068Z", "iopub.status.busy": "2026-10-23T03:52:15.217998Z", "iopub.status.idle": "2026-10-23T03:52:15.222086Z", "shell.execute_reply": "2026-10-23T03:52:15.221784Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Baseline predicts: 0 for every customer\n", "Baseline accuracy: 0.713\n", "kNN accuracy: 0.853\n" ] } ], "source": [ "baseline = DummyClassifier(strategy='most_frequent')\n", "baseline.fit(X_train, y_train)\n", "y_base = baseline.predict(X_test)\n", "\n", "print('Baseline predicts:', y_base[0], 'for every customer')\n", "print('Baseline accuracy:', round(accuracy_score(y_test, y_base), 3))\n", "print('kNN accuracy: ', round(accuracy_score(y_test, y_pred), 3))" ] }, { "cell_type": "markdown", "id": "c007", "metadata": {}, "source": [ "The baseline predicts \"stays\" (0) for everyone and is right about 71% of the time, because about 71% of the test customers stay. Our classifier's 85% accuracy is a real improvement, but the honest gain is the difference between the two, not the distance from 0%.\n", "\n", "With rarer events the problem gets worse. Suppose a card issuer reviews 1,000 transactions, of which 20 (2%) are fraudulent. A \"classifier\" that approves everything is 98% accurate and catches no fraud at all." ] }, { "cell_type": "code", "execution_count": 4, "id": "c008", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.223456Z", "iopub.status.busy": "2026-10-23T03:52:15.223383Z", "iopub.status.idle": "2026-10-23T03:52:15.225451Z", "shell.execute_reply": "2026-10-23T03:52:15.225153Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy of approving everything: 0.98\n", "Fraud cases caught: 0\n" ] } ], "source": [ "actual_fraud = np.array([1] * 20 + [0] * 980) # 1 = fraud, 0 = legitimate\n", "approve_all = np.zeros(1000, dtype=int) # predict \"legitimate\" for every transaction\n", "\n", "print('Accuracy of approving everything:', accuracy_score(actual_fraud, approve_all))\n", "print('Fraud cases caught:', np.sum((approve_all == 1) & (actual_fraud == 1)))" ] }, { "cell_type": "markdown", "id": "c009", "metadata": {}, "source": [ "Accuracy alone cannot tell this useless rule apart from a good fraud detector. We need to count the two kinds of errors separately." ] }, { "cell_type": "code", "execution_count": 5, "id": "c010", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.226404Z", "iopub.status.busy": "2026-10-23T03:52:15.226349Z", "iopub.status.idle": "2026-10-23T03:52:15.227963Z", "shell.execute_reply": "2026-10-23T03:52:15.227637Z" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### Exercise: Baseline for Loan Defaults\n", "# A lender's test set has 400 loans, of which 48 defaulted (label 1).\n", "# A classifier's predictions are summarized below.\n", "# 1. Compute the accuracy of always predicting \"repaid\" (0).\n", "# 2. Compute the classifier's accuracy: it was right on 362 of the 400 loans.\n", "# 3. Print how many percentage points the classifier beats the baseline by,\n", "# rounded to 1 decimal.\n", "# Print exactly:\n", "# Baseline accuracy: 0.88\n", "# Classifier accuracy: 0.905\n", "# Improvement (points): 2.5\n", "n_loans = 400\n", "n_default = 48\n", "n_correct = 362\n", "### Your code starts here.\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": 6, "id": "c011", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.228765Z", "iopub.status.busy": "2026-10-23T03:52:15.228712Z", "iopub.status.idle": "2026-10-23T03:52:15.230342Z", "shell.execute_reply": "2026-10-23T03:52:15.230032Z" }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Baseline accuracy: 0.88\n", "Classifier accuracy: 0.905\n", "Improvement (points): 2.5\n" ] } ], "source": [ "### Solution\n", "n_loans = 400\n", "n_default = 48\n", "n_correct = 362\n", "\n", "baseline_accuracy = (n_loans - n_default) / n_loans\n", "classifier_accuracy = n_correct / n_loans\n", "\n", "print('Baseline accuracy:', baseline_accuracy)\n", "print('Classifier accuracy:', classifier_accuracy)\n", "print('Improvement (points):', round(100 * (classifier_accuracy - baseline_accuracy), 1))" ] }, { "cell_type": "markdown", "id": "c012", "metadata": {}, "source": [ "## The Confusion Matrix\n", "\n", "```{index} confusion matrix\n", "```\n", "\n", "```{index} true positive\n", "```\n", "\n", "```{index} false positive\n", "```\n", "\n", "```{index} true negative\n", "```\n", "\n", "```{index} false negative\n", "```\n", "\n", "```{index} positive class\n", "```\n", "\n", "In a binary classification problem we call one class the **positive class**, usually the event we are trying to detect: churn, default, fraud. \"Positive\" does not mean good; it means \"the thing we are looking for\". Every prediction then falls into one of four groups:\n", "\n", "| | Predicted positive (churn) | Predicted negative (stays) |\n", "|---|---|---|\n", "| **Actually positive (churned)** | **True positive (TP)**: churner correctly flagged | **False negative (FN)**: churner missed |\n", "| **Actually negative (stayed)** | **False positive (FP)**: loyal customer flagged by mistake | **True negative (TN)**: loyal customer correctly left alone |\n", "\n", "A table of these four counts is called a **confusion matrix**, because it shows which classes the classifier confuses with each other. The correct predictions are on the diagonal (TN and TP); the two kinds of errors are off the diagonal.\n", "\n", "We can count the groups ourselves with pandas, using `crosstab` to cross-tabulate actual and predicted labels." ] }, { "cell_type": "code", "execution_count": 7, "id": "c013", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.231276Z", "iopub.status.busy": "2026-10-23T03:52:15.231220Z", "iopub.status.idle": "2026-10-23T03:52:15.240830Z", "shell.execute_reply": "2026-10-23T03:52:15.240226Z" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
predicted01
actual
019519
12561
\n", "
" ], "text/plain": [ "predicted 0 1\n", "actual \n", "0 195 19\n", "1 25 61" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.crosstab(y_test, y_pred, rownames=['actual'], colnames=['predicted'])" ] }, { "cell_type": "markdown", "id": "c014", "metadata": {}, "source": [ "scikit-learn's `confusion_matrix` returns the same counts as an array, with actual classes as rows and predicted classes as columns, in the order 0, 1. For two classes, `.ravel()` flattens it into the four counts in the order TN, FP, FN, TP." ] }, { "cell_type": "code", "execution_count": 8, "id": "c015", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.242369Z", "iopub.status.busy": "2026-10-23T03:52:15.242287Z", "iopub.status.idle": "2026-10-23T03:52:15.245544Z", "shell.execute_reply": "2026-10-23T03:52:15.245014Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[195 19]\n", " [ 25 61]]\n", "TN: 195 FP: 19 FN: 25 TP: 61\n" ] } ], "source": [ "cm = confusion_matrix(y_test, y_pred)\n", "print(cm)\n", "\n", "tn, fp, fn, tp = cm.ravel()\n", "print('TN:', tn, ' FP:', fp, ' FN:', fn, ' TP:', tp)" ] }, { "cell_type": "markdown", "id": "c016", "metadata": {}, "source": [ "`ConfusionMatrixDisplay` draws the matrix as a labeled picture, which is how it usually appears in reports." ] }, { "cell_type": "code", "execution_count": 9, "id": "c017", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.246874Z", "iopub.status.busy": "2026-10-23T03:52:15.246810Z", "iopub.status.idle": "2026-10-23T03:52:15.305502Z", "shell.execute_reply": "2026-10-23T03:52:15.305025Z" }, "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbkAAAGNCAYAAACMvw+TAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAANMVJREFUeJzt3Qd4U2X7x/G7rFI2ZS/ZewnIXsoUBGQJgiivL4gDGSogSxF9RUEUlKGoOFBQQRkKIiAKMgUR2UPKXrJaoKxC6f+6H/4nJl10hLR9+v1cV66E5CTnJA35nWf7hYSERAgAABZKk9QHAADAnULIAQCsRcgBAKxFyAEArEXIAQCsRcgBAKxFyAEArEXIAQCsRcgBAKxFyAEArEXIAQCslS6pDwDx1+DRCRJ84TIfHZLE38ve5JNHkoiIuCnXr1yK13MIuRRIA+7seUIOScPPjwogpBx8WwEA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1iLkAADWIuQAANYi5AAA1kqX1AcAJFS2zBklV84scunKNTl19uJttw/Mnln8M6STU+cuSnj4zWi3yZk9s+TIGhDtY2HXb8ixf0L4gyFOzodekbPBoZI5wF/y5c522+3PhoTKtbAbkjcwq6RLl5ZP2UsIOaQoBfJklwebVZNWDStL3WqlJH26tLJoxRbpMfijGJ/TvW0deeHxllKiSB7z7wuhV2TmD+vltSnfy5Vr1z22fe4/LaRfj6bRvs62vUel0SNvevkdwSbHT4XIgp83y+JV22Td5n1yI/ymtG5cRWaO7xPjc2Z+v07e/nSJHDh6xvw7a+aM8kjbOvJS33aSKWMGHx69nQg5pCiPta8vQ/u0NrevXA0zIReb/o82k9H925vbl6+GycVLVyVfrmzydLf7pHSxfPJQ/6nRPu/oyXMSdj080n3BXnsfsNOM+Wtk7EeLze2M/ulNyMXm3RnL5JVJC8ztAP/0JuC0puGDr1fI3wf/kTnvPSN+fn4+OXZbEXJIUY6fCpaps36Rn1Ztl5ALl+W3mUNj3DZHtkzyYp/Wpmpy4Jiv5Mvv15n7ixfOLZ++0Uua1a0gHZpXl3nL/ozy3C4D35ddQSfu6HuBfQrmzWlOoO5vWMl8/xr3GBvjtsHnL8nYD3+UNGn8ZOLwbtKjXV0TaPuPnJbHh02X5et3me9mxxY1fPoebJPqO54MGzZMGjRoIGFhYUn9t0AcfLFgnYyYMFdW/bFXboR7lrQiq3t3SVPds+CXza6AU1otNOSt2a6qTMBbHmtfT8Y830ka1Swr6dLGXsuw7q8gU13+YJNq8uiD9VwlNq1WHze4i7k964f1/HESKdWH3KFDh2T79u1y82bs1QpIebJludWB5PDxs1EeO3jsVvtHrcrFo32un/hJ/tzZJXfOLHf4KJFaaduwuqtgriiPFSuU21xv2HbA58dlG6orYa2TZ86b63rVSkV5rEH1Mq4g1F6X585f8nj8hw8GSGCOzOb2meCL8tWiDfLmtEWmXQ/wBj2JUms375OIiAiPtrfVf+4119qGrL0uc+XgZCuhrA25CxcuyJdffimrV6+W06dPS+7cueXuu++W3r17S65ct86cWrVqJTt27DC3mzZt6vqSdevWTfr27Wtur1mzRubOnSt///23XLlyRYoUKSJt27aVDh06eOxv0KBBsmHDBpk3b57r9d3179/flBjnz58v2bJli9PxIXF+37JfTp+7KLWqlJDpYx6X6d+uMmfPdaqWlOFPPmCGBGRIn06yZskYJeSyZw0wPeVy5cgsuXNmNT0uG1QvLa37TJCrkXpkAglRu2oJU1OwcdsB6TX8U+nVuaE56Vq/JUjGTFtkOlVdvxEuF0OvEnKJYGXIXb16VVq0aCG7d+/2uH/x4sXy/vvvy8GDB82/NeA0bJzbjhMnbnU4GDNmjIwbN87jNTZu3GhCb9myZTJ16r898xo2bCgff/yxfPHFFzJw4ECP5+j+NNA6duxoAi6ux4dEfg+uXZcBr8+Sz97sJR2b1zAXx8qNe0xPtuoVisqNG/+27e3Zf0K6vzBNflm/y4xZUi0bVJJ3hj0s1SrcJb0faiSTv1zOnwaJFpAxg7w7orv8Z+h0mffzn+biaHRPGVOK27zrMGPmEsnKkNPSkQZIxYoV5fnnn5eiRYvKuXPnZPPmzSaE3EPlxRdfNNsvX75c0qdPb+7Pk+fWeConjB588EEpUaKEpEmTRg4fPiwfffSRzJo1S/r06WNKX6pNmzZSuHBh+eSTT0ypTbd16H3a5vfUU0/F6/iQeIt/2yZNe74lTz58r1QpW1hu3oyQn9fulAmfLZVdi183AXcmONS1vY6fi2zJ6u3y7KtfytzJz0rTuuUJOXiNjqFb/vlgmfb1Ctm656jpadmsXgUzXrN8qxGSNm0a2oUTycqQCwi41eFAqxw7derkul8DSwPIoSGjJSvndsaMGaP0vNSS23fffWdKYqGhoabu/OLFW7Nr/PHHH66QS5s2rTzxxBMyatQoU8pr2bKlKyj1uffcc4+5xOf44B3b/z4m/V6b6XFf/eqlTdXQlt1HXCW22Ozef6t0nyNrJv4s8KrKZQrL5Jd7eNy3etNeU5LTEzMdb4eEszLk6tata0Jm5MiRsnPnTqlVq5ZUqlTJlMacgImLiRMnytixMY9z0dBz17NnT7O9Vls6IadVm1pKc38dbx0fbk/PhCNP4aVtHS/3bWtuz1680XW/nkVHRIg5kYmszX23TmYYEA5v0pqEyFN4aVvxq1N+MLe7tKrJB55IVoacVhV+8803snLlSlOqmj59umzZskVy5MghL7zwgjz66KO3fQ1tlxs/frwp6elzqlatap6vr60dTLSjSeQfQ328S5cuMmPGDNOuVqxYMbPv/PnzS/v27b16fKmVzj1ZMG8Oc7tQvpzmOlOAvxngra5cve7qVakG9mwuRQvmku9/+cvMYlKiSF4Z8Fgz0xkl6PAp+eS7Va5t9fXmTX5WPpu3RrbvPSYnz4RInsBs0ubeqvJ4pwZmm2+X/OHjd4yU1g6sHZbUsX9uzZBz+co1M8BbBWRMLwXy3Pr+qomfL5NDx89KuyZ3S+H8gbL/yCl5d8bPpjOKjpf7b6eGSfRO7GFlyDkaN25sLkrbxN544w3p16+f3HXXXa77nR6VkQNLO6KEh4ebTiQDBgzweEzb8mLy5JNPymeffWba4bSjyaZNm2TEiBGu9r74Hh88VSt/lyz++HmP++6rXU7+nPeKub1iw27p0Hfyvw9GiBloqxd3R06cM7OauPeU1BKfjk96bYBnz1nHx3N+kwXLN/MnQYy0o0jrJyZ43Ldiwx6p0XH0rf/zNcvK/Kn9/n3QT8xEBe6TFajC+XPK7IlPm84pSBwrQ+7HH380JamuXbtG2x1fg8cJkaxZs5rrPXv2uNrXlHbpd4YQaHBlznxrzJQOAdBqzJiUL1/evLa2wx0/flz8/f3l8ccfT/DxwZPOEOGcFUfnxKl/S3FqwudLTQmuffPqpkSnnUyWr9tpAuvSFc8xb3oGXq39KHmkXV2pUamYKdnp/Jg79x2Xb37cIKs3/c2fA7HS+SedWoXoFMh7a2yc47meLUwJbv6yP02JTocUNK1bwfTizZLJn0/bC/xCQkKiNkCkcB9++KEMGTLEFVZ60erH8+fPm5Kbhoy2i6lp06aZHpYaRsWLF5d06dKZcXLaiaRevXqyb98+yZQpk+k5efLkSTPkQJ+7bt0608nkueeei7J/Lenpa6ju3bt7DDWI7/FFp+KDr8vZ85e9+pkBcRW80a2kDPhQRMRNCbt8+2W1rJ/Wq3Xr1qbnog4FOHPmjOmurwGi7WraXuYeINr+pZ1Arl27ZrbTAdsaOBkyZDDbVqlSRS5fvix79+411ZcaakOHxjwpsNLX0/Y45QwbSOjxAQASzsqSnDudTSQkJETy5s0r2bN7VhW40220evHGjRsmfAoUKODxGhpCWprTYQbaq3L//v1mG2dMnTud7FnDUUuGsbXfxef43FGSQ1KiJIeUVJKzsk3OnYZQdEEUmfZs1EtcXiNLliwmxGKivTK1avPll1/22vEBAOLP+pDzJR3MrVWdR44cMaU87V0JAEg6hJwX6bCDS5cumR6TOgdl5BlUAAC+Rch50dKlS03vTO10or01AQBJi5DzIp3/EgCQfFg5hAAAAEXIAQCsRcgBAKxFyAEArEXIAQCsRcgBAKxFyAEArEXIAQCsRcgBAKxFyAEArEXIAQCsRcgBAKxFyAEArEXIAQCsFeeldrZu3SqjR49O1M6qVKkio0aNStRrAADg9ZA7d+6cLF++XBLjxo0biXo+AAB3JOQqVKgg06dPl8TImzdvop4PAMAdCTkNqE6dOsXrxQEASEp0PAEAWCvOJbnbOXjwoGzZskVCQkIkX758cv/993vrpQEASJqQCwoKkgEDBsjq1atd9zVu3NgVcidOnJAGDRpIwYIFZdWqVYndHQAAvqmu1IBr1qyZCbhcuXJJ1apVo2xToEABKVasmGzbts0MQwAAIEWE3NChQyU4OFhatmxpqipjGkfXtGlTc7106dLE7A4AAN+EnDNuzt/fXyZNmiRZsmSJcdvChQub6z179iR0dwAA+C7k9u3bJzdv3pRSpUrddvxb/vz5zbWW+gAASPYhpwGntCR3O6GhoeY6U6ZMCd0dAAC+C7lChQqZ6wMHDkh4eHis265fv95clyhRIqG7AwDAdyFXpEgRKVOmjKmCnDNnTozb7d+/X2bNmmVut2jRIqG7AwDAt70rhwwZYq4HDhxo5rXUgeCOs2fPysyZM6VJkyamurJhw4ZSr169xOwOAIB48QsJCYmQRNBhAxMmTPC4L126dB4rDmjnlIULF7o6oCBxKj74upw9f5mPEUkieONkPnkkiYiImxJ2+aJv567U9eFmz54tderUkTRpbr2cE3A5c+aUvn37mqEGBBwAIEXOXaltbXo5f/68mcPy6tWrEhgYKCVLlnQFHwAAKXaCZpU9e/Zop/YCACApUMwCAFjLKyU57VX57bffypo1a+To0aNy7do10x5XuXJl6dChg9SoUcMbuwEAwLchN3fuXHn++ec9hg84Vq5cKZMnT5Z27dqZ62zZsiV2dwAA+CbkFixYIL169ZKIiAiz1M7DDz9sOptkzZpVjhw5IitWrJDffvtNvv/+ezl9+rQsWrSIjigAgOQ/Tk7nrtTqyGPHjkn79u1lypQpkjlz5ijb6fI6jz32mOlxOW3aNOnatas3jjtVY5wckhLj5JAqxsnt2rXLBJwusaNL7UQXcEqHFjzxxBPmto6XAwDAVxIccjomTlWqVMlUT8bGmc7LeQ4AAMk65AoUKOCxjE5sLl265PEcAACSdcgVL15cypUrJzt27JDt27fHuq1O+6UeeOCBhO4OAADfDgafOHGiBAQEmF6V2nNSe1m60za7p556ynQ+6dy5szRv3jwxuwMA4M4MIdi5c6eMHz8+2hKdluYeeeQRs5Cq/ls7o+ig8L1790pYWJhZPVwnbdbnDxo0KH5HCADAnQ65U6dOmYHfsdGSm14i0xlQ5s+fbxZYJeQAAMku5EqXLi3jxo1L1M60pAcAQLILOQ2oPn363NmjAQDAi1iFAABgLUIOAGAtryy1oz0nN27cKLt37zarEei/o1O0aFHp0qWLN3YJAIBvltoZPny4nDx58rbbNm7cmJADAKSMkPvhhx9cS+1UrFhR8ufPbyZh1k4qTZo0kU2bNpnxdRkzZpRu3bpJ+fLlvXfkAADcyTa50aNHm4DTXperV6+Wfv36mftLlSplViZYu3atjBkzxiyzc+bMGXpnAgBSRsgdOnRI9u3bZ5bYGTlypPj5+UW73TPPPCNNmzY1pb5ffvklMccKAIBvQu748ePmukSJEpItWzZz2wm669eve2zrTMy8ePHihO4OAADfhZxOzOx+rZyFU3X6LnfOEjtHjhxJ6O4AAPBdyBUpUsRc60TMDp2cWQUFBcnly5dd9+u/zc7SMCwPAOA7CU6dXLlyScmSJc3QAZ282blPe1nqygOvvPKKXLlyxbTbffDBB+bxKlWqeO/IAQC4jUQVrTp06CA3b96Ur7/+2nXf4MGDzfWHH35oSnv33HOPKe1lz55devbsmZjdAQDgu5B79tlnZdmyZdK6dWvXfe3bt5d3331X8ubN65r5REtwutSO0zYHAIAv+IWEhHgu5+0lWsLTqkxdMFWrMeE9FR98Xc6e/7fNE/Cl4I2T+cCRJCIibkrY5Yu+n7syOtrJpGDBgnfq5QEAuC26OwIArBXnktzWrVvNNF6JoW1zo0aNStRrAADg9ZA7d+6cmXw5MWJaggcAgCQNuQoVKsj06dMTtTPtcQkAQLILOQ2oTp063dmjAQDAi+h4AgCw1h0bJ4c7KH1mXfKBjxhJYvfx+I1TArwlrZ9I6Vzx++2jJAcAsBYhBwCwFiEHALAWIQcAsBYhBwCwFiEHALCW11YhOHjwoGzZskVCQkIkX758cv/993vrpQEASJqQCwoKkgEDBsjq1atd9zVu3NgVcidOnJAGDRqYZXdWrVqV2N0BAOCb6koNuGbNmpmA04VRq1atGmUbXQ28WLFism3bNrOSAQAAKSLkhg4dKsHBwdKyZUtTVRnTUjxNmzY110uXLk3M7gAA8E3IOUvv+Pv7y6RJkyRLliwxblu4cGFzvWfPnoTuDgAA34Xcvn375ObNm1KqVKnbLqGTP39+c62lPgAAkn3IacApLcndTmhoqLnOlClTQncHAIDvQq5QoULm+sCBAxIeHh7rtuvXrzfXJUqUSOjuAADwXcgVKVJEypQpY6og58yZE+N2+/fvl1mzZpnbLVq0SOjuAADwbe/KIUOGmOuBAwfK9OnTzUBwx9mzZ2XmzJnSpEkTU13ZsGFDqVevXmJ2BwCAbxdN1WEDEyZM8LgvXbp0cuPGDde/tXPKwoULXR1QkEgsmookxKKpSFWLpo4aNUpmz54tderUkTRpbr2cE3A5c+aUvn37mqEGBBwAIMWV5NydP3/ezGF59epVCQwMlJIlS7qCD15ESQ5JiJIcUlJJzmsTNKvs2bNHO7UXAABJgWIWAMBahBwAwFoJrq787bff5KGHHorXcxo1ahTrmDoAAJJFyOm0XteuXYvXc8LCwhK6OwAAfBdyOrA7tlUFtKflX3/9JWPGjJGTJ0/KtGnTzGKqAACkyCEE0dFZUGrXrm3Gzv3++++SO3fuO7m71IEhBEhCDCFAqhoMfjs5cuSQPn36uKb5AgDAqt6VZcuWNdebN2/2xe4AAPBdyJ05c8ZcX7lyxRe7AwDANyF38eJF0+lE6TRfAAAk+96VOkflV199FePjupDq8ePHZcmSJaY9Tlcm6NGjR0J3BwCAb0Nu7Nixcdo2W7ZsMmnSJKlQoUJCdwcAgO9CrnDhwvLUU0/F+LiuPpAlSxYpV66cNG/eXLJmzZrQXQEAkDzHyeEOYJwckhDj5JAqxsmtWbNGihcvLo888khCXwIAgDsqwSGXNm1aCQ4OltDQUO8eEQAASR1y2ibnPgYOAACrQq5y5cqya9cuCQoK8u5RAQCQ1IPBJ0yYIAEBAWb82x9//OGN4wEAwPdDCCIiIswAbx0aoJedO3fK1KlTpVixYrJjxw5p1qyZmaNS/505c+ZoX0PHyQ0aNMh7Rw8AgDdCbuXKldK+fXuzJtyCBQvk1KlTMnfuXI9tdH252NaY05lPCDkAQLIfDF66dGkZN25cvJ5TqFChhO4OAADfhZwGlq4TBwBAql5qBwCApEDIAQCsFe/qyuvXr5sOJAmRPn16syIBAADJMuTWrl2b4MVPnZ6ZAAD4AtWVAABrxbskV6NGDRk/fnyCdsaacgCAZB1yuhBqtWrV7szRAADgRVRXAgCsRcgBAKxFyAEArEXIAQCsFeeOJ7Vq1ZINGzZIpkyZ7uwRAQDg65DTcCtTpoy39gsAwB1HdSUAwFqEHADAWoQcAMBahBwAwFqEHADAWoQcAMBahBwAwFqEHADAWoQcAMBahBwAwFqEHADAWoQcAMBahBwAwFqEHADAWnFeagdIKU6dvSB+fiJ5ArPFuM2585fk/MXL0T6WIX06KZQv5x08QqQWV66GyaXLVyVXzqzip1/KWIReuiohFy5JQMYMZnt4ByGHFO/GjXD5/pfN8t2STbJ28z7zw6ICc2SWLq1qyeBerSRzJn+P57w3Y5l88NWv0b5exdKFZPnnQ3xy7LDT+j/3yOezf5XdQcckIiLCBFfLxtXksYfuk8AcWVzbnT57QVas2y5r/9glW3cdkvDwm1K/Znl5/cVHkvT4bULIIcU7de6iPPPKF65/586ZRa5cuy7nQi6ZINu0/aDMn9pf0qaNWjtfKF8OSZ/O878BpTgkxpyFa2XKZz+a2+nTp5Oc2TObEtr8Jb9LqeIFpE2ze1zbLvx5o3w+59bJVoYM6UzIwbusCbkffvhBxo4dK//73//k3nvvTerDgQ+lT5dGOrWoIZ3urym1q5aQzAG3Sm2LV26VZ0Z/IRu3HZCfVm2TB+6tGuW5X45/UsqXLMjfC16xe99ReX/GT5ImTRp5+tGW0q5FLfH3Ty/Xwq7LkhWbo1RD5smVXTo/UE/q3VNWsmYOkCeGTOUv4WXWhFxwcLBs375dzp8/n9SHAh/TtrcprzwW5f5WjavIU93ukwmfLpG9B05GG3KAN82c+5vcvHlTHu/aRB5qW991v3+G9CbwInMv1e0//A9/jDuA3pWwmlZdqry5Yu6EcvL0eTl97qIPjwq2tg3//tdeSZ8urXRsXdfcdy4kNMYOTvANa0pyQGTavvHNjxskW5YAU6qLTse+kyT4wmXPjiq9W7mqPIG4Onz8jISF3ZAKZYrIsRNn5c0pc+XgkVPmsYL5A+U/XZpIi0Z384H6WIopya1Zs0YGDhworVq1ktatW8vgwYNlz5490W4bFBQkAwYMkObNm0unTp1kxowZUbb55JNPpEGDBrJjx44YH9u9e7dHm5/et2LFCnO/HkuLFi1k2LBh5vHnnnvO1Ra4aNEi6datmzRt2lR69+4tf/31V5R9XLhwQaZOnSrdu3c3x6nba5vi2bNnE/U54V+vTl4gW3cfkTcHdZbA7Jmj/WjOh16R/LmzS4b0aV0dVTr0nSRXrt3qoQnE1cXQWydLadOkkRde/VQOHT1t2uC088nxk+dkzHvfyoIlG/hAfSxFlOQ00D766COP+9auXSvTp0+XP//8U4oVK+a6f9u2bdKvXz8TIo7ly5fL4cOHZeTIka77/vnnH9OGd/ly1KoE57ErV65EafP79ddfzbE4zwsMDDTX+/fvN/ueMGGCjB492vW8TZs2mYD86aefpFq1aua+q1evmoB0D1G1ePFief/99+XgwYOJ+rwg8sYHC2XaNytkxFNtpGOLf9s9HGWK55fPx/aWxrXKSUb/9Kab97K1O2TI2NkmGD/9brU8070JHyXizBkHt233IaleqYQM799Zcgdmkxvh4fL1/FXy8Vc/y8dfLZNWTaqbsZjwjWRfktNSmIZKQECA9O/fX2bPnm1KSm+99ZaULl1awsI8z7g1ZOrVqyezZs2SJUuWyKhRoyRDhgwyefJkuXgx8e0u+jr6+l9++aUp1b355puux8LDw+Xtt9+WQYMGyffffy/z58+XNm3ayLVr1+Sdd95xbbd69WoTcBUrVjRB/fPPP5v3paXCLFn+HUOD+NOwGvb2t/LujGUy4um20u+x5tFu171NHWnZsLIJOOcHqkX9SjJxRHfz71/X7+LjR7xkzRLguj302Y4m4FS6tGmlR6d7pXK5onIx9IrsO3CCT9aHkv3pxLRp08wP0BdffCHNmjVz3V+/fn3p1auXCRZ3TZo0ka+//tr179q1a8uRI0dMFaSWxOrWvdUgnFA1a9aUOXPmxDh7wbhx40wVpEMDsVKlSqZE59DAVn379jXVqQ4t3WmQI2Gu3wiX/q99KfOW/Smjnn1Qnk5ASaxsifzmms4CiK9C+XOZsZj626BDAyLTdjkt5ekMKPCdZF2S0wDbuXOnFC9e3CPgHDoWJX36W2fijrZt20bZTkPGqXJMrPbt28c6PU+7du08/q2lSC1xuu9bg7Zly5am+lQvWurT6k73AET8XL4aJj1f/MgE3GsDOsQacNohRbt5R2fRiq3mmgHhiC+tgqxSvqjpZbl9z2GPx65fvyE7/v++vHly8OH6ULIuyWlVpFY/Zc8e9awoJtFtm+7/Z7Rw/2GLLagilw7dFShQIMbH0qZNG211o+7ffd8azt98842sXLlSli1bZqost2zZIjly5JAXXnhBHn300Rj3gaguXb4m3V/4QH7fsl+e7HqvNKtfUQ4cPe2xjQ60dYYTHDsVLF0HTJVH29eXiqUKSr7c2eXMuYvy429bZca8NWabDi1q8FEj3nTowObtB2T0O19Lr27NpUzxghJ8PlS+XrBKjp44KyWL5ZeihfK4ttdB4jq1lzp95tYY36vXwsy2SqvTnWpPWBhyWqrJkyePab86ffq0ue0tGihOJ5PINHB8oXHjxuaiNATfeOMN02nmrrvuct2P2/v70D8m4JR2NtFLZI+1ryfjhnR1tZEcOn7W9L6Mzn86NpB2TW51EgLio2GtCtK2+T3yw7I/ZOyUuR6PZcuaSYY/29njvj1Bx6T/Sx973Ldpa5D06DfB3K5euYS8M+q//BFsDTmn+k9LOlq6ee+996RMmTLmfu1Eoh1SHnroISlSpEi8X7dEiRLmWrvxN2zY0JQAtQSn/9bS1Z30448/mh6UXbt2lVy5ckV5XNvvCLm488+QTooVyh3rNu7TKRXMm0N+n/OyfLVovWzecUiOnQqRAP/0ZnqvzvfXlAY1Ssfr7wm4e+HJ9lKtUklZunKznNTvVsYMUrl8UXmoTX3JE2lSAp0JRdvqYhLbShqwJOS0x+HSpUtl/fr1UqtWLcmbN69kypTJDAnQ0o/2XkyI++67zww90KEI5cuXl4IFC8rJkyfN0ABtM1u3bp3cKUePHpXhw4ebS+7cuc3lxIkTZkoyrUZNbOeY1EbDaf2cl+L1nCIFAmVI79Z37JiQujWpX9lcbqdsyUIya/LzPjmm1CpZdzxRGgA6zq1Lly6SMWNGOXXqlCkFaduXtl8lpBSntMOK9rgsVaqUCbZ9+/aZ8NRSY6NGjeRO0sHs2rNSq1/PnDljqmM14KpWrWqGTBByAOAdfiEhIRGSQuh4Mx0O4O/vL4ULF/boPKK9F/WxokWLRul8Ettj6tChQ3Ljxg3zuHYS0XY6vWivSKe34+1e48CBAxIaGiqVK1eO9rFLly65enm607bGkJAQU0KNcweb9Jm150zctgW8bPdx5vlE0kjrJ1I6l5+9IYf/R8ghCRFySEkhl+yrKwEASChCDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGCtdEl9AEiAiAg+NiSZtH58+Eg53z2/kJAQfjEBAFaiuhIAYC1CDgBgLUIOAGAtQg4AYC1CDgBgLUIOAGAtQg4AbuPzzz+XoUOHys2bN/msUhhCDinWxIkT5Y033kjqw0Aq8OOPP8oHH3xAyKVAhBxSrG+++Ua++OKLpD4MAMkYIQcAsBZzVyJZWrNmjfz5559y9uxZKViwoNSuXVuqVq1qHrt06ZK89tpr8s8//0hYWJhpK3F06NDBbKv0ucuWLZOgoCBTzVSiRAlp3bq15MyZ02Nfr7/+uvj5+cnw4cOjPZYJEybIhQsX5OWXXzbb3e74kPIcOHBAfv75Zzl27JjkypVLatWq5foeRXby5EmZN2+euS5atKh07txZsmXL5rHNe++9JxcvXpQRI0ZEeb4+pt/hYcOGebT57dq1S8aMGWNe9/vvv5ejR4+a73PlypXNd0+vH3nkEdmwYYP88ssvcv36dXOMLVq0iLKP8PBwWbp0qXnN0NBQc5yNGzeWYsWKSWpDyCFZuXr1qnTq1MmESGQPP/ywaRfRbfTa4X67TJky5j/++PHj5a233pJr1655vIb+GH388ccePwwhISHy0UcfSfPmzaVmzZoe2+/Zs0dGjx4tPXr0MAEXl+NDynHjxg1zkvTJJ59EaW/r2LGjud/dqlWrpGfPnuakx/HOO+/IkiVLpFChQq77Zs+eLadOnYo25PSxc+fOeYSctvnpazRq1Eh69+4tly9fNveXLVvWXPR71a5dO/N91JB0pyGr32nHiRMnpG3btrJv3z6P7dKmTWuO5/nnn5fUhJBDsjJ37lwTIPny5TM/Mvnz5zf/afXsdcuWLWabzJkzmzPed999V65cueJRknPOvrWUpa9x7733mrNX/QHbu3evLFy4UJ566inZsWOHBAQEmG2ffPJJ8yOhQRc55JwfD90mrseHlOOVV14xf2MNgAcffFAqVKhgAmb9+vWybdu2KNv/97//lXLlypkwcr4P+/fvN9/HKVOmJPp4nn76aSlfvrzcd999EhgYKPfcc49HwGoYdunSxQTf8ePHZdasWfLtt99K9+7dpUmTJma7qVOnmoArXbq0tGnTRrJnzy5HjhyRFStWyPbt2yW1IeSQrGgVj5o+fbo0aNDA4zEnRDJmzCjPPPOM6XRy/vx5czuyF1980VQ7abXOoUOHzA+X/mhUrFhRNm7caEKwfv36ZttSpUpJs2bNZMGCBaa3pj5PaTWPdm7R7bSqKK7Hh5QhODhYPvzwQ/N9WrRokdSoUcPj8ehCTktNWkPg6Nu3r9x9990mQLyhVatW8v7777uqxZ3vodLSo35H3b93devWNSW/X3/91RVyzndUS4aBgYEeVZhafZna0PEEycr9999vqhQnTZokK1eudP0HV/Fp89q0aZP50erfv7+8/fbb5odDq3w04JS2pbnT0p1Wbbr31tRqJf1h0ce8fXxIevpd0DZdrWaOHHDKObFxp98nd9q+W6VKFdOO5g3PPfecR8C502OMfGKl7WxK26cdWp2eJk0a0279xx9/uKrstbRaqVIlSW0IOSQr2kC+evVqKV68uKmG1Otq1aqZkplWucSFVh8NHjzYnLm2b9/ePPd///ufqVLShnwVEWnhWT0L1vY897YZLa3dddddprOKN48PyYO2xar4dMYoUKBAlPsyZcpkvmtxFfm75y62Y9Gq8cicKndtW3Q0bNjQ1TGld+/epq1QayO0lkJrPlIbqiuR7GiwvPnmm+a2noVu3rxZXnrpJdMxZO3ata4qmJjOeH/77Tfzo6PVSk888YTHY/o60dHX6tOnjwwaNMj0StPSmrbb6dmwngEn5PiQvOXJk8dc79y50+uvre3G7qV894DTk6EsWbJE+7zI37WE0irUyZMnm9t6HOvWrTMdTvS7vXz5clPSSy1SzztFiqCN69qd2+Hv7y916tSRpk2bmioh916N+piejbufxap06W6du0W+X9vMPvvssxj33a1bNxNuWoLTi/5QPfroowk+PiRv2slIw+a7774zbXLu9LujgZBQeiKk7cA6hMWdnni598y8ExYvXiynT592/TtLlizmBKx69ermhMz9+5saUJJDsqI/NtOmTTNnotohRENHqx+1/UsVLlzYta32HtP/tNpdWtsa9CzYGSent0eOHGmqbYoUKWI6n2jnAK1uchrmI3NCTdvv9PmPPfaY5MiRI8HHh+RNf/y1WnvUqFFm/JmGgPZs1B672ltWqwKjG4MWF9qrcc6cOeZ1tR1Xezj+9ddfptdj7ty55U7StmctuWkbXvHixSVDhgymV6W2z2knG6djVWpByCFZ0fYEPfvV8NKLI2vWrDJkyBDT/uXQDiHapVr/Q+tFabuadvPWsW3aPdz9TFp/eHRIgVZJxkSrNzXktD1Dqy8Tc3xI/gYMGGCqtnVcpfa41YtTG+De4Si+dEybXnRQt16ckygdrqBtYzpO7k7RNuTdu3d7/L9QOuxFS5KRT9xs5xcSEhJzKyiQRPSsV8e16eBrLX1pV+no2jHOnDlj2uC0ekZ/rLS3mQ4TUFotoz3o9DW056NetAu1drfWs2udASUybWPTEqKOT9IxUIk9PqQMWu2tHYq0l6K21WkpyH1wt55MHTx40Ixji9wW7DwW3VAWLRFqKUq/G1qlraUo7bWr3xutKYjL6+sJl47hLFmypLRs2TLax/Q7q1WS7tWtWnILCgoy7YBam1GvXj1Jnz69pDaEHBBpZQMtAeo0Szo4GEDKRsgh1dM2Op2/Uqdh0sG2Wq2jnVRS41kvYBva5JDqaS84Z85JbYvR4QEEHGAHSnJI9TTkdGiBdgzQQbPaaxKAHQg5AIC1GAwOALAWIQcAsBYhBwCwFiEHALAWIQcAsBbj5JAq/P777/Lll1963KfTJ+laYLpGnM5pqZPzJhc6t6FOHKwT6rqvRK10PsJZs2aZeTJ1ns7E0CnRdCJhncbqP//5jySX9xgbb75/pXOR6vySOjVb9+7dxRe8/R4QM0IOqYLO/u6+6nd0dBqvKVOmJIs5KHUWFj1eHbsXOQD+/vtv85gugJnYH0idyFdf69KlSz4PudjeY2y8+f6Vzi3pfDd8FXLefg+IGSGHVEWXxtFVwpVOXHvixAlzJr9p0yYzpZfSeSsB2IGQQ6qiVWPus7+rF1980aya/Omnn5qg07NsndU9udJqtffee89UswKIHSGHVE/b5oYOHWpCTunyPE7I6Rpxuoimth1pNeaSJUvMMjvanvTQQw9JgwYNXJ+f3q8Lsx47dszMfVmuXDl54IEHYl2kUpd4mT9/vlkCSBe3rFKlinlObHR5IT1GXU5FlxaKzrZt28yxHD161LyuLp6pC4C6L+qq702PWek6av369fMo8eqk1ZH56j0mhr4XXWbmyJEjZukkXWamWbNmcW5z1c9O/8660nvOnDnN8ja69NLtJOSzwZ1HyAEikjdvXvPDpOtzXbhwwfWZfP311+bfuhDliBEjzCrgjsqVK5uQO3v2rFkHbOnSpVE+y+HDh8vYsWPNCtGRrV27Vnr27GnWwnOnJTR9TkLac/S1+vbtG+2x6GrnWmrVcHN/b0rXMtOL++fhHnK+fo8JsXPnTunWrZtZBT6yl156yaz6/s4778Q6+bae7DiTdTu0vbBHjx7y7rvvms8wsoR+NvANQg74/+DQgFO6aGZkukq0lpx69eplSgRaIqldu7aZ3Llt27bmB1ZLP507dzaLW2rpT3su6uXZZ5+VwMBAadWqlev1Dh8+LF27djWdL3RxTv1x1iV+9HW0192TTz4Z77+LvpaWHHQxV11NQVem1l6T2vaoC8hq6UQ7mjjGjRsny5cvN70rq1ev7tHxJCAgwHU7Ob3H2OhSSVp6a9KkiVSoUMGUWrVDjZbsdFFSPTHQ8NbAi472sNQTh7vvvlvat29vqrb1velztWeuvr9XX33V4zkJ/WzgO4QcUr3g4GDTJqf0LF9XIohMA1BXFC9btqzH/a+99pr5gdMqOP2RzJEjh+uxQYMGmbazl19+WUaOHGlWI3dWfdZSkv74a7Wodnxxf56GQUJ+ELXEoAGnr6XVg/pj7U5DTcPc8fDDD5uSnIZcsWLForRVOt5+++1k8x5jo8GmPSULFiwY5THdvwbutGnT5Lnnnou2B60GnPaw1WrrNGnSuE5utGSnJTztedu7d2+56667Ev3ZwHcYDI5URX90te1JL3qGre1qGgarV682j+v9+fPnj/I8LXVEDjgtIekSPUqrstx/4Bz9+/c3JYqgoCBXKSosLMz8ICr98Yv8vFq1apkf5PgIDw+XGTNmmNujR4+OEnBKq9q0nSg+ktN7vB0tpUUXcKp58+am5B0aGmp60kZHT3C0560TcI4+ffqYv71+xnPnzk30ZwPfoiSHVEWrkaIbL6cdDAYMGGAu0XHvYOLQUpO2x+jZ+fTp02M8S9dqTqXteVrVqR0w9Dj0xzSm0oxWgWmVXlzpD6iWyvQ1O3ToIN6SnN5jXO3YsUNWrVplqku1uvLmzZvm/n/++cdc6/3RqVq1qumkEpnzHvbs2SObN29O9GcD3yLkkGrHySmd8USrn/QHzt/fP8bnRffjp73vnDP6yLOpREd/cJX2zFRaYtS2vei4V4nFhdOxQ9t+9D16S3J6j7ej1Y1a6tK2x9hoaS6+x+P8/d070CT0s4FvEXKQ1D5OLi60I0dMdMaON95447avUbNmTXPtVIdp9/aYaHVffDiv6ZQavC05vMfb0epnDTj9G2vJS9votITuhOzMmTPN9G4aStGJ7Vidx6L7HsT3s4FvEXJAAjljzvQMvWnTpqYHYVwUKFDAXGtVl7YRZs2aNco27t3548IpaeiYNJ3FxdmHTe8xNvq+Fy1aZG5ru1m9evWibOM8HpPYjsd5zL29NqGfDXyLjidAAmlXceeHTttk4kp7GzoDhGOq5orv1GI62NsJuk8++STOz3NKOc7wieT8HmOjVYfa9qa9JqMLOO1Bq2P2bteW5wyOjzxMYN68eeZ2nTp1Ev3ZwLcIOSARBg4caK4nTpxoOrQ4nRzcac86ZzYVpR0UnMHB48ePNzNsuNNtdeaM+NJB4GrChAnmWCJXy+kgaR3z5c4ZE6gdV6I79uT2HmOi4+90n9retnDhQo/HdOycDubWEuXt6PACLX06NPyHDRtm2uK0NNqxY8dEfzbwLaorgUTQweFaQtAqMh1+oGPVdGC1DqbW3nz6A6s/clrKevzxxz3GUGkXe60Gu++++6Rhw4amC7yOudJA0JKQ+5i2uNBOFxocP/30kzkWHcOlx6Jhpz379HV1kLPO3uLQkol2ndd9aQlIx3tpBxz3ab2S03uMiba96fvSKkkNNB0uoLOq6BRbOhhc2/9KlSplVqOIiZbMtCSny980atTIvD9dEkenRlP6eUQeJpDQzwa+Q8gBieB0H9e5DfVsXn8QnR9FR5kyZaIs4aIh8sMPP8gTTzwh69evNwPNHTpXopYedI27+NDOHlo1qD+077//fpSpunTAd+Q5I7VKUbfXwc5amnPGcrlP65Wc3mNsdLC2Dlv45ZdfTAcTvTjtaBr4WsKLLeQ05HXKs8GDB3uUBrUKVAf9R9dhKaGfDXzHLyQkJPquRoBF9Gx6zZo15gxbB4DHlc7vqKUALQHdrmu+DhbeunWrCRat5tIqNC1NaLjERks1Gi5aotL5MLVEodVuWjrQ3nyRfyC19KMlDH3tmCZovnr1qpnEWTuh6DAJPQbtbRh5oLN7m5Vur9Vy+j5i+px89R5jc7v3ryVFbV/TANISlAaQDoTXv79+D7SXo/t4NZ0lRUt7WrLUFR40KDWUtSSmJUSdAScuawzG57OJy98Q3kHIAQCsRccTAIC1CDkAgLUIOQCAtQg5AIC1CDkAgLUIOQCAtQg5AIC1CDkAgLUIOQCAtQg5AIC1CDkAgLUIOQCAtQg5AIC1CDkAgNjq/wBEVsz/PD1deAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, ax = plt.subplots(figsize=(4, 4))\n", "ConfusionMatrixDisplay.from_predictions(\n", " y_test, y_pred, display_labels=['stays', 'churns'], cmap='Blues', colorbar=False, ax=ax)\n", "ax.grid(False)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "c018", "metadata": {}, "source": [ "Of the 86 churners in the test set, the classifier caught 61 (true positives) and missed 25 (false negatives). Of the 214 customers who stayed, it correctly left 195 alone (true negatives) and flagged 19 by mistake (false positives).\n", "\n", "Accuracy is just the diagonal divided by the total:" ] }, { "cell_type": "code", "execution_count": 10, "id": "c019", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.306802Z", "iopub.status.busy": "2026-10-23T03:52:15.306710Z", "iopub.status.idle": "2026-10-23T03:52:15.308350Z", "shell.execute_reply": "2026-10-23T03:52:15.308123Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy from the matrix: 0.853\n" ] } ], "source": [ "print('Accuracy from the matrix:', round((tp + tn) / (tp + tn + fp + fn), 3))" ] }, { "cell_type": "code", "execution_count": 11, "id": "c020", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.309745Z", "iopub.status.busy": "2026-10-23T03:52:15.309678Z", "iopub.status.idle": "2026-10-23T03:52:15.311618Z", "shell.execute_reply": "2026-10-23T03:52:15.311242Z" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### Exercise: Count the Four Outcomes\n", "# A fraud model was checked on ten transactions (1 = fraud, the positive class).\n", "# 1. Count the true positives, false positives, false negatives, and true negatives\n", "# by comparing actual and predicted.\n", "# 2. Check your counts with confusion_matrix(actual, predicted).ravel().\n", "# Print exactly:\n", "# TP: 2 FP: 1 FN: 1 TN: 6\n", "# sklearn: [6 1 1 2]\n", "actual = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 0])\n", "predicted = np.array([0, 1, 1, 0, 0, 0, 0, 1, 0, 0])\n", "### Your code starts here.\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": 12, "id": "c021", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.312588Z", "iopub.status.busy": "2026-10-23T03:52:15.312528Z", "iopub.status.idle": "2026-10-23T03:52:15.315183Z", "shell.execute_reply": "2026-10-23T03:52:15.314858Z" }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TP: 2 FP: 1 FN: 1 TN: 6\n", "sklearn: [6 1 1 2]\n" ] } ], "source": [ "### Solution\n", "actual = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 0])\n", "predicted = np.array([0, 1, 1, 0, 0, 0, 0, 1, 0, 0])\n", "\n", "tp = np.sum((actual == 1) & (predicted == 1))\n", "fp = np.sum((actual == 0) & (predicted == 1))\n", "fn = np.sum((actual == 1) & (predicted == 0))\n", "tn = np.sum((actual == 0) & (predicted == 0))\n", "\n", "print('TP:', tp, 'FP:', fp, 'FN:', fn, 'TN:', tn)\n", "print('sklearn:', confusion_matrix(actual, predicted).ravel())" ] }, { "cell_type": "markdown", "id": "c022", "metadata": {}, "source": [ "## Precision, Recall, and F1\n", "\n", "```{index} precision\n", "```\n", "\n", "```{index} recall\n", "```\n", "\n", "```{index} F1 score\n", "```\n", "\n", "```{index} sensitivity\n", "```\n", "\n", "Two summary rates answer the questions a manager actually asks about a flagging system.\n", "\n", "**Precision** answers: *of the customers we flagged, what fraction really were going to churn?*\n", "\n", "$$\\text{precision} = \\frac{TP}{TP + FP}$$\n", "\n", "**Recall** (also called *sensitivity*) answers: *of the customers who really churned, what fraction did we catch?*\n", "\n", "$$\\text{recall} = \\frac{TP}{TP + FN}$$\n", "\n", "Low precision means the retention team wastes time and discounts on customers who were never going to leave. Low recall means many customers walk out the door without anyone calling them. The two usually pull against each other: flagging more customers raises recall but tends to lower precision." ] }, { "cell_type": "code", "execution_count": 13, "id": "c023", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.316002Z", "iopub.status.busy": "2026-10-23T03:52:15.315951Z", "iopub.status.idle": "2026-10-23T03:52:15.320259Z", "shell.execute_reply": "2026-10-23T03:52:15.319919Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Precision: 0.762\n", "Recall: 0.709\n", "sklearn precision: 0.762\n", "sklearn recall: 0.709\n" ] } ], "source": [ "tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()\n", "\n", "precision = tp / (tp + fp)\n", "recall = tp / (tp + fn)\n", "\n", "print('Precision:', round(precision, 3))\n", "print('Recall: ', round(recall, 3))\n", "print('sklearn precision:', round(precision_score(y_test, y_pred), 3))\n", "print('sklearn recall: ', round(recall_score(y_test, y_pred), 3))" ] }, { "cell_type": "markdown", "id": "c024", "metadata": {}, "source": [ "Precision is about 0.76: roughly three out of every four customers the retention team calls really were about to leave. Recall is about 0.71: the classifier catches about 71% of the churners and misses the other 29%.\n", "\n", "The always-\"stays\" baseline flags no one, so its recall is 0: it catches none of the churners, which accuracy completely hid. (Its precision is undefined, since it makes no positive predictions; scikit-learn reports 0 and, with `zero_division=0`, does so without a warning.)" ] }, { "cell_type": "code", "execution_count": 14, "id": "c025", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.321059Z", "iopub.status.busy": "2026-10-23T03:52:15.321011Z", "iopub.status.idle": "2026-10-23T03:52:15.324770Z", "shell.execute_reply": "2026-10-23T03:52:15.324456Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Baseline recall: 0.0\n", "Baseline precision: 0.0\n" ] } ], "source": [ "print('Baseline recall: ', recall_score(y_test, y_base))\n", "print('Baseline precision:', precision_score(y_test, y_base, zero_division=0))" ] }, { "cell_type": "markdown", "id": "c026", "metadata": {}, "source": [ "When a single number is needed, the **F1 score** combines precision and recall. It is their *harmonic mean*, which is high only when both are high:\n", "\n", "$$F_1 = \\frac{2 \\times \\text{precision} \\times \\text{recall}}{\\text{precision} + \\text{recall}}$$" ] }, { "cell_type": "code", "execution_count": 15, "id": "c027", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.325743Z", "iopub.status.busy": "2026-10-23T03:52:15.325684Z", "iopub.status.idle": "2026-10-23T03:52:15.328579Z", "shell.execute_reply": "2026-10-23T03:52:15.328295Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "F1 by formula: 0.735\n", "sklearn F1: 0.735\n" ] } ], "source": [ "print('F1 by formula:', round(2 * precision * recall / (precision + recall), 3))\n", "print('sklearn F1: ', round(f1_score(y_test, y_pred), 3))" ] }, { "cell_type": "markdown", "id": "c028", "metadata": {}, "source": [ "F1 is a convenient summary, but like accuracy it treats the two kinds of errors as equally important. When the costs of the errors are known, it is better to use them directly, as in the next subsection." ] }, { "cell_type": "code", "execution_count": 16, "id": "c029", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.329530Z", "iopub.status.busy": "2026-10-23T03:52:15.329474Z", "iopub.status.idle": "2026-10-23T03:52:15.330968Z", "shell.execute_reply": "2026-10-23T03:52:15.330683Z" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### Exercise: Precision and Recall for a Fraud Model\n", "# A fraud model was checked on a month of transactions:\n", "# it flagged 50 transactions, of which 30 were really fraud,\n", "# and there were 40 fraudulent transactions in total.\n", "# 1. Work out TP, FP, and FN from these facts.\n", "# 2. Compute precision, recall, and F1, each rounded to 3 decimals.\n", "# Print exactly:\n", "# Precision: 0.6\n", "# Recall: 0.75\n", "# F1: 0.667\n", "flagged = 50\n", "flagged_and_fraud = 30\n", "total_fraud = 40\n", "### Your code starts here.\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": 17, "id": "c030", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.331863Z", "iopub.status.busy": "2026-10-23T03:52:15.331804Z", "iopub.status.idle": "2026-10-23T03:52:15.333728Z", "shell.execute_reply": "2026-10-23T03:52:15.333394Z" }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Precision: 0.6\n", "Recall: 0.75\n", "F1: 0.667\n" ] } ], "source": [ "### Solution\n", "flagged = 50\n", "flagged_and_fraud = 30\n", "total_fraud = 40\n", "\n", "tp = flagged_and_fraud\n", "fp = flagged - flagged_and_fraud\n", "fn = total_fraud - flagged_and_fraud\n", "\n", "precision = tp / (tp + fp)\n", "recall = tp / (tp + fn)\n", "f1 = 2 * precision * recall / (precision + recall)\n", "\n", "print('Precision:', round(precision, 3))\n", "print('Recall:', round(recall, 3))\n", "print('F1:', round(f1, 3))" ] }, { "cell_type": "markdown", "id": "c031", "metadata": {}, "source": [ "## The Business Cost of Errors\n", "\n", "```{index} cost of errors\n", "```\n", "\n", "```{index} expected cost; classification\n", "```\n", "\n", "```{index} cost matrix\n", "```\n", "\n", "To decide whether a classifier is worth using, put a dollar value on each kind of error. For the churn case, suppose the company's finance team estimates:\n", "\n", "- **False negative (a churner we miss): \\$400.** The customer leaves, and the company loses about \\$400 of profit over the following year.\n", "- **False positive (a loyal customer we flag): \\$50.** The retention team calls and gives a \\$50 discount to someone who would have stayed anyway.\n", "\n", "To keep the arithmetic simple, we assume that the offer always keeps a correctly flagged churner, and we count only the cost of the *errors*. (A fuller analysis would also count the cost of the offers given to true positives; the comparisons below would come out the same way, only with different totals.)\n", "\n", "The total cost of the errors on the test set is\n", "\n", "$$\\text{cost} = 400 \\times FN + 50 \\times FP .$$" ] }, { "cell_type": "code", "execution_count": 18, "id": "c032", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.334683Z", "iopub.status.busy": "2026-10-23T03:52:15.334625Z", "iopub.status.idle": "2026-10-23T03:52:15.337960Z", "shell.execute_reply": "2026-10-23T03:52:15.337671Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "kNN cost: $ 10950\n", "Baseline (flag no one): $ 34400\n", "Flag everyone: $ 10700\n" ] } ], "source": [ "COST_FN = 400 # dollars lost for each churner we miss\n", "COST_FP = 50 # dollars wasted for each loyal customer we flag\n", "\n", "def error_cost(actual, predicted):\n", " \"\"\"Total dollar cost of the false negatives and false positives.\"\"\"\n", " tn, fp, fn, tp = confusion_matrix(actual, predicted).ravel()\n", " return COST_FN * fn + COST_FP * fp\n", "\n", "print('kNN cost: $', error_cost(y_test, y_pred))\n", "print('Baseline (flag no one): $', error_cost(y_test, y_base))\n", "print('Flag everyone: $', error_cost(y_test, np.ones(len(y_test), dtype=int)))" ] }, { "cell_type": "markdown", "id": "c033", "metadata": {}, "source": [ "The kNN classifier's errors cost \\$10,950 on the 300 test customers, far less than the \\$34,400 lost by flagging no one, the baseline that looked 71% accurate. But look at the third line: flagging *every* customer costs only \\$10,700, slightly less than the classifier. Flagging everyone has terrible accuracy (29%), yet because a missed churner costs eight times as much as a false alarm, its 214 cheap false alarms cost less than the classifier's 25 expensive misses plus 19 false alarms. Accuracy and cost can point in opposite directions.\n", "\n", "The same reasoning applies to fraud. If a missed fraudulent transaction costs a bank an average of \\$500 and a false alarm costs \\$20 of an analyst's review time, a classifier that catches more fraud can be worth many false alarms. The costs, not the accuracy, decide." ] }, { "cell_type": "code", "execution_count": 19, "id": "c034", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.338800Z", "iopub.status.busy": "2026-10-23T03:52:15.338751Z", "iopub.status.idle": "2026-10-23T03:52:15.340592Z", "shell.execute_reply": "2026-10-23T03:52:15.340240Z" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### Exercise: The Cost of a Fraud Model\n", "# On 1,000 test transactions, a fraud model had TP = 15, FP = 40, FN = 5, TN = 940.\n", "# A missed fraud (FN) costs $500 and a false alarm (FP) costs $20.\n", "# 1. Compute the total error cost of the model.\n", "# 2. Compute the error cost of approving everything (every fraud is missed: FN = 20, FP = 0).\n", "# Print exactly:\n", "# Model cost: 3300\n", "# Approve-all cost: 10000\n", "tp, fp, fn, tn = 15, 40, 5, 940\n", "### Your code starts here.\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": 20, "id": "c035", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.341413Z", "iopub.status.busy": "2026-10-23T03:52:15.341352Z", "iopub.status.idle": "2026-10-23T03:52:15.343076Z", "shell.execute_reply": "2026-10-23T03:52:15.342761Z" }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model cost: 3300\n", "Approve-all cost: 10000\n" ] } ], "source": [ "### Solution\n", "tp, fp, fn, tn = 15, 40, 5, 940\n", "cost_fn = 500\n", "cost_fp = 20\n", "\n", "print('Model cost:', cost_fn * fn + cost_fp * fp)\n", "print('Approve-all cost:', cost_fn * (tp + fn) + cost_fp * 0)" ] }, { "cell_type": "markdown", "id": "c036", "metadata": {}, "source": [ "## Choosing Between Classifiers by Cost\n", "\n", "```{index} model selection; cost\n", "```\n", "\n", "```{index} comparing classifiers\n", "```\n", "\n", "Once each error has a price, we can compare classifiers on the measure the business cares about. The table below evaluates nearest neighbor classifiers with several values of $k$, plus the baseline, on accuracy, precision, recall, and total error cost.\n", "\n", "(As in Section 1404, a careful analyst would make this comparison on a validation set and use the test set only for the final report. We use the test set here to keep the example short.)" ] }, { "cell_type": "code", "execution_count": 21, "id": "c037", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.343976Z", "iopub.status.busy": "2026-10-23T03:52:15.343912Z", "iopub.status.idle": "2026-10-23T03:52:15.372979Z", "shell.execute_reply": "2026-10-23T03:52:15.372073Z" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
classifieraccuracyprecisionrecallerror cost ($)
0baseline (all stay)0.7130.0000.00034400
1flag everyone0.2870.2871.00010700
2kNN, k = 10.7670.5870.62814700
3kNN, k = 50.8300.7060.69811650
4kNN, k = 110.8530.7620.70910950
5kNN, k = 210.8230.7200.62813850
\n", "
" ], "text/plain": [ " classifier accuracy precision recall error cost ($)\n", "0 baseline (all stay) 0.713 0.000 0.000 34400\n", "1 flag everyone 0.287 0.287 1.000 10700\n", "2 kNN, k = 1 0.767 0.587 0.628 14700\n", "3 kNN, k = 5 0.830 0.706 0.698 11650\n", "4 kNN, k = 11 0.853 0.762 0.709 10950\n", "5 kNN, k = 21 0.823 0.720 0.628 13850" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def evaluate(name, predicted):\n", " return {'classifier': name,\n", " 'accuracy': round(accuracy_score(y_test, predicted), 3),\n", " 'precision': round(precision_score(y_test, predicted, zero_division=0), 3),\n", " 'recall': round(recall_score(y_test, predicted), 3),\n", " 'error cost ($)': error_cost(y_test, predicted)}\n", "\n", "rows = [evaluate('baseline (all stay)', y_base),\n", " evaluate('flag everyone', np.ones(len(y_test), dtype=int))]\n", "for k in [1, 5, 11, 21]:\n", " model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))\n", " model.fit(X_train, y_train)\n", " rows.append(evaluate(f'kNN, k = {k}', model.predict(X_test)))\n", "\n", "comparison = pd.DataFrame(rows)\n", "comparison" ] }, { "cell_type": "markdown", "id": "c038", "metadata": {}, "source": [ "Among the nearest neighbor classifiers, $k = 11$ has both the highest accuracy and the lowest cost, but that agreement is not guaranteed; the exercise below gives a case where the more accurate model is the more expensive one. And none of the nearest neighbor classifiers beats simply flagging everyone on cost.\n", "\n", "All of these classifiers predict \"churn\" only when most of the neighbors churned. Because a missed churner costs eight times as much as a false alarm, it may pay to flag a customer even when only a minority of similar customers churned. Changing that cutoff requires a classifier that reports a *probability* of churn, which is where the next section begins." ] }, { "cell_type": "code", "execution_count": 22, "id": "c039", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.374702Z", "iopub.status.busy": "2026-10-23T03:52:15.374605Z", "iopub.status.idle": "2026-10-23T03:52:15.376141Z", "shell.execute_reply": "2026-10-23T03:52:15.375892Z" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### Exercise: Which Model Is Cheaper?\n", "# Two churn models were checked on the same 300 test customers.\n", "# Model A: FN = 30, FP = 10\n", "# Model B: FN = 12, FP = 60\n", "# 1. Compute each model's accuracy (300 customers in total) and error cost,\n", "# with a missed churner costing $400 and a false alarm costing $50.\n", "# 2. Print the name of the cheaper model.\n", "# Print exactly:\n", "# Model A: accuracy 0.867, cost 12500\n", "# Model B: accuracy 0.76, cost 7800\n", "# Cheaper: Model B\n", "### Your code starts here.\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": 23, "id": "c040", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T03:52:15.377041Z", "iopub.status.busy": "2026-10-23T03:52:15.376978Z", "iopub.status.idle": "2026-10-23T03:52:15.378901Z", "shell.execute_reply": "2026-10-23T03:52:15.378610Z" }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model A: accuracy 0.867, cost 12500\n", "Model B: accuracy 0.76, cost 7800\n", "Cheaper: Model B\n" ] } ], "source": [ "### Solution\n", "models = {'Model A': (30, 10), 'Model B': (12, 60)}\n", "costs = {}\n", "\n", "for name, (fn, fp) in models.items():\n", " accuracy = (300 - fn - fp) / 300\n", " costs[name] = 400 * fn + 50 * fp\n", " print(f'{name}: accuracy {round(accuracy, 3)}, cost {costs[name]}')\n", "\n", "print('Cheaper:', min(costs, key=costs.get))" ] }, { "cell_type": "markdown", "id": "c041", "metadata": {}, "source": [ "## Summary\n", "\n", "- Always compare accuracy with a baseline that predicts the majority class; with imbalanced classes, a high accuracy can hide a useless classifier.\n", "- The confusion matrix counts true positives, false positives, false negatives, and true negatives; `confusion_matrix` and `ConfusionMatrixDisplay` produce it.\n", "- Precision is the share of flagged cases that are real; recall is the share of real cases that are flagged. F1 combines the two.\n", "- When the errors have different business costs, compute the total cost of each classifier's errors and choose the cheapest, which is not always the most accurate." ] } ], "metadata": { "anaconda-cloud": {}, "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } }, "nbformat": 4, "nbformat_minor": 5 }