{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c000",
   "metadata": {},
   "source": [
    "# Evaluating Classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c001",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:44.496179Z",
     "iopub.status.busy": "2026-11-17T23:43:44.496095Z",
     "iopub.status.idle": "2026-11-17T23:43:45.858519Z",
     "shell.execute_reply": "2026-11-17T23:43:45.858093Z"
    },
    "tags": [
     "hide-input",
     "thebe-init",
     "runnable"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.metrics import (accuracy_score, confusion_matrix, ConfusionMatrixDisplay,\n",
    "                             precision_score, recall_score, f1_score)\n",
    "\n",
    "# The shopping session data and the train-test split from Section 1501.\n",
    "shoppers = pd.read_csv(path_data + 'online_shoppers.csv')\n",
    "shoppers['purchase'] = (shoppers['outcome'] == 'purchase').astype(int)\n",
    "features = ['page_value', 'exit_rate', 'product_pages']\n",
    "X = shoppers[features]\n",
    "y = shoppers['purchase']\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=14, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c003",
   "metadata": {},
   "source": [
    "So far you have judged classifiers by one number, accuracy. Accuracy treats every mistake the same, but in business the two kinds of mistakes rarely cost the same. A bank that misses a fraudulent transaction loses the money; a bank that wrongly freezes a good customer's card loses some goodwill and an hour of an analyst's time. An online retailer that fails to nudge a visitor who was ready to buy may lose the sale; one that sends a coupon to a visitor who was never going to buy loses only the cost of the coupon.\n",
    "\n",
    "This section introduces the standard tools for looking past accuracy: a baseline to compare against, the confusion matrix, precision and recall, and the cost of each kind of error. It continues with the real online shopping sessions (UCI Online Shoppers Purchasing Intention, introduced in Section {doc}`../14-classification/1401-classification`) and the same training and test sets as Section {doc}`1501-knn-scikit-learn` (the setup cell above recreates them), using the 11-nearest neighbor classifier chosen there on a validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c002",
   "metadata": {},
   "source": [
    "```{contents}\n",
    ":local:\n",
    ":depth: 2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c004",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.860160Z",
     "iopub.status.busy": "2026-11-17T23:43:45.860075Z",
     "iopub.status.idle": "2026-11-17T23:43:45.876335Z",
     "shell.execute_reply": "2026-11-17T23:43:45.876017Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.888\n"
     ]
    }
   ],
   "source": [
    "knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11))\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "\n",
    "print('Test accuracy:', round(accuracy_score(y_test, y_pred), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c005",
   "metadata": {},
   "source": [
    "## Why Accuracy Is Not Enough\n",
    "\n",
    "```{index} baseline classifier\n",
    "```\n",
    "\n",
    "```{index} imbalanced classes\n",
    "```\n",
    "\n",
    "```{index} majority class\n",
    "```\n",
    "\n",
    "```{index} DummyClassifier\n",
    "```\n",
    "\n",
    "When one class is much more common than the other, the classes are *imbalanced*, and accuracy can look good even for a useless classifier. The simplest comparison is a **baseline classifier** that ignores the features and always predicts the *majority class*, the most common label in the training set. scikit-learn provides one as `DummyClassifier`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c006",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.877499Z",
     "iopub.status.busy": "2026-11-17T23:43:45.877420Z",
     "iopub.status.idle": "2026-11-17T23:43:45.880812Z",
     "shell.execute_reply": "2026-11-17T23:43:45.880473Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline predicts: 0 for every session\n",
      "Baseline accuracy: 0.845\n",
      "kNN accuracy:      0.888\n"
     ]
    }
   ],
   "source": [
    "baseline = DummyClassifier(strategy='most_frequent')\n",
    "baseline.fit(X_train, y_train)\n",
    "y_base = baseline.predict(X_test)\n",
    "\n",
    "print('Baseline predicts:', y_base[0], 'for every session')\n",
    "print('Baseline accuracy:', round(accuracy_score(y_test, y_base), 3))\n",
    "print('kNN accuracy:     ', round(accuracy_score(y_test, y_pred), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c007",
   "metadata": {},
   "source": [
    "The baseline predicts \"no purchase\" (0) for every session and is right about 85% of the time, because about 85% of the test sessions do not end in a purchase. The classifier's 89% accuracy is an improvement, but the honest gain is the difference between the two, about 4 percentage points, not the distance from 0%.\n",
    "\n",
    "With rarer events the problem gets worse. Suppose a card issuer reviews 1,000 transactions, of which 20 (2%) are fraudulent. A \"classifier\" that approves everything is 98% accurate and catches no fraud at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c008",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.882075Z",
     "iopub.status.busy": "2026-11-17T23:43:45.881998Z",
     "iopub.status.idle": "2026-11-17T23:43:45.884319Z",
     "shell.execute_reply": "2026-11-17T23:43:45.884003Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of approving everything: 0.98\n",
      "Fraud cases caught: 0\n"
     ]
    }
   ],
   "source": [
    "actual_fraud = np.array([1] * 20 + [0] * 980)     # 1 = fraud, 0 = legitimate\n",
    "approve_all = np.zeros(1000, dtype=int)           # predict \"legitimate\" for every transaction\n",
    "\n",
    "print('Accuracy of approving everything:', accuracy_score(actual_fraud, approve_all))\n",
    "print('Fraud cases caught:', np.sum((approve_all == 1) & (actual_fraud == 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c009",
   "metadata": {},
   "source": [
    "Accuracy alone cannot tell this useless rule apart from a good fraud detector. The two kinds of errors need to be counted separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c010",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.885477Z",
     "iopub.status.busy": "2026-11-17T23:43:45.885392Z",
     "iopub.status.idle": "2026-11-17T23:43:45.887144Z",
     "shell.execute_reply": "2026-11-17T23:43:45.886862Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Baseline for Loan Defaults\n",
    "# A lender's test set has 400 loans, of which 48 defaulted (label 1).\n",
    "# The counts are given below.\n",
    "#   1. Compute the accuracy of always predicting \"repaid\" (0).\n",
    "#   2. Compute the classifier's accuracy: it was right on 362 of the 400 loans.\n",
    "#   3. Print how many percentage points the classifier beats the baseline by,\n",
    "#      rounded to 1 decimal.\n",
    "# Print exactly:\n",
    "# Baseline accuracy: 0.88\n",
    "# Classifier accuracy: 0.905\n",
    "# Improvement (points): 2.5\n",
    "ex_n_loans = 400\n",
    "ex_n_default = 48\n",
    "ex_n_correct = 362\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c011",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.888047Z",
     "iopub.status.busy": "2026-11-17T23:43:45.887994Z",
     "iopub.status.idle": "2026-11-17T23:43:45.889716Z",
     "shell.execute_reply": "2026-11-17T23:43:45.889406Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline accuracy: 0.88\n",
      "Classifier accuracy: 0.905\n",
      "Improvement (points): 2.5\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "ex_n_loans = 400\n",
    "ex_n_default = 48\n",
    "ex_n_correct = 362\n",
    "\n",
    "ex_baseline_accuracy = (ex_n_loans - ex_n_default) / ex_n_loans\n",
    "ex_classifier_accuracy = ex_n_correct / ex_n_loans\n",
    "\n",
    "print('Baseline accuracy:', ex_baseline_accuracy)\n",
    "print('Classifier accuracy:', ex_classifier_accuracy)\n",
    "print('Improvement (points):', round(100 * (ex_classifier_accuracy - ex_baseline_accuracy), 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c012",
   "metadata": {},
   "source": [
    "## The Confusion Matrix\n",
    "\n",
    "```{index} confusion matrix\n",
    "```\n",
    "\n",
    "```{index} true positive\n",
    "```\n",
    "\n",
    "```{index} false positive\n",
    "```\n",
    "\n",
    "```{index} true negative\n",
    "```\n",
    "\n",
    "```{index} false negative\n",
    "```\n",
    "\n",
    "```{index} positive class\n",
    "```\n",
    "\n",
    "In a binary classification problem one class is called the **positive class**, usually the event you are trying to detect: a purchase, a default, fraud. \"Positive\" does not mean good; it means \"the thing you are looking for\". Every prediction then falls into one of four groups:\n",
    "\n",
    "| | Predicted positive (purchase) | Predicted negative (no purchase) |\n",
    "|---|---|---|\n",
    "| **Actually positive (bought)** | **True positive (TP)**: buyer correctly flagged | **False negative (FN)**: buyer missed |\n",
    "| **Actually negative (did not buy)** | **False positive (FP)**: non-buyer flagged by mistake | **True negative (TN)**: non-buyer correctly left alone |\n",
    "\n",
    "A table of these four counts is called a **confusion matrix**, because it shows which classes the classifier confuses with each other. The correct predictions are on the diagonal (TN and TP); the two kinds of errors are off the diagonal.\n",
    "\n",
    "You can count the groups yourself with pandas, using `crosstab` to cross-tabulate actual and predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c013",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.890842Z",
     "iopub.status.busy": "2026-11-17T23:43:45.890776Z",
     "iopub.status.idle": "2026-11-17T23:43:45.899623Z",
     "shell.execute_reply": "2026-11-17T23:43:45.899168Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>predicted</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2969</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>255</td>\n",
       "      <td>317</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "predicted     0    1\n",
       "actual              \n",
       "0          2969  158\n",
       "1           255  317"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(y_test, y_pred, rownames=['actual'], colnames=['predicted'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c014",
   "metadata": {},
   "source": [
    "scikit-learn's `confusion_matrix` returns the same counts as an array, with actual classes as rows and predicted classes as columns, in the order 0, 1. For two classes, `.ravel()` flattens it into the four counts in the order TN, FP, FN, TP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c015",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.900700Z",
     "iopub.status.busy": "2026-11-17T23:43:45.900629Z",
     "iopub.status.idle": "2026-11-17T23:43:45.904016Z",
     "shell.execute_reply": "2026-11-17T23:43:45.903623Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2969  158]\n",
      " [ 255  317]]\n",
      "TN: 2969  FP: 158  FN: 255  TP: 317\n"
     ]
    }
   ],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print(cm)\n",
    "\n",
    "tn, fp, fn, tp = cm.ravel()\n",
    "print('TN:', tn, ' FP:', fp, ' FN:', fn, ' TP:', tp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c016",
   "metadata": {},
   "source": [
    "`ConfusionMatrixDisplay` draws the matrix as a labeled picture, which is how it usually appears in reports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c017",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.904995Z",
     "iopub.status.busy": "2026-11-17T23:43:45.904932Z",
     "iopub.status.idle": "2026-11-17T23:43:45.941372Z",
     "shell.execute_reply": "2026-11-17T23:43:45.940873Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAGNCAYAAAA4tfmcAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQKhJREFUeJzt3Qd0FGXbxvGb3nvvvQiiIqKIFEWa+CogKKhgFxEFFFFRsWB9VRQUGyp2iuCLICCChSoCglIUpDell9A7+c715Jt1N9mEJBsSJvn/zsnZZHd2ZnYY9pqnTqaoqKhoAwAAvpE5rXcAAAAkDeENAIDPEN4AAPgM4Q0AgM8Q3gAA+AzhDQCAzxDeAAD4DOENAIDPEN4AAPgM4Q0AgM8Q3gAA+EzWtN4BwNOo6yDbs+8QBwRpYtX3/+XII01ER5+y44cPJuk9hDfOGgruXXsJb6SNTJmoiIR/cLYCAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzWdN6B4CMqkSR/Hbi5CnbFXUgUcsXLZTXsmTObNt377fo6OhEvSdv7hyWJ1cO27Zr3xlZP9KvPXsP2p59hyx/3pxWtFC+sMvs2L3f9h88Eva1XDmzWaliBeNd/9Fjx2333oPu/0HmzJQjk4rwBlJJjuxZrWOri+z6q+pb/TqVLHfO7O75rTv32qdf/2yvfzzVjh0/Eed93To1tV5dm1uZEoXc37ujDtpH/5tlr3w42Y6fOBl2W+2a17U+t7eyOtXLur+j9h2yz8fPsdc+nmJ79x+OeP1Inzb8s9PG/fi7TZ651H5dus5OnYq2rm0vtTf73xx2+affHGcjJ80L+1rT+jVs3Ds94zw/7off7OUPJtvK9Vvd+rNny2qN6lW1p+9va+fVKJfinym9IryBVFKtYgl766ku7vdTp0650M6XJ6eVLFrAHr27jV1Qs7x17vNeyHsG9GxrvW5p4X5XCefI0eNWrHA+63tna6tYpqjd/eQncbbzRPf/uNfl0JFjtjvqgHtPz67Nbe7itfbtjCURrR/p11vDf7IPx8x0v+fMkc2dD4lRvnQRy5I5U8hzpYoXiLPc+19Ot0cHfuV+z5UjmxUqkMe279pnP839y+b8vsamDOtDgCcS4e1zt99+u+3evdvGjx+f1ruC09AXoUq/oyf/6ko1R4/FlLI7XVXf3nzyZmvV+FxrWLeK+xKTcqUKW4+bmrnqxXue+szG//i7e75WldL26ct3WcfWF9nn38yxmb+uDGzjiktquuDVth55dbSNmjTflZ5Vyr+1/WUWte9gYNnkrB/pW8UyRazXLc2tdeM6duzYCWt335BEvW/qsIesRNH8p11OtUvyZI9rXG1P1qxZXPV8r+dH2MTpi+3t4T/Z0GdvjfhzZAQ0NPjcqlWrbPny5Wm9G0iE1Ru2uy+p2QtXBYJbvpz8qwt1qVGpVOD5JhdVd19un4//JRCssmzNZnvqza/d7zdf0yBkG33viClx9x881r3Pq/ZWCfzdkdMCFwbJXT/St/tuvtIG9Gxnl15QxbJkCS1JR0oXlOp7Ubp4Qdeko3NPVPoe0Kud+33D5l0pus30jPAGzgI7dsd0WgvuWFYgX+54v9DW/7PTPV58XuXAcwXz57aGF1Z1JZnPxsVcDKgzUIF8ucJuM6nrB+JzKvqUbd4elWDnS1XD63w8ePioq+0JtnPPfveophokDuENnAUd2a5vfZFt2bHXps39txZl64697vGyC6vGeU+jC6u5x/KlClumTDElpNpVy7jHeUvW2iXnV7aFY5+2v7570db/9KrNGfW4XdWkTsg6krp+ID6X3fii1b66v1Vt0c/Ouepxe2nopDgBLWrSUYfJmx9632bMX2HLVm+2r7//ze595nPXtHPfTc04yOmlzXvChAn28ssv2/PPP2/lypWzN99805YtW2b58+e3tm3b2i233BLnPeoMNGbMGJs4caJt3rzZ8uTJYxdffLHdfffdVqJEiSRvt1ixYvb222+7KupChQpZ+/btrXPnziFfah999JH7GTp0qNWuXTtkXd5rH374odWsWTPO+kuWLGnvvfee+1z16tWzl156yS1z8OBBGz58uE2fPt22bdvm9r1x48aunTtnzpxx9lmfWT9qA69WrZr17t07sD2PhgD99NNP9s0339jq1avt+PHjVrFiRbvuuuusdeuYKtdge/bssU8++cTmzZtnO3fudPt60UUXuX0oUCC0Q8qBAwfs888/d/u7Y8cOK1iwoF1xxRV2xx13uH8DhDfosRtdiaPTg+/a4aAOQtPm/2UHDh117Y9vPHGTjZw411V/N6lfw3VwU8909dTVcDB1NitUIHeghDPmjR6WLWsWd0FQpGAeO6dKafvi1bvtjsc/DlSRJ3X9QHz2HzhipYoVsJ17DtjWnfvcSIVfFq2xr9+637Jk+beMeNf1TdyQxH4DvwppT69Xu4J9/NIdVqdGzOgIpIPwVnj88ccfNnv2bHv//fdt375/qxV//PFH27hxo/Xv3z/w3IkTJ+ymm26yqVNjOkZ4Zs6caR9//LGNHTvWzj///ERvV9tQ6B4+/O/wGq1b+6NA9yhctfyhQ4firMt7LXgd3vqnTZtmH3zwQeB9hQsXdo/r1q1zgarHYN9++60Lxy+//DLkeR2Dt956K/D3ggUL3MWLlq1SpUrg+Ycffth9nmDz58+30aNHW8+ePe25554LPB8VFeUuFv7++++Q5bVehfTChQsDz61fv95d1MTeX10ojBgxwiZPnuzCHP/Sxd/AR26wTm3q24MvjbIffwntu6Dqb3U6G9L/ZrulXUP3Ezzc5vya5axS2WKBdm0Nu5HLL65pE6cttt4vjHDjaFWyf/yeq12v8ud6tw+Ed1LXD8R2Qa3ydv1VF9llF1ZzF3onT56yCdMWufN51oKV9uW38+2moH4TS1Zssv++/61rHsqTK7trutFY8d+WbbQX35tk7zzdxQoXzMuBTg/h7Rk0aJC1aNHClbSLFClic+bMcSVUBZZKmPnyxUwioEBVuKqErKCqW7euKwWqZKv33HXXXa4UmdhJAbQ+lR7vvPNOFz5672uvveZKxG3atLGrr746os+l/df6b7vtNitbtqzbhkrH+pwKwnPOOcfuvfdeV5JW4OsiJHZA7tq1yz777DN78sknrUGDBnbkyBG3Xl0YDBkyxAYPHhxYVq9de+21dtVVV1nlyjHtmWvWrLF33nnHfVYdnwoVKrjnp0yZ4oK7fv36dv/997v92759u7swUBgH0/HRfumYqFaidOnS7rh/+umn9t1339lTTz3lak0QQ6WRd5/pah1a1nNfdF4bdWwjJ86zVeu32d03NHHV4keOHbdvpy9xw3nW/PCy7TtwODCcJ2p/zAWgwrbHgM8DpWV1jnt6yHhr2ehcq1m5lCvle23aSVk/EFu3G5rGOa/bNb/QnYPdnvzUfvhlWSC8dT7e0PtdO3DoiA0f2M014+gC9vCRY66X+QvvTbT7nv3CRr7enQOdnsK7WbNmNmrUqMDfl1xyiW3atMlVR6sEe+mll7rnFao6IVSSVOh4FLJXXnmlLVq0yH755Re77LLLErVdVRF/9dVXgSryhg0butC79dZbXYky0vDWPqqqO7gKXhcZS5cutVq1armSa3AVucJRARzs5MmTrvTeqlWrwHNqJqhRo4YL2mCvvvqqzZo1y8aNG+eCVVXzulhQKVvNDb/99lsgvHPliuno1Ldv35B1K/j1XHDJXaXwDh062LBhw0K2p6p4/Xz99dfuAixLlpgephmZqrU//e+dduWltVzv8+ET5ia4/II/1rufYNc2u8D11g1+fuW6re5Rpe1w1dwb/tnlwlszZiVn/UBi6TyTvf9/QSlzfl/tStx3dGhsbZqeF3g+V87sri38qykLbMrsP12HNs0KiHQS3tdcc02c584991z3qBKpV2W+cuVKq169ekhwi0Lj+uuvd+GtoVWJDW9VXcfusKOSa+7cuVNkiFa7du3irH/JkphJNLp27Rq2bTv2c9oX1UoEU02EQlihHEwlYAV9fPbvj+n1KQpsXRSpZuPGG290x1Tt+VqvF+yiY+qF+OWXXx6YWlOP+lG/A61XzQcqkWdk+fPmsi8HdbeLzq1o9w343A0TS4hKMqqKDKbqxn73xFw0asy4R+2Nf676x2pXK2PVKpSwVRu2BV5Tu7WqOHWB9s+2qGStHwh24sTJwHCv2NRsI2VLxDQDikrYsmN33Kl61blNswDq++Lo0eOEd3oK79idoyRr1pjd1xeSHDsWc3LkzRu+zUSd3OTo0aOJ3m64dSlsFZje9rzn4qOScXxKlfp3XK/HW2+4zxzfPoZrBtAFi3dsRBcbCm51wOvTp48r2Wsbeu/3338f0t4tOXLksEmTJtkPP/zgagBUtb548WK3z48//ri78BCvJkA1IfqJT+wag4xG41nHvX2/m0FKVYTzl66zSmVDh8ZoLml9iXme6dnWsmfN6qar3Lozyo0Df+iOVnZO5VL2+7KNrrQSbOiX091Ull8O7m4vvDvRlq/ZYmVKFHTv0TCd6fP/ChnOk9T1I31TqXfbzphw3bw9ZjSCanHWbtoRuAgsXiTme3Tpyr+tx4Av7Lb2l9k5VUpZscL53Tk0dupvgdqkDq3qBdatKYF1sThh2mLr899RbqrgQvnz2MYtu+ytL350pfLqFUvQ5p3ewjsxFKhq61bpWyXO2B2k1F4tartNLFU7d+kSM6WlR2276nmt9mWPty2VLmNT4CVF+fLl3aPat9X5LqV4JXp1blO1f7Avvvgi7HsU7C1btnQ/ot7pCm71IK9UqZLr/KdRAKJ28RtuuCHe7SfluKdHF9YqH5j6UVOY6ie2gcO+c8HuyZwpk5t7XD/BVqzbajf3HRqn1PzFN3OtWYNzXLvjhy/cHvLaP9v22AMvjgx5LqnrR/o2fd5f1uXh0Jq5cT/87n6kfYsL7aMX73C/Z8uW1Z0n/V6Lme40tofvbO1GLng0d36/bm3cReXH/5vtfoJputSBj3Y6A58qfUpX4S0KGfXE7t69uysper231RauNmoFfJMmTRK9PoVao0aNrGPHju5vVQFr3RJcVe11/tI21UNbJVqVuPW3SrVJoapn1RKojV9NAD169AhUlastXD3I1TM8qVTilhkzZlinTp3cOlUy12dU34HY1G9Aw7/UdOBdnKiGQVVbep+qyxXe6o/g7a+OVfPmmvYwayDs1WNfFzDnnfdvO1dGdOjwsUAJJj5qr45944eV67fZ1U3Ps7IlC7lhOFNm/2GfjJ0dMkubR/82tz/2kX0/Z5nrDKfZrPYdOGIzf13hZliLdP1I39TWHLs2KJhqbzznVitj88f0dxeMi//a6IYlqmSu4V43/aeBK2nHphkAG5xf2ZXMVSuk/xOaXKh+nYpuGJlGNyCDhvdjjz3mekmrh7OCT6VYlZK9IWZ6XaXzxLrwwgtdD2x10FKAaWiagkvr1fMe9RjXeGl1NlMPcbXtbt261Q0BU7uxOskllrbzwgsvuIB+9tln7b///a8rtaptXz9NmzZNVnirzVrjtDVcThcUqv7esmWLa48Ot4/qha6x6Kpi1xhz7ZfXfq1wVqc40YXK66+/bt26dXM9zVWNr+VV/a/ldRGjPgaPPvqoZWQa91rvugFJeo9uGao7juknKUZMmOt+ztT6kT5dfklN++3rZxK9fNUKJVzTS1I0qlfd/SAy6W6GNQWoglula4Xs2rVrXXCXKVPG3njjjSSHnoZpKbzUTq6xzFqnSpeaZCW4TTpbtmyu9Fq1alUX2JoARaV89b5OSknfo85qqilQBzGFoD6HgvuCCy5wHciSQ53YVMrWRY0CWM0LKkmrE9s999wTZ3mVznWBoosdNQesWLHCvU+hrR7yukjxqGZCN0fRRYA+v4Jf7d8KefXI79evX7L2GQAQV6aoqKiYrsFnKQWWQkA9nGN34EroNVG7t0q/mt3La5dNLI2b7tWrlxtOpZncFEga86wg86qf47NhwwbX8137pfBS8OlHY7W9Xtqn2/dgqjnQWG6VlL1Odx7N+qZtBQdp8Gsq9caeZU28WgGV6LNnz2579+51+63jFK5mQvuv4FbJPb4OgR5dLGn9unhRDURix9TXbvuC7dobd5IbIDXs+fXfSY6A1BQdfcqOHfp3pE+6qDZXkMRXzZ3Qa6Jq3pSa1UtBpBJrYnjjpD2qQo49Levp9j1Y0aJF3U84uiCIT0KvKYSD6QIioTbpcJ8hPrrAiH2RAQBIOemu2hwAgPSO8AYAwGfO+mrztJzRTZ3DYleBAwCQ1gjveCSlTRoAgNREtTkAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMA4DOENwAAPkN4AwDgM4Q3AAA+Q3gDAOAzhDcAAD5DeAMAkF5vCbpkyRIbMGBARBs777zz7Omnn45oHQAAZHSJDu/du3fbjz/+GNHGTpw4EdH7AQBAEsK7Vq1aNmzYsIiOWfHixTnmAACkVngreDt06BDp9gAAQITosAYAQHoteZ/O+vXrbfHixRYVFWUlSpSw1q1bp9SqAQBASob3mjVrrHfv3jZ79uzAc02bNg2E95YtW6xRo0ZWunRpmzVrVqSbAwAgw8scaXA3b97cBXeRIkXs/PPPj7NMqVKlrGLFirZ06VI33AwAAKRhePfr18/27NljrVq1clXm8Y0Dv/LKK93j1KlTI9kcAACIJLy9cd85cuSwIUOGWN68eeNdtmzZsu5xxYoVHHQAANIqvFevXm2nTp2yqlWrnnb8dsmSJd2jSukAACCNwlvBLSp5n86BAwfcY+7cuZO7OQAAEGl4lylTxj2uW7fOTp48meCyc+fOdY+VK1dO7uYAAECk4V2uXDmrXr26qwofM2ZMvMutXbvWRowY4X5v2bJlcjcHAABSorf5I4884h4feOABN++5Jmjx7Nq1y4YPH27NmjVz1eaNGze2hg0bRrI5AABgZpmioqKiIzkSGh42aNCgkOeyZs0acgcxdWqbOHFioOMaEE7tti/Yrr2HODhIE3t+fYsjjzQRHX3Kjh3an7pzm+v+3KNHj7YGDRpY5swxq/OCu1ChQnbfffe5IWUENwAAZ9Hc5mrL1s/evXvdHOdHjhyxwoULW5UqVQKBDgAAzrIbk0iBAgXCTpEKAABSDsViAAAyYslbvcy/+uor+/nnn+3vv/+2o0ePuvbuOnXqWPv27a1evXopsRkAAJAS4T127Fjr06dPyDAxz4wZM+ytt96ya6+91j3mz5+fgw4AQFqG9/jx4+3OO++06Ohod0vQzp07u05q+fLls02bNtn06dNt5syZ9s0339iOHTts0qRJdGADACCtxnlrbnNVi//zzz/Wrl07e/vtty1PnjxxltNtQG+55RbXA33o0KHWqVOnSPcZ6RTjvJGWGOeNDDHOe/ny5S64dStQ3RI0XHCLhpDdfffd7neN9wYAAJFJdnhrTLece+65rpo8Id60qN57AABAGoR3qVKlQm73mZCDBw+GvAcAAKRBeFeqVMlq1qxpf/75p/3xxx8JLqvpU+Xqq69O7uYAAEBKTNIyePBgy5Url+tlrp7k6nUeTG3i3bt3d53WOnbsaC1atIhkcwAAIClDxZYtW2YDBw4MWwJX6fvmm2+2MmXKuL/ViU2TtaxcudKOHTtmOXLkcDcr0fv79u3LgQcAIDXCe/v27W5CloSopK2f2DTj2rhx42zPnj2ENwAAqRXe1apVs1deeSWijalkDgAAUim8FbzdunWLcHMAACBS3FUMAACfIbwBAMiItwRVT/Jff/3V/vrrL3d3Mf0dToUKFeyGG25IiU0CAJBhpcgtQR9//HHbunXraZdt2rQp4Q0AQFqG94QJEwK3BK1du7aVLFnS3XxEnduaNWtmCxcudOPDc+bMaTfeeKOdc845ke4vAAAZXkRt3gMGDHDBrV7os2fPtp49e7rnq1at6u40NmfOHHvxxRfd7UB37txJb3UAANIyvDds2GCrV692twLt37+/ZcqUKexyPXr0sCuvvNKV0n/66adI9hUAAEQS3ps3b3aPlStXtvz587vfvQA/fvx4yLLeDUkmT57MQQcAIK3CWzckCX4UlcJF06AG824FumnTpuRuDgAARBre5cqVc4+6AYlHNyWRNWvW2KFDhwLP62+3scwMKwcAIFLJTtMiRYpYlSpV3BAx3bTEe069znUnsWeeecYOHz7s2sXfe+899/p5550X8Q4DAJDRRVQUbt++vZ06dcpGjRoVeO7hhx92j++//74rnV900UWudF6gQAG79dZbI99jAAAyuIjC+/7777fvv//e2rRpE3iuXbt29sYbb1jx4sUDM62pxK1bgnpt3wAAIPkyRUVFRdsZoBK5qtRz5MjhqtOB06nd9gXbtfffvhJAatrz61sccKSJ6OhTduzQ/tSf2zwcdU4rXbr0mVo9AAAZFt2/AQDwmUSXvJcsWeKmQ42E2r6ffvrpiNYBAEBGl+jw3r17t7vpSCTiu1UoAAA4A+Fdq1YtGzZsmEVCPdABAEAqhbeCt0OHDhFuDgAARIoOawAA+MwZG+cNJFm2PLo1HQcOaeLESb4KkUaioy3rqaTNcUHJGwAAnyG8AQDwGcIbAACfIbwBAPAZwhsAAJ8hvAEA8JkUu6vY+vXrbfHixRYVFWUlSpSw1q1bp9SqAQBASob3mjVrrHfv3jZ79uzAc02bNg2E95YtW6xRo0bu9qCzZs2KdHMAAGR4mSMN7ubNm7vgLlKkiJ1//vlxlilVqpRVrFjRli5d6u5MBgAA0jC8+/XrZ3v27LFWrVq5KvP4bhl65ZVXusepU6dGsjkAABBJeHu3CM2RI4cNGTLE8ubNG++yZcuWdY8rVqzgoAMAkFbhvXr1ajt16pRVrVr1tLf6LFmypHtUKR0AAKRReCu4RSXv0zlw4IB7zJ07d3I3BwAAIg3vMmXKuMd169bZyZMnE1x27ty57rFy5crJ3RwAAIg0vMuVK2fVq1d3VeFjxoyJd7m1a9faiBEj3O8tW7ZM7uYAAEBK9DZ/5JFH3OMDDzxgw4YNcxO0eHbt2mXDhw+3Zs2auWrzxo0bW8OGDSPZHAAAMLNMUVFREd2BXsPDBg0aFPJc1qxZ7cSJE4G/1alt4sSJgY5rQFjZ8phlysTBQZo4cTKir0Ig+aKjLeupQ6k7t/nTTz9to0ePtgYNGljmzDGr84K7UKFCdt9997khZQQ3AABnSck72N69e90c50eOHLHChQtblSpVAoEOnBYlb6QhSt7wU8k7RcMbiAjhjTREeCNDVZsDAIDURXgDAJBRbgk6c+ZMu/7665P0niZNmiQ4JhwAAJzB8Nb0qEePHk3Se44dO5bczQEAgEg7rCmIE7rRiHqeL1q0yF588UXbunWrDR061Jo2bWoFCxZMzuaQEdBhDWmIDmtIM2djb3PNunbJJZe4sd/z5s2zokWLnsnNwc8Ib6Qhwhtp5mzsba6Sdrdu3QLTpQIAAB/0Nq9Ro4Z7/P3331NjcwAApGupEt47d+50j4cPH06NzQEAkK6d8fDev3+/66wmmi4VAACk0VAxzWE+cuTIeF8/efKkbd682aZMmeLau3WnsS5duiR3cwAAICXC++WXX07Usvnz57chQ4ZYrVq1krs5AAAQaXiXLVvWunfvHu/ruptY3rx5rWbNmtaiRQvLly9fcjcFAACCcFcxnD0Y5400xDhvZIhx3j///LNVqlTJbr755uSuAgAAJEOywztLlixuetQDBw4kdxUAACA1w1tt3sFjuAEAgA/Cu06dOrZ8+XJbs2ZNyu4VAAA4M5O0DBo0yHLlyuXGby9YsCCSVQEAgJTubR4dHe0mXtEQMP0sW7bMBg4caCtWrLA///wzMId5xYoVLU+ePGHXoXHeffv2Tey+IaOhtznSEL3N4afe5oke5z1jxgxr166duyf3+PHjbfv27TZ27NiQZRTk+omPZlojvAEASKNJWqpVq2avvPJKkt5TpkyZ5G4OAABEGt4KYt2nGwAApMNbggIAgJRDeAMAkN6rzY8fP+46niVHtmzZ3B3GAABAKob3nDlzrEqVKsnamNdTHQAAJB/V5gAApPeSd7169dzkLMnBPb0BAEiD8M6bN6/VrVs3BTYNAACSg2pzAAB8hvAGAMBnCG8AAHyG8AYAIL3eEvTQoUP2999/W+7cua1s2bJnfs+Q8XBLUKQhbgkKP90SNNHhDZxxhDfSEOENP4U31eYAAPgM4Q0AgM8Q3gAA+AzhDQCAzxDeAAD4DOENAIDPEN4AAPgM4Q0AgM8Q3gAA+AzhDQCAzxDeAAD4DOENAIDPEN4AAPgM4Q0AgM9kTesdAPCv6Oho275rn2XJksWKFsob76HZsXu/HTh0JOxruXJkt5LFCkS0PLBn70E7evyEFS+czzJnPn05b8++gxa175Dlz5PLioQ5d0+ePGUbt+xKcB05smez0sULcvATgfAG0tiRo8dt7PcLbeyUBbbwj/V2+Ohx93zxIvmtS9tLrfctLS1H9tD/qs+9842N/nZ+2PU1vqi6jXnzvoiWR8Y0f8laGzHhF/v+5z9t7/6Y+0vnyZXDrmlW1x675z9xQnnj5l02cdoi+/7nP2zhn+vt1Klo63x1AxvYr3OcdW/btc8a3/hCgtuvd25FG//uAyn8qdInwhtIY6s3bLM+L450v2fKlMmF9v6DR1wJ/PWPptiSvzbZFwPvCfvecqUKW5ZYpaKEStFJXR4ZS/9BX9my1Zvd78UK57Ojx47bvgNHbNSkufbr0rX23bC+litn9sDyQ7+cZp+OnR0oNWv5+GTNktkqlCka9jVdKKjU3vzS2in+mdIrwtvnbr/9dtu9e7eNHz8+rXcFyZQzRza78T+XWIdW9e2icyu6v1V9/tV3C6zPSyPthznL7JffV9uldavGee+k9x90YZ9YSV0eGUujetXtwdtbu8d8eXK655as2GTdn/zE1mzcbv+busC6XNswsHyFUkXs3puaWYvLzrVjx05Y5wffiXfdOu9+HtU/7Gs393nXZi1caR1a1z8Dnyp9Irx9btWqVbZ9+/a03g1EoGqFEjbo8ZtCnlMJ/Pqr6tuCP9bZp1//bCvXbw0b3kBKeur+dnGeO69GOet1awvr+99Rtm7TjpDXunW+IvD7nN9XJWubf2/d7YK7Sf0atHcnAb3NgbNY0UL53GPxwuFLy2pj3LIjynZFHUjU+pK6PCCq0pYyJQql+AEZOXFuoK0ciUfJGzibO7JNXWAliua3ppfUDLvMFV3/a3v+/4tVy918bfgObsldHhnT5m17XE/zfQcO27xFa+z1j79zpeKOKVytrR7ooyfPt4L5c7uqdyQe/2PjMWHCBHv55Zft+eeft2LFitnbb7/tqqgLFSpk7du3t86dO7uqTc9HH33kfoYOHWq1a4d2uvBe+/DDD61mzZpx1l+yZEl77733bNmyZVavXj176aWX3DIHDx604cOH2/Tp023btm1WokQJa9y4sWvnzpkzpj0q2JgxY9yP2sCrVatmvXv3DmzPo7bUn376yb755htbvXq1HT9+3CpWrGjXXXedtW7dOs469+zZY5988onNmzfPdu7c6fb1oosucvtQoEBoR6cDBw7Y559/7vZ3x44dVrBgQbviiivsjjvusDx58iT+rITz8Ctf2obNu1xntdxBnYSCqWNbyaIFXEl6286YDm7zFq210W/0sCxZMke8PDKmrg8PtRXrtgb+/s8VF9gzvdpb/ry5UnQ70+ctty3bo+z2Do25gEwiwjseCq0//vjDfvzxRxe6hw8fDrw2depUmz17tgt0j8JVyx86FFOqCea9FrwOb/3Tpk2zDz74IPC+woULu8d169a5QNVjsG+//daF45dffhnyfP/+/e2tt94K/L1gwQKbOHGiW7ZKlSqB5x9++GH3eYLNnz/fRo8ebT179rTnnnsu8HxUVJS7WPj7779Dltd6FdILFy4MPLd+/Xp3URN7f3WhMGLECJs8ebILc5zeqVOnrN9rX7kOa68+coM1a3BOnGXOr1nOOrSs59rBs2fL6kowk2YstkdeHm0//7bKxnz3q3W++pJkL4+MrXSJQnbk2AlXXa6e4N/NXGKF8ue2Ab2vc+dPStGwNOHcSzrC+zQU0Co93nnnnS58VAJ97bXXXIm4TZs2dvXVV1skFLha/2233WZly5Z121Dp+JZbbnFBeM4559i9997rStIK/JkzZ8YJyF27dtlnn31mTz75pDVo0MCOHDni1qsLgyFDhtjgwYMDy+q1a6+91q666iqrXLmye27NmjX2zjvvuM961113WYUKFdzzU6ZMccFdv359u//++93+qXOcLgwUxsF0fLRfOiaqlShdurQrfX/66af23Xff2VNPPWVvvvlmRMcqIzhx4qT1en64ff39by64u7T9t2dvsDs7Ngn5W6Xma5vVde/v8cznNm3u8pAvxKQuj4zt81fvCRnL/ezb4+3z8XOsaOF89tAdV6XINjQU8sdfllntamWsdrWyKbLOjITwPg1VEX/11VeBKvKGDRu60Lv11ltdiTLS8FYwqqo7uAp+zpw5tnTpUqtVq5YruQZXkSscFcDBTp486UrvrVq1Cjx38cUXW40aNVzQBnv11Vdt1qxZNm7cOBesqprXxYJK2Srx/fbbb4HwzpUrpoqsb9++IetW8Ou54JK7SuEdOnSwYcOGhWxPVfH6+frrr23QoEFu5jCEd/joMbvr8Y9t2rzl9tpjne2m/yS9A0+NSqXc4979h8/I8sh4ypcuYq/16+xK31NmLU2x8B797Tw7cfKUdWrDRWNyEN6noarr4GAVlVxz585ty5cvt0i1a9cuzvqXLFniHrt27Rq2bTv2c9qXFi1ahDyXL18+F8IK5WAqASvo47N///7A7wrsSy+91LWd33jjje5CQ+35Wq8X7LJo0aJAiF9++eXuYkD0qJ/Nmze79ar5QCVyxKWqya4Pv28L/9xgbzxxsxsmFh+VlrNmDX8RpKrw2L2Ck7o8MqbjJ05atnjOk3+27XGPam5JCfpeGDlpnmXPlsXat6yXIuvMaAjv08ibN+4cvQpbBeaxY8dCnouPSsbxKVUqpuQTzFtv7A5hCe1juLmHVcpVadqjiw0Ftzrg9enTx5XstQ299/vvvw9p75YcOXLYpEmT7IcffnA1AKpaX7x4sdvnxx9/3F14iFcTsGnTJvcTn9g1Boixe+9Bu6HX2/bHqn/skbvb2EV1Ktq6v0PH06o3bqH8MZ3+tFzv54db17YNrWblUq4qc9vOvTbuh99s1KSYKVDbt7gw8N6kLo+MSZOx9Ht1tN10zaVWvVJJK1Yon+tt/tuf623ol9PdMpdfEtr/4tDho27aU9m6Y6971Bz63vmbN3dON1NbbD8vXGUb/tnpOsJ55zWShvA+DVU7d+nSJeQ5te2q57Xalz1eZyyVLmNT4CVF+fLl3aPat2+6KXTyjkh4JXp1blO1f7Avvvgi7HsU7C1btnQ/ot7pCm71IK9UqZKdf/75Vq5cOfea2sVvuOGGeLevNnPEtWj5Rhew8soH37qf2B64taX1uyemiUallZXrt1n/wWPDHs4Hb29ljS6qHvg7qcsjY8qeNYstX7PZnhz8v3jnHX/gtn+bz2Tmryvsric+CnlOc53rR9Sv4p0Bod81MnJiTEe1TvSzSDbC+zQUao0aNbKOHTu6v1UF3L17d/d7cFW11/lLpVP10FaJViVu/a1SbVKo6jl//vw2atQoq169uvXo0SNQVa62cPUgV8/wpFKJW2bMmGGdOnVy61TJXJ9RQ9liUw90Df9S04F3caIaBlV56X2qLld4N2vWLLC/OlbNmze3rFmzBsJePfZ1AXPeeecleZ8zAg0DqxjPnM+eQgVyB36vVbWMzR75uJvcQqWlLTv2uhJOneplXPuhvmSDJXV5ZEx1apSzmSOecG3RS1dsss3bo9xUvZXLF7cWl9W2a66oG2c4YZ7cOeKdr1yKFYlb6lZp/a+1W+yCc8pb0/o1zshnyQgyRUVFxTRQIoR6b/fq1ct1/FJbrsJLPxs3bnTBpdKxOn55VdsKKbUJa8iUqtTVtrt161Y3BOySSy6xX375xfX+rlu3bsj61Wmsbdu2cY6+hmJ5AZ09e3ZXalVvc/00bdo0MJe5wlI9wFeuXBlnHXpNY741flzU7qx91H6pTVzV31u2bHHPq21b+6ge4erpLhpvrrHoojHm+vxe+7XCWZ9fveFFnfq6devmjo2q8bW8qv+1vC5iLrvsMlcFn6BseXR1wJmINHHiJF+FSCPR0Zb1VNxhxglhVobT0DAttQ8fPXrUBbPCSaGoSVaC26SzZcvmSq9Vq1Z1ga0JUBTi6n3dpEnoMJ3EUGc19WZXBzGF4Nq1a11wX3DBBa4DWXIosFXKVmleAazAV0landjuuSfuXatUOtfQMU1Mo+aAFStWuPfpgkY95L3gFtVM6IJCFwH6/Bp+pvZvhbx65Pfr1y9Z+wwAiIuSdzxil4wVSBrzrCDzqp/js2HDBjtx4oTrla3wUvDpR2O1vV7aCmKFm5Y5Xcc0ta9rLLdKyqqeDqZZ37St4CANfk2l3tizrIlXK6ASvUr2e/fudfut9mt9xti0/wpuzbAWrhNfsH379rn1ezUQ4TrThUXJG2mIkjf8VPImvONxumptnAGEN9IQ4Y00Q7U5AADpH23eAAD4DNXm8UhKmzRSCNXmSENUmyPN0OYNXyO8kYYIb6QZ2rwBAEj/aPMGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfIbwBgDAZwhvAAB8hvAGAMBnCG8AAHyG8AYAwGcIbwAAfCZrWu8AEBAdzcFA2uH8g4/OPcIbZ48Th9J6D5CB8WUIP6HaHAAAnyG8AQDwGcIbAACfIbwBAPAZwhsAAJ8hvAEA8BnCG8jAduzYYf369bMJEyak9a7A5ziXUhfhDWRge/bssffee89+/vnntN4V+BznUuoivAEA8BnCGwAAn2FGQKR7w4YNs7Vr19oLL7xgmzdvtq+//tq2b99ulSpVsg4dOli+fPnCvm/27Nk2d+5c27dvn5UqVcpatWpllStXTtZ2169f79qVd+3aZdWqVbN27dpZnjx5QpZ//fXX7fjx4/boo4/GWZdeO3XqlPXt2zfs+jdt2uTWr893/fXX2/nnn++WOXnypP3000/2+++/25EjR9z+t2nTxgoXLhx2n6dNm2Zz5syxTJkyWZMmTaxRo0Zhl1u+fLn98ssvtnHjRsuZM6fbXvPmzS1btmxxlj169KhNnjzZVq1a5X6vWLGiW7ZkyZJxlj1x4oR9//33tnjxYjt8+LBVqFDB/vOf/1jx4sUtI3v//ffdv/Fzzz1n69atc//Wu3fvturVq7tzKXfu3CHLDxw40D0Gny/Br2XOnNn69OkTdv36N504caI7lzp16mR16tRxy3AunV0yRUVFcTcIpGtt27Z1QTxmzBi75ZZb7MCBA4HXFA5Tp061EiVKBJ7bu3evW27GjBkh68mSJYv7wnviiSeStN1PPvnE7rrrLhdcnnLlyrmLiKpVqwaeq1+/vgvYpUuXxlmXXlOwKYRjr/+zzz6zO++8071XPvjgAxfgS5Yssdtvv93WrFkTsq78+fPb559/bk2bNrWVK1faxRdfbPfcc49rsxw9enTIsj179nRf6LE/V+xjIwoSfaYyZcoEnvvrr79cuGzdujVkWQW+QqRLly6B5/744w933HVBEkzB9M4777j1ZFRXX321LViwwIYOHWrdunWzY8eOhZzD48aNcxejnrp167rH4PMl+LWsWbPar7/+Gmf9CnGdC9659PHHH1v79u05l85ClLyRIajUqiBTKaJx48bub4X5hg0b7JVXXrHXXnstsOwDDzzgwkkhd8MNN7gSooJl/Pjx9uqrr1qVKlWsc+fOid7ufffdZxdccIELS30pqtSk0pOCSuGrUlAkn+vee+91JV99LpWo9bt6/l533XW2c+dOd6Fw1VVXuQsUbffbb791nzuYQlv71rVrVxcCq1evds8NGTLEbrrpJjvnnHMCy86bN89dTFx44YUuqPfv328LFy50JXz1XNeFgUfHS8F93nnnWcuWLV1tg/bhxx9/dMHu0YVDx44d3bKXXHKJK/FrWS2jYFKgnHvuuSEXOxmNLt50MVWvXj1XK3Lo0CH75ptv3L/lrbfe6s5Z1ZhEsv7777/fnas6lwoVKuT+v3AunZ0Ib2QI0dHRLiyDS5EKBJVCVFXsUdWhwqJAgQI2a9YsK1++fOC1sWPH2h133GFvvvlmosNb21VJVe/xvlgfe+wxVwWvkrG+cK+44oqIPpdCevDgwSHPv/TSSy64W7dubZ9++qnlyJEj8JqaAVTlGky1Agrf4JCuWbOmPfXUUzZ9+vSQ51UFrpKbjs+2bdvce1XqXrZsmQvlYAp2NUto3XpP8PYU4sFNAApu1Wo8/PDDIevQxYNK3SoFqokgo1K1tS5wBg0aFHhO55KaIHQu6d9DoR7J+nWxGnwhK5xLZyfCGxlGr169Qv4uVqyY1apVy/7888/AcypBKhBVAg0OblFIqpSukFLVe968eRO13YceeiikRJQrVy5XwlH1p6oqIwlvr6YgNrVbi8IuOLhFNQr6CaZScXBAixcECujg0tm7774bp3o9mEqEXhusqu/Vhv3iiy+6ANbxVohrn3Rx4FENhGzZssX69+/v/g1Ej/rRexRQGV1wO7XoOOtc6tGjhzt3IwlvefDBB+M8x7l0diK8kSGovVphHZuCVIEUXCqVsmXLhl2PqqBVlasSZWLDu3Tp0nGe89qFve2djhdmsanKPfZFhkRFRbkLhnCvhROu85gXwMHHR6V4BXfRokVdRzIdD7Vfa1tffPGFu7BRVb5H4a11q11eJWhdCKgqVu2od999t3uvqCOffPTRR/HuY3BfhYxIx/hMnkvqbBjcX8HDuXR2IryBIF4vbLX5hqPnFZgFCxZM9HFTB6zgUqZ4nciKFCkSEpZqXwxXnfnPP/+EDVh9oYdrM9eFir6kdaHh9RZOCV4Tg0rTwR2kROEdjtpP9eMFgZoKVFugiWFGjRrlntfFgDz99NNxago8Gb3Huf491dQQu93fO5eCRxDoXAquMfFoNINqN8Jd1Ok8Ctdmzrl0dmKcNxBEPa9VRTt8+HBbtGhRyLF5++233ZenOmqpxJ5YAwYMCOlprmFqGvoll156aeB5faEq3GbOnBnyha0qZw2bSgqvKl7tx7Hbt9XBSR3wksMbCqYQCKZeyip1x6ZOfuq979FFz7XXXuuGy02ZMsUOHjzonm/WrJl71OdXBzxVAwf/qAOb3pPRPfPMMyE9zdVPwOvv0LBhw5BzSX0evCpv71xSM0rwuZgYnEtnJ0reQKzS3W233WYffvihtWjRwoWK19tcbYoSu0NVQlSa0XAdXRToy1U9ulV6VUhddtllrme1R9XQ6omuqmb1DldHL21T427V8zcp9BnUCUzj1NV7WOGnz6ZaAI3PVqck9d5OKn0GDQdTJyl1ulO1t8Zka3ibqlxVQxDs+eefd2GsHtIa362Sndr5FfQq0XnV5uoIqF7qb7zxhquWb9CggStJKoA0PlzL64LHG7+eEemicv78+e5c0kWfLuh0LuniSG3dOsbB55I6Fqqfhs4l9dz/7bffXIdMdcZMCs6lsxPhDcSi0olCVqVvlQ49ClO9ptBKLIWV2nHV3jty5MjA8/qiVe/pYAptBeN3333nerx729T71Qs7uO35dPQ+lXq7d+/uwlrDw4LHBScnuEXD7dSjXPuooXZefwKFtIaQxQ5vdVLTuHP1hNZP8D6oJkPvFYWLhj317t3bHXMdh2C6AMnIwS06VjoXbr755pBzScP2YvcV0GgI/ftrDgPvWKqTouYcCDdxS0I4l85OTNKCdE+hoNmiFGThXlPVo3p+x6ZZ0RRI6pym0rfabZNSavEmUVFnLJWONFxKVdiq/lVJOL7x3Sotqwe8vmxVwlWpW1/W6gimL+7EfK5gqjVQ6VjhrzHqKtV6w7ZUAzBixAg3Djv2bGreawrO4CpZ0QQfmmVNJWcdF81Ap9BV+6s6ogXPtKYLIS2vJgc9rxK4ahzi+/w67iqda/sam16jRg03FC0j8yZRUTu2dy5pbLzOJdXgxHcsVW2ufyedtzqX1Gyhi1Itf+ONNwaWU9Br3eH+HwTjXDp7EN7AGRIc3kBKhTcgdFgDAMBnCG8AAHyGDmvAGaKbhainLxAptUWrGQbw0OYNAIDPUG0OAIDPEN4AAPgM4Q0AgM8Q3gAA+AzhDQCAzzBUDPAx3ZpT06QG09SXmiu8cuXKduWVV8a5dWda+vvvv+3ll192U74+++yzIa9pyk/Nw62bbmgu+EhoqtaJEye6KV87depkZ8tnTEhKfn7ReaHzQ+eA5phPDSn9GRA/whvwMc01rbtxJXRjlK5du7q7iAXPN55WNLe79rd06dJxgk3zuXufJdIv/iVLlrh1aQ731A7vhD5jQlLy84vuIqb1aV7z1ArvlP4MiB/hDaQDZcuWtUceecT9rhuYqPSnO4npVpqfffaZ5ciRw1599dW03k0AKYTwBtIB3S3qlltuCXnuscces1tvvdVVH+v2o/pb98g+W+ne6Sol6k5ZABJGeAPp+P7PKo0rvHU70N9//921f+r3Bx980C0zcOBAO378uE2ePNlVwes2k5rWNfje2bqb1YwZM9ytU7Nnz261atVyd7nSBUN8dCe1sWPH2sqVKy137txWt27d004Vqztm6dahqupWm2k4+gwzZ8509w3PlSuXa9dv3bq1u3Wop2fPnrZ06VL3+88//+z+9mi5/v37x1lvan3G5IqOjrb58+e7z79p0yb3b1i+fHl3b/mqVasmuhpdbeA7duywokWLWps2bdytYE8nOccGZx7hDaTz6nTPvn373KO++L12yY4dO9q9997r7gvu0X2fFd5btmxx9+bWbU1je/zxx+3NN98MO9/2Dz/8YHfddZe7H3cw3ZP7iSeeSFZ7qcL6nnvuCbsvastX27I+hwT3AVCw6sejUn1weKf2Z0wOhedtt93mmkJie/LJJ9285y+++GK89/RWM8p9993n7uMdTJ3qdC94vVd9I2JL7rFB6iC8gXRMgehRaSs2VbXri1vBpxKcglDBrVBSKXL9+vWuqr1Dhw6u1/r+/ftt2rRpNnfuXLvjjjts/Pjxrke356+//rIuXbrYkSNHrEKFCta5c2e33UWLFtmXX35pvXr1SvJnUAlXJUyFl9ruFRgq5eoiZM2aNa7WIDigFSrqbT5p0iS77LLLQjqsqVreczZ9xoTowkWl3pYtW1rNmjWtTJky7kJMtRRTp0619957z3WOi2+7Cu29e/faxRdfbNdcc42rkVGvcF2AvPvuu27fH3rooZD3JPfYIPUQ3kA6pS98lZAkX758Vq9evTjLKKynT58eUkKXvn37ui9uVV+PGTPG8ubNG3jt0Ucfteeff95Vuavkpy9zzzPPPONCTRcACs/g9ykArrvuuiR/Dq1Twa0qbw1/qlGjRsjr6oi3bt26kAsSVcFr+yoJx+4L4NFnOFs+Y0Lq16/vOh4WL148zmuqtleI6oJFF2DhRhQouHWx8dZbbwWe69Gjhyt5v/TSS+74qWRfpEiRiI8NUg+TtADpgEpnatvVj6pI27dv78Laa/tVVa7aZWNTiSt2cB89etRGjBjhftcXfvAXt0cXBfnz53dtsF6Vu0pralOVAQMGxHmfOqSprTQpDh065EqzorCJHdyiwFJIJ8XZ9BlPR6XqcMEtulBQG/TOnTtDalmCacy/9jU29XvQv70uRILnCkjusUHqouQNpAPqaBZuvLdKq+plrpJVOOGqPBcvXuxCU52TBg8eHO821bFM1q5d6wJGY6tPnjzpvuwvv/zysO9Rta1Kq4mlquhjx465zmkpGYpn02dMyrGYM2eO67CmfVdbdnBfho0bN9oFF1wQ530NGjQIKVV79NlbtGjhRiIohG+//faIjg1SF+ENpLNx3qJwURtlnTp1Al+y4ZQrVy5sRyVRaCY0AYzn4MGDgclJ4lunRz2kk0I9o0XhkJKTzJxNn/F0VKpWsM6aNSvB5Q4cOJDk/fE+h3ecIzk2SF2EN5BOx3knRrhg93oeqyPTU089ddp11K5dO+R9qnaNjwIhKbwe1OqclpLOps94OmrTVnDrgkydyNRpTf/e3sXMBx984JpHNJwsnIT21Xst+DxI7rFB6iK8AYTw2sBVytS0mmrbTAyv6lSdyxS24S4M1AkqKbySodapns7qeJfePmNCVEWuce0K1O+++87OPffcOMuMHDkywXUktD/eayVLloz42CB10WENQAhN3KGbaqhNVe2hiaUqenWKU8lz9OjRYZdJTDVs7HVqqJLamT/99NNEv0/ttaIJaM72z5gQrzOYLhrCBbde1+QrCZk3b56tXr06zvPqfKcpdL128UiPDVIX4Q0ghEqT999/v/v9hRdecMORwtF45y+++CLwd86cOe366693v2so0apVq0KWf+ONN2zhwoVJOtoak6zJWbx16o5VsSmYvB7gnmLFirlHDbE62z9jQrwSsUJas5wFU0cxDQFLqFpcdOHzwAMPuCFjHvUw10gD1WZ4s61FemyQuqg2BxCHvuxVYtMkIGpz1VAjTYyiEq3Gj2/YsMH9qDSoAPFo3K/eo7BRT/bGjRu7krPaZJcvX+5mOIsdeKfTp08f1+armb7UcUuBol7VKlVrkhYNkVJvevWc9mhyFlU1q1Sq/VC7rPY9eHrUs+kzxkeTwGgb+uyqwta4a1VrqzrdK3FXrFgxwapxTb6j9+uYNWnSxIWzeq2rY5qOkcZrqzd/sOQeG6QewhtA2BKv2lI1VEhjfb0v62AaX6zbjQZTKW7ChAlu6lANbdIsXh6Fj6byDC7lJYY6Zv3vf/9zJd0PP/zQlbSDq4EVlsHB7YWeJlN57rnn3Jzt+vGW9cL7bPqMCRk2bJi7OPnll19c6Ab3B9DkLOqwllB4a+53zQGvz60Z0YJnm3vllVfC3i40uccGqSdTVFRU+C6KAM56CiWVwNRGqfHFiaFqVG+ea80hntBQMlHHLIWUxhHrvarKVWkvoeFSorHDKoFqSlOV+hSoGo+u4FO7seZVD6YStKqcFbDx3ZhE4481LahKf+q8pv1QiMRHQ6C0Tk2xqn1XYIWbjzu1PmNCTvf5NcZcVdXqga99u/DCC93vmiFP+63ahipVqgSW13mh80O1DpqwR0PJNK2php7pAqRhw4ZhJ+6J5Ngk5t8QKYPwBgDAZ+iwBgCAzxDeAAD4DOENAIDPEN4AAPgM4Q0AgM8Q3gAA+AzhDQCAzxDeAAD4DOENAIDPEN4AAPgM4Q0AgM8Q3gAA+AzhDQCAzxDeAAD4zP8Br/fEY8ungNwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 400x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(4, 4))\n",
    "ConfusionMatrixDisplay.from_predictions(\n",
    "    y_test, y_pred, display_labels=['no purchase', 'purchase'], cmap='Blues', colorbar=False, ax=ax)\n",
    "ax.grid(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c018",
   "metadata": {},
   "source": [
    "Of the 572 buyers in the test set, the classifier caught 317 (true positives) and missed 255 (false negatives). Of the 3,127 sessions without a purchase, it correctly left 2,969 alone (true negatives) and flagged 158 by mistake (false positives).\n",
    "\n",
    "Accuracy is just the diagonal divided by the total:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c019",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.942448Z",
     "iopub.status.busy": "2026-11-17T23:43:45.942372Z",
     "iopub.status.idle": "2026-11-17T23:43:45.943997Z",
     "shell.execute_reply": "2026-11-17T23:43:45.943730Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy from the matrix: 0.888\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy from the matrix:', round((tp + tn) / (tp + tn + fp + fn), 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c020",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.944897Z",
     "iopub.status.busy": "2026-11-17T23:43:45.944842Z",
     "iopub.status.idle": "2026-11-17T23:43:45.946459Z",
     "shell.execute_reply": "2026-11-17T23:43:45.946184Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Count the Four Outcomes\n",
    "# A fraud model was checked on ten transactions (1 = fraud, the positive class).\n",
    "#   1. Count the true positives, false positives, false negatives, and true negatives\n",
    "#      by comparing ex_actual and ex_predicted.\n",
    "#   2. Check your counts with confusion_matrix(ex_actual, ex_predicted).ravel().\n",
    "# Print exactly:\n",
    "# TP: 2 FP: 1 FN: 1 TN: 6\n",
    "# sklearn: [6 1 1 2]\n",
    "ex_actual = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 0])\n",
    "ex_predicted = np.array([0, 1, 1, 0, 0, 0, 0, 1, 0, 0])\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c021",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.947461Z",
     "iopub.status.busy": "2026-11-17T23:43:45.947398Z",
     "iopub.status.idle": "2026-11-17T23:43:45.949977Z",
     "shell.execute_reply": "2026-11-17T23:43:45.949724Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 2 FP: 1 FN: 1 TN: 6\n",
      "sklearn: [6 1 1 2]\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "ex_actual = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 0])\n",
    "ex_predicted = np.array([0, 1, 1, 0, 0, 0, 0, 1, 0, 0])\n",
    "\n",
    "ex_tp = np.sum((ex_actual == 1) & (ex_predicted == 1))\n",
    "ex_fp = np.sum((ex_actual == 0) & (ex_predicted == 1))\n",
    "ex_fn = np.sum((ex_actual == 1) & (ex_predicted == 0))\n",
    "ex_tn = np.sum((ex_actual == 0) & (ex_predicted == 0))\n",
    "\n",
    "print('TP:', ex_tp, 'FP:', ex_fp, 'FN:', ex_fn, 'TN:', ex_tn)\n",
    "print('sklearn:', confusion_matrix(ex_actual, ex_predicted).ravel())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c022",
   "metadata": {},
   "source": [
    "## Precision, Recall, and F1\n",
    "\n",
    "```{index} precision\n",
    "```\n",
    "\n",
    "```{index} recall\n",
    "```\n",
    "\n",
    "```{index} F1 score\n",
    "```\n",
    "\n",
    "```{index} sensitivity\n",
    "```\n",
    "\n",
    "Two summary rates answer the questions a manager actually asks about a flagging system.\n",
    "\n",
    "**Precision** answers: *of the sessions the classifier flagged, what fraction really ended in a purchase?*\n",
    "\n",
    "$$\\text{precision} = \\frac{TP}{TP + FP}$$\n",
    "\n",
    "**Recall** (also called *sensitivity*) answers: *of the sessions that really ended in a purchase, what fraction did the classifier catch?*\n",
    "\n",
    "$$\\text{recall} = \\frac{TP}{TP + FN}$$\n",
    "\n",
    "Low precision means the marketing team spends coupons and attention on visitors who were never going to buy. Low recall means many buyers go unnoticed and get no nudge. The two usually pull against each other: flagging more sessions raises recall but tends to lower precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c023",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.950967Z",
     "iopub.status.busy": "2026-11-17T23:43:45.950906Z",
     "iopub.status.idle": "2026-11-17T23:43:45.956042Z",
     "shell.execute_reply": "2026-11-17T23:43:45.955576Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.667\n",
      "Recall:    0.554\n",
      "sklearn precision: 0.667\n",
      "sklearn recall:    0.554\n"
     ]
    }
   ],
   "source": [
    "tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()\n",
    "\n",
    "precision = tp / (tp + fp)\n",
    "recall = tp / (tp + fn)\n",
    "\n",
    "print('Precision:', round(precision, 3))\n",
    "print('Recall:   ', round(recall, 3))\n",
    "print('sklearn precision:', round(precision_score(y_test, y_pred), 3))\n",
    "print('sklearn recall:   ', round(recall_score(y_test, y_pred), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c024",
   "metadata": {},
   "source": [
    "Precision is about 0.67: roughly two out of every three sessions the retailer flags really end in a purchase. Recall is about 0.55: the classifier catches about 55% of the buyers and misses the other 45%.\n",
    "\n",
    "The always-\"no purchase\" baseline flags no one, so its recall is 0: it catches none of the buyers, which accuracy completely hid. (Its precision is undefined, since it makes no positive predictions; scikit-learn reports 0 and, with `zero_division=0`, does so without a warning.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c025",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.957988Z",
     "iopub.status.busy": "2026-11-17T23:43:45.957907Z",
     "iopub.status.idle": "2026-11-17T23:43:45.962358Z",
     "shell.execute_reply": "2026-11-17T23:43:45.962066Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline recall:    0.0\n",
      "Baseline precision: 0.0\n"
     ]
    }
   ],
   "source": [
    "print('Baseline recall:   ', recall_score(y_test, y_base))\n",
    "print('Baseline precision:', precision_score(y_test, y_base, zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c026",
   "metadata": {},
   "source": [
    "```{index} harmonic mean\n",
    "```\n",
    "\n",
    "When a single number is needed, the **F1 score** combines precision and recall. It is their *harmonic mean*, which is high only when both are high:\n",
    "\n",
    "$$F_1 = \\frac{2 \\times \\text{precision} \\times \\text{recall}}{\\text{precision} + \\text{recall}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c027",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.963516Z",
     "iopub.status.busy": "2026-11-17T23:43:45.963447Z",
     "iopub.status.idle": "2026-11-17T23:43:45.966494Z",
     "shell.execute_reply": "2026-11-17T23:43:45.966117Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 by formula: 0.606\n",
      "sklearn F1:    0.606\n"
     ]
    }
   ],
   "source": [
    "print('F1 by formula:', round(2 * precision * recall / (precision + recall), 3))\n",
    "print('sklearn F1:   ', round(f1_score(y_test, y_pred), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c028",
   "metadata": {},
   "source": [
    "F1 is a convenient summary, but like accuracy it treats the two kinds of errors as equally important. When the costs of the errors are known, it is better to use them directly, as in the next subsection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c029",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.967582Z",
     "iopub.status.busy": "2026-11-17T23:43:45.967517Z",
     "iopub.status.idle": "2026-11-17T23:43:45.969074Z",
     "shell.execute_reply": "2026-11-17T23:43:45.968812Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Precision and Recall for a Fraud Model\n",
    "# A fraud model was checked on a month of transactions:\n",
    "# it flagged 50 transactions, of which 30 were really fraud,\n",
    "# and there were 40 fraudulent transactions in total.\n",
    "#   1. Work out TP, FP, and FN from these facts.\n",
    "#   2. Compute precision, recall, and F1, each rounded to 3 decimals.\n",
    "# Print exactly:\n",
    "# Precision: 0.6\n",
    "# Recall: 0.75\n",
    "# F1: 0.667\n",
    "flagged = 50\n",
    "flagged_and_fraud = 30\n",
    "total_fraud = 40\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c030",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.970052Z",
     "iopub.status.busy": "2026-11-17T23:43:45.969990Z",
     "iopub.status.idle": "2026-11-17T23:43:45.971876Z",
     "shell.execute_reply": "2026-11-17T23:43:45.971584Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.6\n",
      "Recall: 0.75\n",
      "F1: 0.667\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "flagged = 50\n",
    "flagged_and_fraud = 30\n",
    "total_fraud = 40\n",
    "\n",
    "ex_tp = flagged_and_fraud\n",
    "ex_fp = flagged - flagged_and_fraud\n",
    "ex_fn = total_fraud - flagged_and_fraud\n",
    "\n",
    "ex_precision = ex_tp / (ex_tp + ex_fp)\n",
    "ex_recall = ex_tp / (ex_tp + ex_fn)\n",
    "f1 = 2 * ex_precision * ex_recall / (ex_precision + ex_recall)\n",
    "\n",
    "print('Precision:', round(ex_precision, 3))\n",
    "print('Recall:', round(ex_recall, 3))\n",
    "print('F1:', round(f1, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c031",
   "metadata": {},
   "source": [
    "## The Business Cost of Errors\n",
    "\n",
    "```{index} cost of errors\n",
    "```\n",
    "\n",
    "```{index} expected cost; classification\n",
    "```\n",
    "\n",
    "```{index} cost matrix\n",
    "```\n",
    "\n",
    "To decide whether a classifier is worth using, put a dollar value on each kind of error. The data do not record costs, so the numbers below are assumptions made for this example, the kind of estimate a finance team would supply. Suppose the retailer plans to send every flagged session a nudge, a reminder email with free shipping, at a cost of about \\$3 per session.\n",
    "\n",
    "- **False negative (a missed buyer): \\$30.** Without a nudge, the retailer loses about \\$30 of profit it would have gained by closing the sale.\n",
    "- **False positive (a flagged non-buyer): \\$3.** The retailer pays for a nudge to someone who was not going to buy.\n",
    "\n",
    "Together with a cost of zero for each correct prediction, these two numbers form a simple *cost matrix*: one cost for each cell of the confusion matrix.\n",
    "\n",
    "To keep the arithmetic simple, assume that the nudge always secures the sale of a correctly flagged buyer, and count only the cost of the *errors*. (A fuller analysis would also count the cost of the nudges sent to true positives; the comparisons below would come out the same way, only with different totals.)\n",
    "\n",
    "The total cost of the errors on the test set is\n",
    "\n",
    "$$\\text{cost} = 30 \\times FN + 3 \\times FP .$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c032",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.972827Z",
     "iopub.status.busy": "2026-11-17T23:43:45.972771Z",
     "iopub.status.idle": "2026-11-17T23:43:45.977094Z",
     "shell.execute_reply": "2026-11-17T23:43:45.976734Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kNN cost:              $ 8124\n",
      "Baseline (flag no one): $ 17160\n",
      "Flag everyone:          $ 9381\n"
     ]
    }
   ],
   "source": [
    "COST_FN = 30     # dollars lost for each buyer the model misses (an assumption)\n",
    "COST_FP = 3      # dollars wasted for each non-buyer the model flags (an assumption)\n",
    "\n",
    "def error_cost(actual, predicted):\n",
    "    \"\"\"Total dollar cost of the false negatives and false positives.\"\"\"\n",
    "    tn, fp, fn, tp = confusion_matrix(actual, predicted).ravel()\n",
    "    return COST_FN * fn + COST_FP * fp\n",
    "\n",
    "print('kNN cost:              $', error_cost(y_test, y_pred))\n",
    "print('Baseline (flag no one): $', error_cost(y_test, y_base))\n",
    "print('Flag everyone:          $', error_cost(y_test, np.ones(len(y_test), dtype=int)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c033",
   "metadata": {},
   "source": [
    "The kNN classifier's errors cost \\$8,124 on the 3,699 test sessions, less than half of the \\$17,160 lost by flagging no one, the baseline that looked 85% accurate. The third line is the surprise: flagging *every* session costs \\$9,381, only about \\$1,300 more than the classifier, even though its accuracy is just 15.5%. Because a missed buyer costs ten times as much as a false alarm, its 3,127 cheap false alarms cost far less than the baseline's 572 expensive misses. Accuracy and cost can point in opposite directions: flagging everyone has the lowest accuracy of the three, yet the second-lowest cost.\n",
    "\n",
    "The same reasoning applies to fraud. If a missed fraudulent transaction costs a bank an average of \\$500 and a false alarm costs \\$20 of an analyst's review time, a classifier that catches more fraud can be worth many false alarms. The costs, not the accuracy, decide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c034",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.978129Z",
     "iopub.status.busy": "2026-11-17T23:43:45.978051Z",
     "iopub.status.idle": "2026-11-17T23:43:45.979590Z",
     "shell.execute_reply": "2026-11-17T23:43:45.979289Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: The Cost of a Fraud Model\n",
    "# On 1,000 test transactions, a fraud model had TP = 15, FP = 40, FN = 5, TN = 940.\n",
    "# A missed fraud (FN) costs $500 and a false alarm (FP) costs $20.\n",
    "#   1. Compute the total error cost of the model.\n",
    "#   2. Compute the error cost of approving everything (every fraud is missed: FN = 20, FP = 0).\n",
    "# Print exactly:\n",
    "# Model cost: 3300\n",
    "# Approve-all cost: 10000\n",
    "ex_tp, ex_fp, ex_fn, ex_tn = 15, 40, 5, 940\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c035",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.980424Z",
     "iopub.status.busy": "2026-11-17T23:43:45.980367Z",
     "iopub.status.idle": "2026-11-17T23:43:45.982072Z",
     "shell.execute_reply": "2026-11-17T23:43:45.981754Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model cost: 3300\n",
      "Approve-all cost: 10000\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "ex_tp, ex_fp, ex_fn, ex_tn = 15, 40, 5, 940\n",
    "cost_fn = 500\n",
    "cost_fp = 20\n",
    "\n",
    "print('Model cost:', cost_fn * ex_fn + cost_fp * ex_fp)\n",
    "print('Approve-all cost:', cost_fn * (ex_tp + ex_fn) + cost_fp * 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c036",
   "metadata": {},
   "source": [
    "## Choosing Between Classifiers by Cost\n",
    "\n",
    "```{index} model selection; cost\n",
    "```\n",
    "\n",
    "```{index} comparing classifiers\n",
    "```\n",
    "\n",
    "Once each error has a price, you can compare classifiers on the measure the business cares about. The table below evaluates nearest neighbor classifiers with several values of $k$, plus the baseline, on accuracy, precision, recall, and total error cost.\n",
    "\n",
    "(As in Section {doc}`1501-knn-scikit-learn`, a careful analyst would make this comparison on a validation set and use the test set only for the final report. The test set is used here to keep the example short.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c037",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:45.982952Z",
     "iopub.status.busy": "2026-11-17T23:43:45.982898Z",
     "iopub.status.idle": "2026-11-17T23:43:46.047804Z",
     "shell.execute_reply": "2026-11-17T23:43:46.047471Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classifier</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>error cost ($)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>baseline (flag no one)</td>\n",
       "      <td>0.845</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>17160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>flag everyone</td>\n",
       "      <td>0.155</td>\n",
       "      <td>0.155</td>\n",
       "      <td>1.000</td>\n",
       "      <td>9381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kNN, k = 1</td>\n",
       "      <td>0.857</td>\n",
       "      <td>0.536</td>\n",
       "      <td>0.554</td>\n",
       "      <td>8472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kNN, k = 5</td>\n",
       "      <td>0.878</td>\n",
       "      <td>0.623</td>\n",
       "      <td>0.537</td>\n",
       "      <td>8508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>kNN, k = 11</td>\n",
       "      <td>0.888</td>\n",
       "      <td>0.667</td>\n",
       "      <td>0.554</td>\n",
       "      <td>8124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>kNN, k = 21</td>\n",
       "      <td>0.889</td>\n",
       "      <td>0.664</td>\n",
       "      <td>0.566</td>\n",
       "      <td>7932</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               classifier  accuracy  precision  recall  error cost ($)\n",
       "0  baseline (flag no one)     0.845      0.000   0.000           17160\n",
       "1           flag everyone     0.155      0.155   1.000            9381\n",
       "2              kNN, k = 1     0.857      0.536   0.554            8472\n",
       "3              kNN, k = 5     0.878      0.623   0.537            8508\n",
       "4             kNN, k = 11     0.888      0.667   0.554            8124\n",
       "5             kNN, k = 21     0.889      0.664   0.566            7932"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def evaluate(name, predicted):\n",
    "    return {'classifier': name,\n",
    "            'accuracy': round(accuracy_score(y_test, predicted), 3),\n",
    "            'precision': round(precision_score(y_test, predicted, zero_division=0), 3),\n",
    "            'recall': round(recall_score(y_test, predicted), 3),\n",
    "            'error cost ($)': error_cost(y_test, predicted)}\n",
    "\n",
    "rows = [evaluate('baseline (flag no one)', y_base),\n",
    "        evaluate('flag everyone', np.ones(len(y_test), dtype=int))]\n",
    "for k in [1, 5, 11, 21]:\n",
    "    model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k))\n",
    "    model.fit(X_train, y_train)\n",
    "    rows.append(evaluate(f'kNN, k = {k}', model.predict(X_test)))\n",
    "\n",
    "comparison = pd.DataFrame(rows)\n",
    "comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c038",
   "metadata": {},
   "source": [
    "Among the nearest neighbor classifiers, $k = 21$ has both the highest accuracy (0.889) and the lowest cost (\\$7,932), with $k = 11$ close behind (0.888 and \\$8,124). That agreement is not guaranteed; the exercise below gives a case where the more accurate model is the more expensive one. This section keeps $k = 11$, the value chosen on the validation set in Section {doc}`1501-knn-scikit-learn`, because choosing $k$ by looking at the test set would spoil it as a final check. Every one of these classifiers beats flagging everyone on cost, but only by about \\$900 to \\$1,450, a modest saving.\n",
    "\n",
    "All of these classifiers predict \"purchase\" only when most of the neighbors ended in a purchase. Because a missed buyer costs ten times as much as a false alarm, it may pay to flag a session even when only a minority of similar sessions ended in a purchase. Changing that cutoff requires a classifier that reports a *probability* of purchase, which is where the next section begins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c039",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:46.049032Z",
     "iopub.status.busy": "2026-11-17T23:43:46.048967Z",
     "iopub.status.idle": "2026-11-17T23:43:46.050732Z",
     "shell.execute_reply": "2026-11-17T23:43:46.050326Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Which Model Is Cheaper?\n",
    "# Two models were checked on the same 300 test sessions.\n",
    "#   Model A: FN = 30, FP = 10\n",
    "#   Model B: FN = 12, FP = 60\n",
    "#   1. Compute each model's accuracy (300 sessions in total) and error cost,\n",
    "#      with a missed buyer costing $30 and a false alarm $3.\n",
    "#   2. Print the name of the cheaper model.\n",
    "# Print exactly:\n",
    "# Model A: accuracy 0.867, cost 930\n",
    "# Model B: accuracy 0.76, cost 540\n",
    "# Cheaper: Model B\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c040",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-17T23:43:46.051694Z",
     "iopub.status.busy": "2026-11-17T23:43:46.051625Z",
     "iopub.status.idle": "2026-11-17T23:43:46.053908Z",
     "shell.execute_reply": "2026-11-17T23:43:46.053449Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model A: accuracy 0.867, cost 930\n",
      "Model B: accuracy 0.76, cost 540\n",
      "Cheaper: Model B\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "models = {'Model A': (30, 10), 'Model B': (12, 60)}\n",
    "costs = {}\n",
    "\n",
    "for ex_name, (ex_fn, ex_fp) in models.items():\n",
    "    accuracy = (300 - ex_fn - ex_fp) / 300\n",
    "    costs[ex_name] = 30 * ex_fn + 3 * ex_fp\n",
    "    print(f'{ex_name}: accuracy {round(accuracy, 3)}, cost {costs[ex_name]}')\n",
    "\n",
    "print('Cheaper:', min(costs, key=costs.get))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
