{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c01",
   "metadata": {},
   "source": [
    "# Comparing Groups Without Random Assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c02",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.362605Z",
     "iopub.status.busy": "2026-10-24T04:20:03.362352Z",
     "iopub.status.idle": "2026-10-24T04:20:03.710384Z",
     "shell.execute_reply": "2026-10-24T04:20:03.710017Z"
    },
    "tags": [
     "hide-input",
     "thebe-init",
     "runnable"
    ]
   },
   "outputs": [],
   "source": [
    "path_data = '../../data/'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.dpi'] = 100\n",
    "plt.rcParams['axes.spines.top'] = False\n",
    "plt.rcParams['axes.spines.right'] = False\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03",
   "metadata": {},
   "source": [
    "```{contents}\n",
    ":local:\n",
    ":depth: 2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c04",
   "metadata": {},
   "source": [
    "In {doc}`1001-ab-testing` the store decided which visitors saw which checkout page. It assigned the pages by chance, so the two groups were alike apart from the page, and a difference in purchases could be traced to the page.\n",
    "\n",
    "Most business data does not come from an experiment. Customers choose a plan, click a button, join a loyalty program, or open an email. The two groups you can compare were formed by the customers themselves. This section applies the same permutation test to such a comparison, shows what the test can and cannot tell you, and introduces the main reason that choice-based comparisons mislead: **confounding**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c05",
   "metadata": {},
   "source": [
    "## The Live Chat Question\n",
    "\n",
    "```{index} observational study\n",
    "```\n",
    "\n",
    "```{index} association\n",
    "```\n",
    "\n",
    "An online store has a \"Chat with us\" button on its product pages. A customer who clicks it can ask a sales assistant a question before buying. Marketing has a proposal: put the button on every page, and staff more assistants to answer it.\n",
    "\n",
    "The proposal rests on a hunch that chat helps customers decide to spend more. To check it, an analyst pulls 300 recent orders and records, for each one, whether the customer used chat, whether the customer had ordered before, and the order value. (The store is hypothetical and the orders are simulated for this book, built so that returning customers both use chat more and spend more.)\n",
    "\n",
    "Nobody assigned these customers to chat or to no chat. Each customer decided. A study in which the researcher only records what happened, without assigning the groups, is an **observational study**. In an observational study, a difference between the groups is at best an **association**: the two things go together in the data. Whether chat itself changes what customers spend is a different question, and the rest of this section is about the distance between the two."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06",
   "metadata": {},
   "source": [
    "## Comparing the Two Groups\n",
    "\n",
    "```{index} groupby; comparing groups\n",
    "```\n",
    "\n",
    "Load the orders and look at the first rows. Each row is one order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c07",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.711763Z",
     "iopub.status.busy": "2026-10-24T04:20:03.711666Z",
     "iopub.status.idle": "2026-10-24T04:20:03.717913Z",
     "shell.execute_reply": "2026-10-24T04:20:03.717439Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>used_chat</th>\n",
       "      <th>returning_customer</th>\n",
       "      <th>order_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>O0001</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>O0002</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>56.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>O0003</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>72.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>O0004</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>36.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>O0005</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>73.99</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  order_id  used_chat  returning_customer  order_value\n",
       "0    O0001          0                   0        23.09\n",
       "1    O0002          0                   0        56.80\n",
       "2    O0003          1                   1        72.53\n",
       "3    O0004          0                   1        36.12\n",
       "4    O0005          0                   0        73.99"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orders = pd.read_csv(path_data + 'chat_orders.csv')\n",
    "orders.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08",
   "metadata": {},
   "source": [
    "The column `used_chat` is 1 if the customer used chat and 0 if not. The column `returning_customer` is 1 for customers who had ordered before. Start with the question marketing asked: how many orders used chat, and how do the two groups compare in order value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c09",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.719332Z",
     "iopub.status.busy": "2026-10-24T04:20:03.719271Z",
     "iopub.status.idle": "2026-10-24T04:20:03.722414Z",
     "shell.execute_reply": "2026-10-24T04:20:03.722082Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Orders with chat:    62 (20.7%)\n",
      "Orders without chat: 238\n",
      "\n",
      "Mean order value, no chat: 69.68\n",
      "Mean order value, chat:    85.6\n"
     ]
    }
   ],
   "source": [
    "chat_counts = orders['used_chat'].value_counts()\n",
    "print('Orders with chat:   ', chat_counts[1], f'({chat_counts[1] / len(orders):.1%})')\n",
    "print('Orders without chat:', chat_counts[0])\n",
    "\n",
    "group_means = orders.groupby('used_chat')['order_value'].mean()\n",
    "print()\n",
    "print('Mean order value, no chat:', round(group_means[0], 2))\n",
    "print('Mean order value, chat:   ', round(group_means[1], 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c10",
   "metadata": {},
   "source": [
    "The 62 customers who used chat spent more, on average, than the 238 who did not. The histograms show the two distributions. Because the groups are different sizes, the vertical axis uses the share of orders in each group rather than counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c11",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.723383Z",
     "iopub.status.busy": "2026-10-24T04:20:03.723330Z",
     "iopub.status.idle": "2026-10-24T04:20:03.792264Z",
     "shell.execute_reply": "2026-10-24T04:20:03.791901Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAngAAAGICAYAAADS7Z2+AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAURRJREFUeJzt3Qm8TPX/+PH3tVz7eu27ZF9KdkIifGXPFilE5GsnS6USJSpLFEKbcO1CCklKtkRCZN93rn1n/o/35/s785+5m7lz596Ze+7r+XgM95z5nHM+85ntPZ81yOFwOAQAAAC2kcTfGQAAAIBvEeABAADYDAEeAACAzRDgAQAA2AwBHgAAgM0Q4AEAANgMAR4AAIDNEOABAADYDAFeAvLYY4/JyJEjJdAdPXpUihUrJvPnz/d3Vmz5/CWU8r106ZLJ5zfffOPvrCQYgfDcxiQPsX2OA+HxAnZFgOdDf/75p/z3v/+VqlWrmi/z+vXry4cffihXrlzxyfn//fdfOXPmjAQCzYd+MM+cOTPCfXfu3DF51Q//+LheQuGr5y+hlO+9e/dMPi9evBin17GTQHhuY5KHyJ5jfz9eAP9DgOcjQ4cOlYoVK8qtW7dkxIgR5hftiy++KF9//bX5cN2yZYvYyd27d80Hc1hYmC2vl9jERflmypRJdu/eLS+99FKcXgfRi8syj+w5BhAYkvk7A3YwZcoUE9R99NFH0r9/f+d+rcVr3LixVK9eXRo2bCh///23ZMuWza95BeJL0qRJzY8b2BfPMRC4qMGLpdu3b5vau5IlS0q/fv0i3J82bVoZP368aSYZM2aMc3+DBg2kR48ecvbsWfN/uXLl5J133jH3WbWAlStXNrf3339f7t+/H2UeFi1aJM2bN5fHH3/cNA+//fbbcvXq1Uj7uWzcuFGee+45KV26tPzyyy/m/h9//FFatmwp5cuXl6effloGDx4s586di/J6W7dulaeeesr8rfnUc+tN/46s2bpZs2Ym2G3VqpXs2rUrQppSpUo5z1GmTBlTNtOnT5cHDx7E+HqWzZs3mzTr1q2L9HqjR4922+dpGTysrL15/iLz3XffmfyULVtW6tSpY5r69byBUr6elFn4/llxdR2rqXDSpElSt25d8xifeeYZ+fTTT83+yN4H69evNz++9H3Xu3dv53P4+++/S5MmTczz27Fjxyib1B/2OtDuGX379nU7pn379ub6v/32m3PfkSNHzL4FCxYE1HPraR6i6oN38+ZNeffddyO8B6Lrh+rJ4/XF5x2QaDgQK7/88otDi3Ho0KHRpsuSJYujWLFizu2iRYs66tat66hdu7Zj1qxZ5jwTJ0503L9/3/HMM884MmTI4Pj8888dmzZtcnzwwQeOV1991ZEiRQpH79693c7bs2dPs/+dd95xbNiwwbF06VJH6dKlHY899pjj2rVrJs2+fftMHl955RVHw4YNHStWrHCEhoY6fvrpJ8eiRYscSZIkMcdv3rzZsW7dOseHH37oqFOnTpSP5caNG87H/eabbzp2795tbqdPn3a7Xo8ePRzNmzd3rFq1yrFmzRpH+fLlHVmzZnVcvXrV7Xx79uxxnkPzMGbMGEeaNGkcr7/+ukfXi4xeT9P/8MMPEe5LmjSpo3///s5tT8vAk7KO6fMXme7duzuSJUvmGDx4sOPXX391/Pzzz45BgwY5+vbtGzDl60mZnTt3zpxz7NixcXqdBw8eOBo3buxInTq1Y/z48abM9b2kj7FBgwbmOXEtt65duzpatmxpymzZsmWOPHnyOJo2bepYu3ato1mzZqa8ly9f7siXL5+jRo0aEfLkyeugc+fOjuzZs5u8WY89ZcqU5rU3cOBA57n0NaJ5OnDgQMA8tzHJQ/jnWGl5P/30046MGTM6pk2bZp4Pfc7++9//RngPxPTxxvbzDkhMCPBiST/A9MPk66+/jjZd5cqVHcmTJ3cL8PSL688//3Tu0y8D/SDS882fP9/t+Pfee88RFBTk9uG4cuVKk1a/zFwdP37ckSpVKsfIkSPdPvAKFCjguHv3rtv12rVr5yhevHiE/N65cyfax3Ps2DFzzgkTJkS4z7qeBrT37t1z7t+xY4fZP2nSJMfDaFCk5WXlI7rrxTbA86QMPC3rmDx/kdHAIvwXZvj8BEL5elJmkX35x8V15syZY845ffp0tzQzZsww+/V/13LT81lBn9LjdH/NmjXdylPf07p/27ZtXr8OrGM1yND3e8eOHR1ly5Z1HqeBZsGCBZ3bgfDcxiQPkT3H+oNV9y1cuNDtvKNHj47wHojJtXzxeQckJjTRxpKOAlPJkyePNp3er81FVtOJKlq0qDzxxBPO7aCgIPn+++8lderUpqkifPOOBuSu5s6da/rAdOjQwW1/7ty5pUKFCvLDDz+47ddmrmTJkrldL0+ePLJv3z6ZPHmyXL582S2/saVNI5o/1+YkfWza4duVbvfq1UuqVasmJUqUMM0rn3zyiekcfvDgQYlrnpSBp2Udk+cvMvPmzZMkSZJIt27dItwX/jnxZ/nG5esmptfRMtdyeOGFF9yOff755yVlypSybNmyCOWmZWyx3oNVqlRxK09r/549e2L8Oqhdu7Z5f61cudJsr1q1yjS3a57++usv08SsnwU///yzaU4OxPeOp3kIb/ny5SZd06ZN3fbrY4/qPeDJtXzxeQckJgyyiKVcuXKZ/0+ePBltOr0/e/bsbl8s+fLli5Du1KlT5pyu6awPsfD7tP+OfmhVqlTJbP9fjay5nThxQnLmzOmWPrLrDRkyxJxHvyR69uzpnN5F/9b8xkbevHkjHXV3/vx55/bOnTtNHx39ctW+VZrHFClSmD5ouq19eeKaJ2XgaVnH5PmLzPHjxyVHjhwmMAnk8o3L101Mr6PvLS2z4OBgt2M1GNByD//e1KDRVcaMGaPdf+HCBec+T18HWbJkMQGdBngDBw40AZ72j9MBV1r+P/30kxQuXNicO7IALxDeO57kITKnT58274HwAVVk74uYXMsXn3dAYkKAF0v6y1l/JeovcdcRtK70g0l/Tbdu3dptf2Rf4jooI7I5pbQzs2vtn0qXLp35lRvVJKHhv/Aiu16GDBlk9uzZppOydjBfs2aNqS3RjtrayTlz5sziLddf5K5cf8V//vnnpmZTv5T0sbh2pI6NNGnSmP/Df8lp2YYf8OBJGXha1jF5/iKTPn16c7yW0cNqHPxZvnH5uvHmuYlqChANoLSm3JNy86Q8Y/Ke08Bt3LhxcvjwYTOCfuzYseY9qEGeBn66XwMeHTgSSM9tTPIQGX0PRPZ8aA1sVO8BX5e98uRHEmBnNNHGkv5S1zmgdKSfjtiKzLBhw8yXdfhRdZHRUWH6Bb9jxw63/frFFp5+MegkyvrlZ42Oc7098sgjHj8O/fDUmpFRo0aZL1D9Fb5hw4Yo01sfnq6jFL2hedcvcdcvKLV48eJYXS9//vzm//3797vtj24kXXRl4GlZx+T5i4xeR0fLrl69Wnwhrso3vl830V1Hy/zGjRsR3n86KlMDa/0R5isxec9pgKej7N966y1JlSqVMx+6X2v09KajeL0NhuP6ufWW1ihqYB1+FOyvv/4aq/P68vMOSAwI8HxAf5lrc4xOu6D9faxfnfqLVYO6L7/80kzLoRMhP8zLL79smp66du1qplBRe/fuldDQ0Ai/UDWtThWg/bt0WhDL9evX5auvvjKTLD+MTvGiXzTWh7/+r1OLaK2kNiFFF9jql4v2J4qNmjVrmsc5Y8YMs619h954440Ifbliej2db1C/EDQY0GZT9c8//5jajfC1BZ6UgadlHZPnLzI6NYf2P3rllVdk06ZNzv0avOh8i4FSvvH1uvH0udHmP+23eOzYMbNPm+y0DLXpVp8LX4nJe+7JJ580gZ2uIKHPg/X861Qumj/9sRFZ86y/n1tflJG+/7TcrSZW/aHl6XsguvPG9vMOSEwI8HxAaxf016l+oehNf5Hrr0n9ctEvae10HFXzbXh6rDbfaC2O9inRPkRt27Y1AWL4Jjv9Za41Q7Vq1TJzpWmfFU2vX3b6JahfMA+jx2mAql8ABQsWNNfX5mZtBilSpEi0x+rEzjooQK8Xk7m1XGntpwbBnTp1Mo9Xy0zpkm+xvZ7Oi6ZNtdoXR8ule/fuZj45b8rA07KOyfMXGQ0I9Itfz6fzl4WEhJhzDhgwwMwDFyjlG1+vG0+uo8+FlpkGMppGy1xrcPWYtWvXmjL0lZi857Q/nDbHarOkayCn89XpjwD9IRibAC8u3zuxoeWt7wENvPRxavm0a9fOzIWnZeHtQBxffN4BiUmQDqX1dybsRidH1Q83/RWr/VEic+jQIfNrVj+gomLVPFmdh7UmSDt+R7YahvYr087k+uFpfdBb9Jf9gQMHzHn0Sy8yWjOitQpWnyZP6bm11kRHE+sHrn6gR3c93a9NSuE7ROvx+nj1eP0g12YYzY9+YesXZXTXexg9j9b4WGl1ZJ4+zqxZs3pVBtGVtbfPX2S0eU+vo/l0fR0FUvlGV2ZaTjoCVs+h53Lly+u40iY8HaGqZab9GcNfM7Jys/brc2kNrLCuqTVP4ffH5HWgNWzaZK+DCKx+odZkvNqsrLWQrjXKgfDcxiQP0T3H4d8D+rmo5fTBBx/IoEGDvH68vvi8AxIDAjwAQJzTpmqdykZrW7V5GUDcIsADAPjU1KlTzdx02mdOaZ+5Fi1aOLutMCcdEPfogwcA8Cldm7tz586mSV37RuoIYu1DqoPQCO6A+EENHgAgTmg/Q+17p330wk/nAiBuEeABAADYDE20AAAANkOAFws6w4xOy8BMMwAAIJAQ4MWCtVSQ/g8AABAoCPAAAABshgAPAADAZgjwAAAAbIYADwAAwGYI8AAAAGwmmb8zAAAAvHP//n25e/cuxWcTyZMnl6RJk/rkXAR4AAAkMDr/6unTp+XSpUv+zgp8LGPGjJIjR45Yr9tMgAcAQAJjBXfZsmUz6/zGNhhAYATtun7z2bNnzXbOnDkTfoCnL9Lz589Lvnz5JDg42GfHXL58WY4dOyYFCxaUNGnSRJrm1KlTkixZMsmaNWusHgMAAPHVLGsFdyEhIRS6jaRKlcr8r0GePr+xaa716yCLe/fuycsvvyzZs2eXGjVqmAfzzTffxPqYf/75Rzp37iyFChWS0qVLyx9//BHhPJ9++qkJ/MqVKyfFihUzt19++cXnjxEAAF+y+txpzR3sJ/X/Pa+x7Vvp1wDv/fffl2XLlpmA7OTJk/LJJ59Ix44d5a+//orVMWvXrpVKlSrJ77//HuWvn927d5uATs+hkfKzzz4rTZo0kXPnzsXJYwUAwJdolrWnIB81t/s1wPv888+lS5cupqZNvfjii1KkSBGZNm1arI559dVXTZqommW1ynPixImSP39+5/aAAQPkypUr8ueff/r4UQIAAMSvJP7sIHrixAlT0+aqSpUqUQZZ3hzjqV27dpn/8+bNG6vzAACAwHL48GFTM7Z//35JLPw2yOLChQvm//AdRHVbB0/46hhP6GCMHj16mGbakiVLRpnu9u3b5mbRGj8AAALF0qVL4/V6jRo18jit9qHXed70tmfPHnnkkUec93Xo0MF8j2sXrEB2+vRpM7p1x44dUqpUKQlkfgvwdOSqunPnjtt+DaD0yffVMQ+jQ5IbN24sKVKkeOgAj5EjR8qwYcO8ug4Sjvj+gPT2wxIAEiLtFvXGG2/I7Nmz/Z0VW/NbE23u3LlNdalOU+JKt6NqJvXmmOjcvHlTGjZsKBcvXpSffvpJMmfOHG36IUOGmNo+66ZTsAAAAM/17NlT5s6dG23XKu029cwzz5gRpZkyZTL97fW7Ojo6aLJdu3amVS9z5szSqVOnCBNB//zzz1KxYkVJly6dFC9eXH744YcIFUkaZ2gQqjWMQ4cOdY5m1RpIa246naFD0z355JMB+9T7LcBLmzatVKhQQZYvX+5WE6eBVq1atdyeMKvN3NNjYhLc6ahZfcI9mQdPa/nSp0/vdgMAAJ7TAKt58+YycODAKFvW6tevL7ly5ZJDhw7Jpk2bzMwZ7du3j/Kc169fl5o1a5quXOvXr5cDBw5I9erV5bvvvnNLN2HCBJk6daqpGGrRooU8//zzbt2tNIjTCYc1tpgzZ46EhobKuHHjnMGfVcGkTbSabt26dQH71Pt1omNt7tQgS9uxdaDE2LFjzcjXbt26OdO89dZbsnHjRtm5c6fHx2jEfvz4cTlz5ozZ1hdIlixZzJx5etMnUKdE0XPqk6fprLRaS6i/FgAAQNzQKc9KlCghP/74ownmXM2aNUuuXbsmkyZNMjV4Ou/t5MmTTQWPfm9H1vdNj9Hv8c2bNzu/wzt27Bgh3ahRo+Sxxx5ztsqNGDHCBGvVqlVzS6fBnF6vf//+8sUXX8hrr70mCY1fp0nRJ1U7VOq8dX379pUMGTKYaFjXYbNowFW4cOEYHaP3tWnTRnr37m0GTXz88cdme/78+eZ+feFozaDW2mlVsd5n3dasWRPPpQAAQOKi3+u6IMHgwYPlwYMHbvdpbZ02gbpO5KyLEuiqVXpfZLZt22aOeVgFTWGXeELPry1zYWFhzn065VqZMmVMi6E2weq0a0ePHpWEyO9LlWnAFj56dxXZoIaHHaO1c3qLigaDVo0gAACIf2+//bY8+uijMnPmTLf92vQZ00mA9RhPJggOiibNypUrTcXQt99+a7p9aQXS119/bebJTYj8WoMHAAASpxw5cki/fv3MQAbXKci05U0rYbSvvGXr1q1mBg0dGBGZJ554wjS1hh9UERPad0+bZZs1a2YqgjQY1CZfV9aMHeFrHQMRAR4AAPAL7dumgyoWL17s3Ne2bVvTfNq9e3ezlOi+fftMU2mDBg2inHtOj9G+9jqKdu/evSbQ++qrr0wNnKd0TXpt6tV+/9qVa8aMGTJ9+nS3NDo6V0fgajAY6EEeAR4AAPALDZa0Bu/WrVvOfRrcrVixwgyWLFCggBl1W7RoURNwRUUHW2r/e+07p+kfffRR+e2336LtrhVe69atzTKnOpBT++jraNs+ffq4pdFavfHjx5t5cbVPYCBPkxLkiK6xOwoatX7yyScmsj148KAZnqy0nbpXr16SL18+SQx0aLW20euceEyZYh9MdAwgkGkwpLNDFCxYUFKmTOnv7CBAn1+vavB0VKoGeDoCVatWLTqCZfjw4V5nBgAAALHnVYA3ZcoUMwv1K6+84ra/Tp06bu3oAAAASCABni7RpaNcwg85TpUqlVy9etV3uQMAAED8BHjaLrxly5YIAd68efOiHMIMAACAAJ7oWJfu0DXhdIkPtXr1arPciK7xFn5IMQAAABJAgKfDiHU910GDBpkRtdr3TteK0wV5dQ4aAAAAJMClynTSQb2dOnXKBHm5cuXyaJkQINCnKgEAIKGL9Vq0OXPm9E1OAAAA4L8AT2d5js6yZcu8zQ8AAAD8MYpWlwBxvT3yyCNy//59Wb58uYSEhMQ2TwAAAD5z9+5dqV+/vuzZs8er48+cOWOO125ptq7B08EUkZk0aZJs3rw5tnkCAAA2c+TIEenatauJIYoVK+Z238CBA836r6+99lqcXPv+/ftmfdtLly55dfzNmzfN8dbSrN4KCwuT559/3iwYkT9/fgnoPniudATtu+++68tTAgAADw1ZuCNey2pk89Iep9WFEKIKsrRyKE+ePGJ3t2/fNmUQH4tC+DTA279/v5k+BQAAwFs//fSTzJ49W65cuSJPPPGE9OzZU9KmTWvu05k7ZsyYYebfVZUqVZLu3btLcHCw8/grV67Ihx9+KDt37pRChQrJyy+//NBr6nlnzpxpzutwOKRp06bSqlUrtzQXLlyQb775Rnbv3m0CUq15dB1s+uyzz5rawqRJk5oaurZt28qTTz7pPL9uq27dupnHU6pUKfnoo48CJ8AbMGBApNWOug5tmzZtfJEvAACQCK1atcoEV8OGDTPB2bZt20yzpjV9lgZJ+/btkx49ekjq1KlN9zCNP9asWWOma9PgrEGDBqY5VRdm0H5ztWrVivaaekzr1q1l3bp1MnjwYMmWLZssWLDAnKNjx47OdC1atDDX1bTazPr000/Ljh07JFmy/4VTvXv3NoGcVnZpvrXf3pw5c0zgp3nTeYQ1n7pYhAaAmTJlirNy9CrA04g4PM3k8OHDTeYBAAC8oatj6QIKVmVSs2bN5PLly+Zvbd7U2+HDhyVDhgzOmT20Nm3lypVSr149M5PHli1bTJ8/XYRBaSCogVlUdJDookWL5O+//5YSJUqYfRpUak2gq7fffls6d+5s/q5atarkzp3bBHhly5Y1++rWrRthxpExY8Y4Azwr0KxWrZqpvYtLXgV4VrUoAACAL1WuXNksfTpy5Ehp1KiRlCxZ0hnMae2eBkra519r3fSmtFl0165dJsDbuHGjada1gjsr2IouwNOgskyZMs7gzpI+fXq3bQ3qLLrAgzYLaw2hFeDt3btXpk2bJgcPHpRr167J6dOnTQunP/i0Dx4AAEBkrD5yd+7ciXTwgXW/Ns9qbZr2dZs4caIJ4nQAp9acaU2e1taFD9Z69eolRYoUMX9rQJU+XGAWfjs8DcY8aS517eenNNjUJlml/fIqVKhggk9tytWgVJtjv/jii8AO8DSznpo/f763+QEAADakfc6SJElimldr1Kjhdt/Ro0dNM6ZFmzqt5s6vvvrKDJLQ/mwFCxY0/e2071v4YMuiaVatWhVhEGh09BhtptVgTfPoDc1X6dKlTd88S/ip4+JzSVePH0WWLFk8vgEAALhKkSKFNG7c2NTKWX3qlNZwnT171tTcKR2UcOjQIef9hQsXNoGX1uS98MILcuPGDRkyZIiz5kzNnTvX9LlTLVu2NH/rKFylAx7ee++9aJ8MHbhx8eJF0yxs0XOsXbvW4ydR+/lpc63mT2kgqwNAXOliEBpAatNtwNTgTZ48OW5zAgAAbO2zzz4zo1K1xkz71um0Ixrcffnll85BB9q0qbV1WkOXMWNG2b59u7zzzjuSN29ec/93331nzqFBXYECBUzt3FNPPWX63yndN2bMGJPm448/lvPnz0uVKlUeWrs4b948c4wGnDrpss7XFxoa6vFj69Chg3kc1gpfVpOtDviw6GjbF1980QSUjz/+uOn3F1fTpAQ5rB6KiDEdXaMvRP0l8rD2fcSMNRw+sdEOxQAQnVu3bpkaLg2SUqZMmSAL68SJE2YggvZ704Ao/OPQQRMaIOn3q656EX4ZVK2V0xGvOo1J8eLFI209PHXqlBn0oFOt6IAIHWWrgZ41YCMy2hdQpzfRQEyDL6sZWFey0No8bVrWmjqLNgVroKYBoZWvv/76y+RLm2u1v+E///xjmpRd6WPTMkiXLp2Zxy8unl+vAzzN/O+//27azcNPbqwT+CUGBHhxhwAPAOwb4CHun1+vRtFqdKtDjjUy1apPnQfm5MmTpn1cM5RYAjwAAIBA5NVQkX79+kmnTp3k3LlzZvv48eOmJq9mzZry0ksv+TqPAAAAiOsAb+vWrdK3b1/nkF+tydN5aaZOnSrTp0/35pQAAADwZ4Cnfc8yZ85s/tb12rQGT+ms0ToaBgAAAP7j3Wx+Lp588kl54403zAK9uqhvXK+tBgAAxLlMF+zF4aPn1asATxfbtYwePdrMQaNDh3VJDp3jBgAAxI3kyZOb/60JdWEvN/7vebWe53gdRasTDlp0Mr8//vhD7t69G+vMAACA6CVNmtRMAGx1idJ52eJzCSzEXc2dBnf6vOrzq89zvAd4OimfLhfSpk0b5+R+BHcAAMSPHDlymP/p924/GTNmdD6/8R7gVatWzazXptOl6GLAGuw1adLEbXZnAAAQN7TGLmfOnGago7agwR6SJ08e65q7WK9koYv8/vzzzzJz5kxZuHCh2W7evLkJ9p555hlJDOy+kkViXU3Cn1iqDADg11G0SZIkkTp16piFdc+cOWMW59X117RGDwAAAAl4mhRdzHfSpEkyatQos/Bv2bJlY3yOCxcuyJ49e8z6a748JiwszASd165d8+m1AQAAbDnR8VdffWWaYnUFi08++UTq1asn//zzj1nlwlPab0CXNsuVK5c5l/Yl0JrA2B6zY8cO6dChgxQuXNgEnFu2bPHJtQEAAGwb4OmKFTqpcaFChWTt2rVy8OBBee+996R48eIxOo8es3LlSvn333/l2LFjMnnyZOnSpYts27YtVsesX7/erIu7YcMGn14bAADAtgHenDlz5PTp0yYo0pUsvJ1/R9eu7dy5sxQoUMBst23bVooWLSrTpk2L1TFdu3aVjh07SqpUqXx6bQAAgITAq2lSGjduLL7ou3fy5EmpWLGi2/7KlStH2czrzTFxeR4AAADbBHi+oIMbVEhIiNt+3T5//rzPjvHleW7fvm1urn0RAQAAbDeK1lvWyheuAZO1HdWqGN4c48vz6OTOOu+ddcubN6/H1wQAALB9gJc7d27Td0+bS11p02lUgZM3x/jyPEOGDDGTGls3HZwBAABgiwAvpqNlI5M2bVrTB2758uXOfToX3erVq6V27drOfRpE6UjXmBzjq2uHlyJFCrNihesNAADAFn3wdASt1mBpM2VsvPvuu/Lss8+agLFKlSoybtw4SZcunRkFaxk2bJhs3LhRdu7c6fExFy9elKNHjzoXYd6/f79z8V5rAV9PzgMAAJBoavCee+45s3pFbOmyZlqLtmnTJtP8qZMNr1u3zi1wzJcvnxQrVixGx+i2TnQ8cOBAeeyxx2TixIlme/HixTE6DwAAQEIU5HA4HN4EeAsXLjTBU4kSJSQ4ONjtfl3lIjHQUbQaEGptph2ba5cuXervLCQ6jRo18ncWAACJtYlWJxBu166dc/vevXu+zBMAAADiO8D79ttvY3NNIKCF7vduZRZfWL9wh1+uO7J5ab9cFwBgs2lSAAAAEEAB3oMHD8yo09KlS0uaNGmc+wcMGGBGrwIAACCBBXgff/yxfPLJJ9KzZ0+5ceOGc78GfMOHD/dl/gAAABAfAd6UKVNk7ty58sorr7jtr1OnjttUJAAAAEggAZ6uLlGyZEnzty755Tq69urVq77LHQAAAOInwCtYsKBs2bIlQoA3b948nyxjBgAAgHieJqV///7Svn17GTFihNnWNVx//PFHmTBhgkyfPj0W2QEAAIBfArwuXbqYyY0HDRpkRtRq37vs2bObkbWuEyADAAAggQR46tVXXzW3U6dOmSAvV65cbs21AAAASGABniVnzpy+yQkAAADiN8Br2LChxyddtmyZt/kBAABAfI2iLVasmPOWNWtW+f777+X48eOmBk9v+rfu0/sAAACQAGrwPvroI+ffbdq0MStWvPnmm25pdFTtrl27fJtDAAAAxH0fvJ9//lkmT54cYX+PHj2kaNGi3pwSAAAA/pzo+O7du7Jz584I+3fs2GHuAwAAQAKrwevYsaO0bNlShgwZIhUqVBCHw2FWthg5cqR06tTJ97kEAABA3AZ4o0ePlmzZssl7770nZ8+eNft0u1+/fjJgwABvTgkAAAB/Bnjnz5+XwYMHm9uFCxfMvpCQEF/lCQAAAPEd4OmqFbp6hSKwAwAAsEmAd+LECcmdO7fvcwSISOh+lr0DACBeR9H26dNHevfuLRcvXvT6wgAAAAigGrwJEybI0aNHZdGiRZI9e3YJDg52u//w4cO+yh8AAADiI8AbOnSoN4cBAAAgUAO8zp07+z4nAAAA8F8fPFdhYWH0xQMAAEjoAZ5OkTJ27FjJkSOHZM6c2UyVon/rPl3VAgAAAAmsiVZXsBg3bpy89tprUrlyZQkKCpINGzbI8OHD5fr16/Lmm2/6PqcAAACIuwDv888/lzlz5kidOnWc+2rWrCnlypUza9ES4AEAACSwJtozZ85IxYoVI+zXfdbatAAAAEhAAV7RokVl1qxZEfbPnDlTihQp4ot8AQAAID6baIcNGyatWrWSZcuWOWvyNm3aJCtWrJC5c+d6mxcAAAD4qwavefPmsnHjRkmbNq3MmzdP5s+fL+nSpTNBnt4HAACABFaDp8qXLy+hoaG+zQ0AAAD8P9ExAAAAAgsBHgAAgM0ERIB34cIF2bNnj9y6dcunx3iSRqd1+ffff+XatWsxzjcAAECCDvAGDBjg/HvLli0+ufjdu3flpZdekly5cskzzzwj2bJlky+++CLWx3iSRgM/7Ueo07o0btxYsmbNao65c+eOTx4bAABAwAd4Y8aMca4zW6FCBZ9cXJc8W7lypalBO3bsmEyePFm6dOki27Zti9UxnqTp2rWrZMiQQU6fPm3Sbd++XRYuXCifffaZTx4bAABAwAd4efPmlW+++Ub27dtntvfv3x/lzVNTp06Vzp07S4ECBcx227ZtzSTK06ZNi9UxnqQ5ceKEVKtWTVKmTGm2tSavYMGCZj8AAECimCbl/fffN7Ve169fN9uFCxeOMq1V0xedU6dOycmTJyMseVa5cmXZunWr18d4et433nhDhg4dKiVLlpT8+fObGr+LFy+axwgAAJAoArx27dpJ69atTZOm1uYdOnQoVhfWARAqJCTEbb9unz9/3utjPD1v06ZN5YcffjBNtzly5DBB4YgRI+TRRx+NMs+3b982N8uVK1c8frwAAAABOdFxsmTJJE+ePGb1Cqv501vJkyc3/7sGTNa2dZ83x3h63gYNGkimTJlMwJo6dWrT9Fy1alUzQOO1116L9PojR440y7QBAADYbpqUFi1aOP8OCwszTZsxlTt3bgkKCjJNqq60Jk1rCL09xpM0+rcutdajRw8T3FlNzlqrp8uuRWXIkCFy+fJl500HcAAAANgiwHvw4IGMHTvWNG1mzpzZNH/q37rPk/53Stex1X5yy5cvd+7T+epWr14ttWvXdu7TIEpHuXp6jCdpMmbMKEmSJJEzZ8645Um39fFEJUWKFJI+fXq3GwAAgC3WotVpSMaNG2eaMnXwgtaYbdiwQYYPH24GYbz55psenefdd9+VZ599VooXLy5VqlQx50yXLp3bQAdtEtXatp07d3p8zMPSaK3dCy+8IK+//rpptn3kkUfMIItly5bJggULvCkSAACAhB3gff755zJnzhypU6eOc1/NmjWlXLly0qlTJ48DvLp165qatgkTJsiSJUukdOnSsm7dOjM/nSVfvnxy6dKlGB3jSRqdSmXKlCkSGhpqBmboSNpVq1a51R4CAAAkREEOT9tUXQQHB5sRqeGbKLVfmq4aEX6Ag13pKFoNGvVx27G5dunSpX67duj+IEmM8uXP75frjmxe2i/XBQAEUB88nTR41qxZEfbPnDnTTBgMAACABNZEq/3iWrVqZfqsWRMKb9q0SVasWCFz5871dR4BAAAQ1zV4zZs3NwMfdMSqzomnU4voIAYN8vQ+AAAAJLAaPFW+fHkzQAEAAAA2qMEDAABA4CLAAwAAsBkCPAAAAJshwAMAALAZAjwAAACb8XmAp1OnAAAAwEYB3vXr1319SgAAAMTVPHgNGzaMSXIAAAAEeoCnS5E1atRIgoOD4y5HAAAAiL8Ar2nTptKiRQtp3bp1lGnmzJkTuxwBAAAg/vrgvfrqqzJ58uRo04SEhMQuRwAAAIi/Grynn35aChUqFG2a8+fPxy5HAAAAiN9RtPnz54/dFQEAAJAwpklhehQAAIAEGuCFhYXJlClTnNvnzp2TChUqmAmOy5QpI4cPH/Z1HgEAABCXAd6bb74pt2/fdm5/8MEHJrhbv369lCxZUgYPHhzTUwIAAMAfgyyswRNz586V1atXO7eXL18uo0aNksKFC8uQIUOkbt26zvuyZMniy7wCAADAlwGeVTOnTbRjxoyRZMmSmX53Bw4ckCVLlpjb/fv3TXBnpZ02bZqnpwcAAEB8B3hWsLZx40Zp3ry5NG7cWL788ks5evSo8769e/fK5s2bCewAAAASyjx4qn///tKyZUspUaKE7Nq1S0JDQ533LVy4UJ577jlf5xEAAABxGeB17NjRTHasNXWVKlWS6tWrO+9LlSqVdOvWLaanBAAAgD8DPFWjRg1zC693796+yBMAAAACYaJjAAAABAYCPAAAAJshwAMAALAZrwK806dP+z4nAAAA8F+AlytXLt9cHQAAAIExilYDvBMnTkju3Ll9nyMEjND9Qf7OAgAAiK8avD59+pgpUS5evOjN4QAAAAi0GrwJEyaYJcoWLVok2bNnl+DgYLf7Dx8+7Kv8AQAAID4CvKFDh3pzGAAAAAI1wOvcubPvcwIAAAD/z4N34cIF+eOPP3yTEwAAAPgvwLt06ZI0bdpUsmTJIhUrVnTu130bNmyI8fnOnj0rO3fulBs3bvj0GE/SOBwO2b9/v5w8eTLG+QYAALBNgDdw4EC5ffu2/Pvvv277u3fvLsOHD/f4PHfu3JF27dpJvnz5pFGjRpItWzaZOnVqrI/x9Lzff/+9FChQQGrWrGluDRs2ZGQwAABInAHesmXLZMqUKVKkSBG3/RUqVJC1a9d6fJ4RI0bImjVrZN++fXLo0CGZPn26dO3aVf78889YHeNJmvXr15sax0GDBpk5/TRtz549zd8AAACJLsALCwuTTJkymb+Dgv7/ZLjXrl2TJEk8P+W0adPMgI28efOa7datW0vx4sVNQBabYzxJ8/bbb5taO611tNSrV09Kly7tcf4BAABsE+CVK1fOzIEXPsAbNWqUVK1a1aNznDp1yty01s9VpUqVZOvWrV4f40kabcL99ddfpXHjxnLz5k3ZtWuXCVoBAAAS7TQpI0eOlAYNGpggSQcp6Lx4K1askB07dnjcRKsjcFVISIjbfh24Yd3nzTGepDl//rwJ8vbu3SuFChWSjBkzmsmZ69SpI998843Zjoz2O9Sb5cqVKx49VgAAgICvwatevbr89ttvpvarcOHCEhoaKgULFjQjaF1H1UYnefLk5n/XgEnpOa37vDnGkzRJkyY1/y9ZskS2bNki//zzj+mrpzV5r732WrSBbYYMGZw3qwkYAAAgwdfgqccff1xmzpzp9YXz5MljmnfDT0+i2zr61dtjPEmTNWtWSZ06tbRs2VJy5cpl9umSa9pXb+7cuVHmeciQIdKvXz+3GjyCPAAAYKuJjrUW7/PPPze3devWxejYNGnSSOXKlc1UJa61bKtXr5batWs79x05csTUrHl6jCdpdCCI/q199VydPn06QtOuqxQpUkj69OndbgAAALaowdOgq0WLFmbaEa35UmfOnJHy5cvL/Pnzo6yBC0/nzKtfv74ULVpUqlSpIuPHjzf933RKE9c0GzduNBMWx+SYh6UZNmyY1KhRwzS7VqtWTTZt2iQzZsyIVa0kAABAgq3B0ylIMmfObAYmWKNW9W+dOqVLly4en0dr0XRwxvbt203Apc2dWhPoWjOmExGXKlUqRsd4kqZs2bJmQIgGjq+//rps27bNHNOqVStvigQAACBgBDl0GGwMpUqVykwMrP3dXB0/ftwMutAm0cRA++DpYIvLly/bsrm23dhl/s5CopMvf36/XHdkc+Z/BABJ7DV4uXPnlgcPHkTYr/vCB30AAABIAAGeNsN26tRJDh486Nynf3fs2DFGTbQAAAAIkEEWn332mRw9etRMEqx98bSV11oJYv/+/eZ+i/bNA4CoDFm4w2+FQ9M0ALvyKsDTlSsAAABgowBPR9ECAADAhhMdAwAAIPAQ4AEAANgMAR4AAIDNEOABAADYjFcBni5NNmbMGOf2hAkTzATHNWvWlGPHjvkyfwAAAIiPAK9///7OFStOnDghAwcOlL59+0pISIi5DwAAAAkswFu1apXUq1fP/P3jjz9K3bp1TWCnExz/8ssvvs4jAAAA4jrAu3fvnty4ccMZ7NWuXdv8nTJlSrl79643pwQAAIA/JzrWvna65myNGjVkyZIlMnLkSLN//fr1Uq1aNV/lDQAAAPFVg/fpp59KqlSpZP78+aZZtmDBgmb/F198IW+99ZY3pwQAAIA/a/CSJk0q8+bNi7BfAz4AAAAkwBq8XLly+T4nAAAA8F8NngZ4Oj1K7ty5fZMLAH61dOlSP169gB+vDQD25FUNXp8+faR3795y8eJF3+cIAAAA8V+DpytXHD16VBYtWiTZs2eX4OBgt/sPHz4cu1wBAAAgfgO8oUOHen9FAAAABF6A17lzZ9/nBAAAAP4L8CwXLlyQgwcPSoUKFXyTGwB+Ebo/yG8lny+/3y4NALbl1SCLS5cuSdOmTSVLlixSsWJF537dt2HDBl/mDwAAAPER4A0cOFBu374t//77r9v+7t27y/Dhw705JQAAAPzZRLts2TLZuHGj5MuXz22/NtWuXbvWV3kDAABAfNXghYWFSaZMmczfQUH/v+/OtWvXJEkSr04JAAAAH/EqGitXrpyZAy98gDdq1CipWrWqr/IGAACA+GqiHTlypDRo0EB+/fVXcTgcZl68FStWyI4dO2iiBQAASIg1eNWrV5fffvtNbt68KYULF5bQ0FApWLCgGUHrOqoWAAAACWgevMcff1xmzpzp29wAAAAg1hgRAQAAkFhr8NKmTevxSXU0LYCYO3rkCMUGAIi/AE/72Vm2b98u77//vnTp0sW5TNkff/whU6dOlddffz32uQIAAEDcB3gNGzZ0/j169GiZNWuWNGnSxLmvXbt2UqtWLRkzZoy88cYb3ucIAAAA8d8H76+//jLBXHi6b9u2bbHLEQAAAOI/wMuYMaMsWLAgwv758+c7V7gAAABAApomZdiwYab/3dKlS00fPJ3seMuWLWZ7+vTpMT7f2bNnze2RRx6R1KlT++wYT8+rg0L27Nkj2bNnl7x588Y4/wAAAAm+Bq9jx45mouPkyZObvnizZ882f+u+F1980ePz3Llzx/Tdy5cvnzRq1EiyZctmBmrE9piYnrd9+/ZmguaPP/7Y47wDAADYqgZPR9S2adNGqlSpEquLjxgxQtasWSP79u0zNWdz5syR559/Xp544gmz3q23x8TkvBMnTpSrV69K6dKlY/VYAAAAEnQNntZ4PXjwINYXnzZtmnTu3NnZLNq6dWspXrx4tM28nhzj6Xl1uhddV/frr7+WoKCgWD8eAACABBvgFSlSRHbs2BGrC586dcrcrHn0LJUqVZKtW7d6fYyn571+/bqphRw/frzkzp07Vo8FAAAgwTfR9urVS9q2bSvvvfeelChRQoKDg93uL1CgwEPPceHCBfN/SEiI2/4sWbI47/PmGE/P26NHD9PE3KJFC/HU7du3zc1y5coVj48FAAAI6ACvW7du5v9mzZpFer+Oqn0YHZShXAMmdfPmTed93hzjSZrly5fLokWLZPHixWb0r3W/jrjV7fLly0d6fW3O1RHEAAAAtgvwdPBCbOXJk8f0ezt58qTbft3W0a/eHuNJGp0W5dFHH5UBAwY47z927JhcunRJ9u7dK5s2bZKkSZNGuP6QIUOkX79+bjV4TKsCAABs0QdPg6Pobp5IkyaNVK5cWb7//nvnPq1FW716tdSuXdu578iRI7Jr1y6Pj/EkTatWrUxNnetN+xXqSFv9O7LgTqVIkULSp0/vdgMAALBFDZ5FJwfevXu3aZLVvnjFihWL0fHDhw+X+vXrS9GiRU1/OB3woKtkdO3a1S3Nxo0bZefOnTE65mFpAAAA7MqrGrywsDBp0qSJmXqkZcuWzmlIdJ82c3pKa9RWrFhhpivRvm3a3Llu3Tq3mjEdsFGqVKkYHeNJmvBKliwZZdMwAABAQhLk8GRERDgvvfSSmSbls88+c05H8scff0j37t3NhME6r1xioH3wMmTIIJcvX7Zlc227scv8nQUkAvny5/fbtUc2Z4JzAPbkVRPtkiVLTLOpNoFatN+bLltWtWpVX+YPAAAA8dFEe+vWLcmcOXOE/bpPBzQAAAAggQV4WkunU4ZooGfRvwcPHizVqlXzZf4AAAAQH02048aNk3r16pk558qUKWP2/f3332ZFCx3cAAAAgAQW4OlACp3seMaMGWaOOp1YWOeWa9++vZmHDgAAAAlwHjwN5KwlywAAAJDA++ABAAAgcBHgAQAA2AwBHgAAQGIN8MqXL+/8+6OPPoqr/AAAACC+AjydBuXu3bvm79deey221wUAAIC/R9GWKlXKrDVbsWJFsz1t2rQo03bu3Nk3uQMAAEDcBXjTp083q1esWbPGbI8YMSLKtAR4AAAACSDAK1u2rPz444/mb53Y+PDhw3GZLwAAAMTnKNpz5855ez0AAAAE4koWWbJkMf/v2bNHdu/eLQ6HQ0qUKCHFihXzdf4AAAAQHwFeWFiYdOjQQZYsWSJJkyY1Tbb37t2Txo0by9dffy0ZM2b05rQAAADwVxNtnz595NixY7Jhwwa5ffu23Lp1y/yt+3r37u2LfAEAACA+a/C05m7jxo1StGhR577KlSvLrFmzpGrVqt7mBQAAAP6qwdMau8yZM0fYr/tu3rzpi3wBAAAgPgM8raXTOfE00LPo34MHD5Zq1ap5mxcAAAD4q4l23LhxUq9ePcmTJ4+UKVPGuZRZcHCwrFixwhf5AgAAQHwGeKVLl5Z9+/bJjBkzZNeuXWYUbatWraR9+/aSJk0ab/MCAAAAfwV4SgO5bt26+SIPAAAA8HcfPAAAAAQuAjwAAACbIcADAACwGQI8AAAAm/E6wLt69aosWLBAPvzwQ+e+3bt3i8Ph8FXeAAAAEF8B3p49e6REiRLSq1cvGThwoHP/Bx98ILNnz/bmlAAAAPBngNe3b18z593x48fd9mvA99FHH/kqbwAAAIivefA2bNhgaup0gmNXxYoVMxMfAwAAIIEFeNrP7vbt2+Zv1yDv4MGDkiFDBt/lDgDi0JCFOxJl+Y5sXtrfWQAQiE20devWldGjR7sFeGfPnpUePXrIf/7zH9/mEAAAAHEf4I0ZM0YWL14shQsXlgcPHkj16tXlkUcekRMnTpiBFgAAAEhgTbR58+aVv//+W2bNmiVbtmwxQV7btm3NwIu0adP6PpcAAACI2wCvTZs2EhoaKl26dDE3AAAAJPAm2qVLl8rNmzd9lomTJ0/KX3/9JdeuXfPpMQ9LozWP+/btk0OHDsn9+/e9yjsAAECg8SrAe+qpp2TZsmWxvriOxG3durUUKlRIWrZsKdmzZ5dJkybF+hhP0owaNUpy584tzz77rNSoUcP0Ifzhhx9i/ZgAAAASZBOtznen/e2+//57s6JFcHCw2/19+vTx6DzDhw+XdevWyf79+02wpUufaUBWvnx5qVChgtfHPCyN1taFhYWZfoRZs2Y107689dZbJs2BAwdMQAgAAJBQBTm8WDy2VKlS0d6/c+dOj86TM2dOeeWVV2TYsGFu59ZRuVHV5HlyjDfnPX36tDlu+fLlHk/1cuXKFTPv3+XLlyV9+vRiN+3Gxr6WFghk+fLnl8SIefAA+/OqBs/TAC462j9Og6rwNXWVKlWSbdu2eX2MN+dVW7duNf8XLFjQ68cEAACQYAM8X7h48aL5P3PmzG77Q0JC5MKFC14f4815db9O0tyiRQvT/BwV7dtnreBh1eABAADYKsDT1SuOHj0q9+7dc9tfuXLlhx6bPHly879rwKR0dK51nzfHxPS8GqQ1aNBAsmTJItOnT482zyNHjnRr9gUAALBNgKcrVujExr/++muk93vSrU8nS06SJIlpUnWl2/mj6BfjyTExOe/Vq1elfv36ZtDFTz/99NB+dEOGDJF+/fq5BYd6PQAAgAQf4OkoWR2QcOzYMRPgnDt3TjZv3iy9evWSvn37enSO1KlTm5o+nVOvXbt2Zt+NGzdMoDV06FBnuoMHD5p57MqUKePRMZ6e1wru7ty5I6tWrZKMGTM+NM8pUqQwt/ikj8N//rfOMAAASASjaLNly2YGJeTJk0eCgoJME23SpEll06ZN8tJLL8mePXs8Os+aNWukbt26MnjwYKlSpYp88skn8u+//8r27dudtWmdO3eWjRs3Ogd2eHLMw9LcvXtXateubfZ9/fXXbsGdzp2nU6cEyihafwZ4ofsJ8GBvjKIFYFde1eBpjZ0Gd9ZgBu2LpzV6pUuXNqtCeKpWrVqyevVqmThxomnu1eO/+OILt2BJAy7XVTM8OeZhabRGT2vudMTsO++845an119/XRo3buxNsQAAACTcGjyttbMO05qwatWqycCBA2Xq1Kny2WefmeW/EgNq8ICEjRo8AHblVQ1ekyZNnH+PGDHCLPelq0ekTJlSZsyY4cv8AQAAID4CvMWLFzv/1j5uOlWK9rsrUKCAmW4EAAAACXyi47Rp05p1XgEAAJCAA7zvv/9efv/9d+fKEa4mT54c23wBAAAgPgO8N998U0aPHi3Vq1eXTJkyeXttAAAABEqAN23aNFmxYoWZjgQAAACBJYk3B+nExpUqVfJ9bgAAAOCfAE+bZnV5LwAAACTgJtpx48a5rS7Rtm1b6dChgzz66KNm4uPwa9UCAAAgwAM87XfnSpf5Wrt2rbmFR4AHAACQAAK8nTt3xm1OAAAA4L8+eAAAALDJNCk3btyQBQsWSPv27Z375s2bJwMGDJCwsDBp2rSpTJkyRVKlShUXeQUAnzp65IjfSjRf/vx+uzYA+4tRDZ6uULFjxw7n9pEjR+TFF1+U0qVLy8CBA2XNmjXywQcfxEU+AQAAEBc1eN988418++23zu2FCxdK9uzZZfHixZIsWTKpWrWq/Pe//5Vhw4bF5LQAAADwVw3e3r17zbQoFl2Ltl69eia4U5UrV5Zjx475Mn8AAACIywAvJCREDhw4YP5+8OCBrFu3TqpUqeK8//Lly5I+ffqY5gEAAAD+CvDq1q0rr776qqxcuVIGDRpkBlbUr1/fef/WrVulQoUKvswfAAAA4rIP3ogRI6RJkybOZtmxY8dKjhw5nPdPnDhR+vbtG9M8AAAAwF8BXs6cOWXz5s1y4sQJ0xSbLl26CKNs8zP0HwAAIOEEeJbcuXNHup/gDgAAwP9YyQIAAMBmCPAAAABshgAPAADAZgjwAAAAbIYADwAAwGYI8AAAAGzGq2lSAACxc/TIET8WYWlJbJYuXeq3azdq1Mhv10biRQ0eAACAzRDgAQAA2AwBHgAAgM0Q4AEAANgMgywAIJFhwAFgf9TgAQAA2AwBHgAAgM0Q4AEAANgMAR4AAIDNBESAd+zYMdmyZYtcuXLFp8f4Kg0AAEBC4tcA79atW/Lcc89J0aJFpX379pIjRw6ZMGFCrI/xVRoAAICEyK/TpAwbNkw2b94sBw4ckJw5c8rixYulWbNmUrFiRalUqZLXx/gqDQAAQELk1xq8L7/8Ujp37mwCLNW0aVMpVaqU2R+bY3yVBgAAICHyWw3eyZMn5cyZM1KuXDm3/VqDtm3bNq+P8VWayNy+fdvcLJcvXzb/x2X/vRs3boi/3L0V5LdrA4g7N244/Fa8/urv7M/PUvp4w9fSpUsnQUFBgRngXbx40fwfEhLitl+3rfu8OcZXaSIzcuRI07QbXt68eaM8BgACzTx/ZwBArGgFU/r06QMzwEuePLlzsIOrmzdvSnBwsNfH+CpNZIYMGSL9+vVzbj948MAEhBoYPiyS9vZXnwaPOtL3YU8kKDNeZ/GH9yblxmstcCWG92e6dOkemsZvAZ4WfpIkSeTEiRNu+3U7X758Xh/jqzSRSZEihbm5ypgxo8Q1fYHa9UUaVygzyozXWeDi/UmZ8Tqz8SCL1KlTS9WqVWXJkiXOfdevX5effvpJnnnmGee+/fv3O/vFeXKMr9IAAAAkVEEOh8NvvW3Xrl1rAqr+/ftLlSpVzDx0hw4dkr/++kvSpk1r0uhI140bN8rOnTs9PsZXaQKhmjlDhgwetbWDMuN1Fn94b1JuvNYCF+/PAJgmpWbNmrJmzRo5cuSIjB8/XkqWLCnr1q1zC7AKFy4sTzzxRIyO8VUaf9Pm4LfffjtCszAoM15n/sV7k3LjtRa4eH8GQA0eAAAAbLoWLQAAAHyHAA8AAMBmCPAAAABsxm/z4OHhDhw4IJcuXZLixYubqV3w/yek/vPPPyMUR+nSpc2o4/Bp//nnH7P/0UcfTXRFeOrUKfM6iqxsLDoZqC7dV6RIkShHa3uSxi7OnTsn//77r5QoUUIyZ87sdp8OytKyCN+hu0KFChHOo0si6tya+rrLlCmT2JV24z548KBZxrFQoUJRDgrT18/Ro0elYMGCkiVLFq/T2IXO2qDLpz3yyCOSKlUqt/tOnz5tpggL78knn4yw7/z58+ZcOodr9uzZxe7vTX3/6WON6vWhiw/oZ16ePHmca817k8YWdJAFAsvFixcdNWvWdKRLl85RuHBhR/r06R2hoaH+zlbA2L17tw4MclSoUMFRrVo15+2PP/5wSzdv3jxHhgwZnGX45JNPOi5cuOBIDLQsnnvuOUfWrFlNWa1ZsyZCmps3bzqaN2/uSJUqlaNYsWLm/08++STGaexix44djueff96RPXt2U2aLFi2KkGbQoEHmNeX6umvWrJlbmrt37zrat2/vSJkypaNEiRKOFClSON577z2HHU2ZMsWRP39+R6FChRxFixZ1ZMqUyTF9+nS3NA8ePHB0797dlINVHlqOMU1jF99++63j0UcfdRQoUMA8Vv1sGj9+vFuaSZMmmdeP6+tMb/fu3XNLp2XkWmZahlqWdrN9+3ZHrVq1HLly5XKULVvWfA61aNHCcf36dbd0w4YNcyuPDh06RCgzT9LYBQFeAHrhhRccpUqVcly+fNlsT5gwwREcHOw4dOiQv7MWUAHesWPHokxz5MgR8+a1gpErV66YMtUv8MTgyy+/dMydO9dx8ODBKAO8wYMHO/LkyeM4efKk2daARtNu3LgxRmnsQn9E6Zfv+fPnow3wateuHe15PvjgA0eWLFlM2avVq1c7kiRJ4lixYoXDbt59912396G+7vSxbt261blv8uTJ5sfqzp07zfamTZvM59mcOXNilMYu9PVx4MAB5/bChQvN6+2XX35xC/A0aH7Y61U/4zZv3uz8gZImTRoTdNvN4sWLzWvCoq85/fH69ttvO/ctW7bMkSxZMsevv/5qtvfu3evImDGj4+OPP45RGjshwAswV69eNR9s06ZNc+7TXxf6hTF8+HC/5i3QAjx9k+oXiQZv4b3//vuOkJAQt19mWqbJkyd3Bs6JgX4QRhXgaU3VO++847ZPg+CuXbvGKI0d34PRBXjVq1c3rzv9cojsl3+RIkUcffr0cduntcetW7d2JAZau/Lpp586tytWrGhqSVw1bNjQUa9evRilsbMcOXK41fJqgFewYEHH33//7di1a5fj9u3bEY6pW7euo2nTphEqBypVquRIDKpWrero3Lmzc1tbGurUqeOWplu3bo6SJUvGKI2dMMgiwGh/sTt37ki5cuWc+5ImTWome7aWbMP/PP/88/LCCy9ISEiIdOvWzfQBsmhZPf7446bsLBUrVpS7d+/Krl27En0Rav8w7e/k+jqzysh6nXmSJjFav369tG/fXmrUqCG5c+eW+fPnuy15uHfv3kRbZrrikPZ7tfq7aiXC9u3boy0PT9LYmfbr1L5l4fsIHz58WFq2bCkNGjQw/c0+/fRTt/u1bCIrM12NyY7T2z548MAsRqBLir7++uum32Hv3r0fWh67d+92fjd4ksZOGGQRYLTzp9KgxZVua4d5iFltZNmyZfLss8+a4tAvh1q1aplBBKNGjXKWY2Rl6FrGiVl0rzPrPk/SJDbVq1eXnj17msBOv0Tff/99adu2rRl8UqZMGQkLC0u0ZaaBXceOHc3Sj3Xq1HEGvPrFGV15eJLGrvQHZ4cOHcyAnqZNmzr368A6/bFfrFgxsz1jxgx58cUXzYCM//znP9F+xmlZ6uCNNGnSiJ3o4xo8eLBcu3ZN9u3bJ//973/N+84SVXk8ePDADFbUASiepLETavACTPLkyc3/t27divDhGRwc7KdcBRYd+WQFd+qxxx6T7t27S2hoqFs5RlaGinL07HXGazEifd1pcKeCgoJMTYJ+KSxYsCBRl5m2OrRo0cJ8SWpZJEnyv68WXmdRu3//vqkJ1tGy3333ndvrQ5fStII7pemqVauWqD/jdKSx1uBpDeWePXtk3rx5Zi35mJRH8kRWZgR4ASZ//vzmf51ewZVu69BwRE6/ZF3LTMsxsjJUlKNI3rx5zZdwdK8zT9IkdhrkZcuWzVlGWbNmNVMaJaYys4I7nVpG1/d2nXZCp0wJ/94MXx6epLFjcKc1chqwaJnptDC++ozLkSOHM7C2K/1satOmjfzwww8PLY906dI5pynyJI2dEOAFGO2HoW/2JUuWOPcdP37czPv2zDPP+DVvgUKbdMJbtWqVlCpVyrmtZaX9LbTsLPorWb8wXKv1EysNQqpWrer2OtNy1f4t1uvMkzSJ/bWnry/tv2O99jQgfvrpp93KTAMg/SKyY5lpE6P2E9PmxF9++cXUroenj3vp0qVuwY12sXAtD0/S2IU2B2qz7Nq1a01wF9n8nOFfZ9osuWHDhgifcd9//705n0Vfd3Yss8g+87Xm07W5VR+3vs/u3bvn9plf5/+6C3iaxlb8PcoDkQ9/16HcI0eONEPoy5cvb+Z8s+tcPTGlIxl19JROA7JkyRIzckxHHv/www/ONFpWlStXdpQrV86xYMECx6hRo0yZzpw505EYnDlzxvHbb785p2DQ6WJ0W6ePsei0DDqqWKdC+e6778zoMp2aQUeRxiSNXegciVpGK1euNGWmI7F125ruRJUuXdpMc6GvNZ0SROd+0xF4riO5//zzTzOHWa9evczrs3Hjxmb+rnPnzjnspmXLlmZqjtmzZ5uysm6ur7M9e/aYKVBefvllUx5t2rQxI9yPHj0aozR20aVLF/N5pa8f1zJznTpF32dvvfWWmdZDy1ZHxupryHVKGi0bLSOd+knLrFOnTqYMtSztpkGDBo4333zTsXTpUvNYX3nlFfN5rtuW06dPm9HIOlJW07z66qtmRPdff/0VozR2EqT/+DvIRET6K2P69OmmT0ulSpVk4MCBUa5EkNjoS3b27NnmF/+VK1ekaNGipsOtzqLvSu/78MMPzS9fLTvtAN6wYUNJLK+f9957L8L+Tp06mZvl999/N6PzdLSsrnahnZi1iceVJ2ns4LfffpMhQ4ZE2N+qVSvp1auXc9WACRMmyB9//GGadXRAwauvvhph9QatcR8/frxp/tG+VIMGDbJlc2PdunVNh/7wXnrpJenSpYtzW0euf/zxx2bEqL5P9fMsfM2VJ2nsoHnz5nL27NkI+5977jnp27evs8Zu0qRJ5jWpza06i4IO7gm/iozWYo0ePdqsyqDNj9onrWTJkmI3+hqbMmWKqfXU2l1thenatWuE1hh97ehAO+0ukCdPHunTp4+ULVs2xmnsggAPAADAZuiDBwAAYDMEeAAAADZDgAcAAGAzBHgAAAA2Q4AHAABgMwR4AAAANkOABwAAYDMEeAASLZ0gdvny5RKodDLWFStWxNv1Nm/ebJYdC0/XTI2MLvyuNwCBhwAPQIJy+PBhs06prmSis9LHhq5h3K9fPwlU+hh1FYz4oKt06EovkS1U/9FHH0V6jK7pqcfoygsAAgsBHoAEQQOQRo0ameXSdOk0XcpJ/27WrJmEhYX5O3sJ3gcffGAWYy9cuLDbsoDbtm2TU6dOmSXrXBdpV+XLl5cSJUqY5dsABBYCPAABTwOL//znP3Ly5EnTrKpr7WrTqv599OhRU4v04MEDk1bX+QwNDTXbumbs/PnzTXCodB1LXc9Smz11bd2o6BqfixcvNusY37592+2+nTt3yurVq83+NWvWyLx58yIcf/r0aZkzZ06EgEjzb+3X2kfNp960pk4fy8Ns3bo1QnPpvn37Im1mju4xhHfr1i2z9nWHDh2c+y5fvmzW2m3atKkpY63p1DU7d+/e7XZs+/btTbDNsuZAYEnm7wwAwMNoELVlyxbZtGmTZMuWzbk/a9as8tlnn0nlypVl0aJFZsF27UP2/PPPy7fffisnTpwwC5JrTV+KFClMDZUGVhqobN++XYoVK+Z2HQ28OnfubAKmSpUqyfHjx+X69evy3XffSfHixU0aDRhnzJghadKkMdfPkSOHtGzZ0u08eq0XX3xRMmTIIPXr13fuHz9+vGkWbt26tRw7dswEYOrq1atmYfmXXnop2tqwL774wuTpySefdO7TYHXixInSoEEDjx9DeBo0ahrX82q5XrlyRfbu3Wvyq3nV4O7mzZtux9auXds8Vi3Pxx9//CHPJID4QoAHIOCtXLlS8uTJIxUrVoxwnwYxep8GOhrgWTR40756lrfeeksuXLhgAsBMmTKZwEcDkixZsjjTfPjhhyZQ0dq0dOnSmX19+/aVLl26uNWcHTx40NS6ac1hZPT8WuM4c+ZMZ4CnNVyzZ8+WPn36mG0NplwDKq0lK1OmjLRo0UJq1qzpdVl5+hjC1wwWLFhQUqVK5dx38eJFyZ49uwlWLZEFiLly5TKPV2tLCfCAwEGAByDgaZNnvnz5orxf79M0rnr16uW2PXfuXFOzpcGI0qCwXbt2bqNUv/zyS6latarZpwGZ3kJCQkwzpzZjpkyZ0qTTYCiq4M7ywgsvSMeOHeXGjRuSOnVq+fXXX00TrdYuWvScf/75p+njpjVvGizpSNbYBHiePgZX2oRtlYtr0+vkyZOlVatWJqDVEb1FixaN9Jp6rNUMDiAwEOABCHhp06aVQ4cORXm/Bhf58+d325czZ063ba0hK1CggNs+DdTCj9DVGj1thnWlTbAaqFnBUfhzR0YDwKRJk5qmUQ3qtDbv6aefdh6rgxaaN29urleoUCETBOpgEe1DGBuePobw5atNtK60NlGbZ7WpW4PD6tWrm/N+9dVXEWpS9VirthBAYCDAAxDwNKDQQEkHRmizoSvdp8Ff9+7d3fYHBQW5bWstVvjRtuG306dPbwYVDBw4MNr8hD93ZDSQ0iZjDez0fw24xo0b57z/tddeM7V8H3/8sduo1OgGKyRJksQ5mMSitXLePAZX2k9R+ybqufUaFg1GNZ8ajC5YsEB69OhhaiV37drlTKNTpGiAHVXtHgD/YBQtgICnozt1UMMbb7wR4T7dp4MZtKN/dLS/mzWoQWkw47qttL+cDmS4c+eO234drOENqwn466+/NiNZtcbOok3KrkGRjob9+++/oz1f7ty5zehYVzqSN7aP4amnnjK1cDt27HDu0z6KrrQ2Upt+wzfFbty4UYKDg6VatWrR5h1A/KIGD0DA09GqCxcuNHPeaeChNWJa06X7tHO/1u6F70MW3ttvv21qyLS5tFatWia40z5xWrNnGTVqlGmK1IEbOqI1WbJkZmCCjhzVa8SUBk5a46hTjDRp0sQ0hVq0lm3YsGEm8Lt7966MHTvWNNNGR/vDvfPOO/Lyyy+bYOunn34yjz9z5syxegw6EljnGNRBII899pjZp0GiTgejAaOO+NV58jSP3bp1cztWp33RMn1Y3gHEryAHkxcBSCC09kj7hFk1XRqM6GAA1wBHR8m+++67MmvWLLfmRqXTfEyZMsUEVRoA6aAGrWFzbSbVKUu++eYbM/hB+5VpzZ8GlNa5tKlV58LTQMsTU6dONYFSz5493Wq5dFCFzj2nU79o7aReQ2vD8ubNa2r+lI4C1uBMgyuLPnbtB6ePQfOmj11HGcfkMURGp6HRfoNaQ2gFopofzYOeS4PLevXqmalmLOfOnTOjldevX08TLRBgCPAAAM7aP63l1LntXGltY/jmbLVkyRIzeEX75gEILAR4AIBoae1g//79KSUgASHAAwAAsBlG0QIAANgMAR4AAIDNEOABAADYDAEeAACAzRDgAQAA2AwBHgAAgM0Q4AEAANgMAR4AAIDNEOABAADYDAEeAACA2Mv/A/doMBZfbM5MAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 700x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bins = np.arange(0, 340, 20)\n",
    "fig, ax = plt.subplots(figsize=(7, 4))\n",
    "for label, color, name in [(0, 'tab:gray', 'No chat'), (1, 'tab:blue', 'Used chat')]:\n",
    "    values = orders.loc[orders['used_chat'] == label, 'order_value']\n",
    "    ax.hist(values, bins=bins, density=True, alpha=0.6, color=color, label=name)\n",
    "ax.set_xlabel('Order value ($)')\n",
    "ax.set_ylabel('Share of orders per $1 of order value')\n",
    "ax.set_title('Orders that used chat sit somewhat higher', loc='left')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12",
   "metadata": {},
   "source": [
    "The blue histogram sits to the right of the gray one, but the two overlap a great deal. Many orders without chat are larger than many orders with chat. The comparison is about averages.\n",
    "\n",
    "Our test statistic is the observed difference in means, **chat minus no chat**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.793455Z",
     "iopub.status.busy": "2026-10-24T04:20:03.793377Z",
     "iopub.status.idle": "2026-10-24T04:20:03.795463Z",
     "shell.execute_reply": "2026-10-24T04:20:03.794942Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed difference (chat minus no chat): $15.92\n"
     ]
    }
   ],
   "source": [
    "observed_diff = group_means[1] - group_means[0]\n",
    "print('Observed difference (chat minus no chat): $' + str(round(observed_diff, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c14",
   "metadata": {},
   "source": [
    "Customers who used chat spent about \\$15.92 more per order. The question is whether a gap this large could appear just from the luck of which orders happened to come from chat users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c15",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.796745Z",
     "iopub.status.busy": "2026-10-24T04:20:03.796609Z",
     "iopub.status.idle": "2026-10-24T04:20:03.800118Z",
     "shell.execute_reply": "2026-10-24T04:20:03.799723Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Compare Returning and New Customers\n",
    "# The store also wants to know how returning customers differ from new ones.\n",
    "#   1. Use groupby on 'returning_customer' to find the mean order_value of\n",
    "#      new customers (0) and returning customers (1).\n",
    "#   2. Compute the difference: returning minus new.\n",
    "#   3. Also find the share of orders in each group that used chat\n",
    "#      (the mean of 'used_chat').\n",
    "#   4. Print the results rounded to 2 decimals.\n",
    "# Print exactly:\n",
    "# Mean order value, new: 59.7\n",
    "# Mean order value, returning: 91.05\n",
    "# Difference (returning minus new): 31.34\n",
    "# Chat share, new: 0.09\n",
    "# Chat share, returning: 0.37\n",
    "import pandas as pd\n",
    "path_data = '../../data/'\n",
    "orders = pd.read_csv(path_data + 'chat_orders.csv')\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.801147Z",
     "iopub.status.busy": "2026-10-24T04:20:03.801077Z",
     "iopub.status.idle": "2026-10-24T04:20:03.805750Z",
     "shell.execute_reply": "2026-10-24T04:20:03.805341Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean order value, new: 59.7\n",
      "Mean order value, returning: 91.05\n",
      "Difference (returning minus new): 31.34\n",
      "Chat share, new: 0.09\n",
      "Chat share, returning: 0.37\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import pandas as pd\n",
    "path_data = '../../data/'\n",
    "orders = pd.read_csv(path_data + 'chat_orders.csv')\n",
    "\n",
    "by_returning = orders.groupby('returning_customer')[['order_value', 'used_chat']].mean()\n",
    "\n",
    "print('Mean order value, new:', round(by_returning.loc[0, 'order_value'], 2))\n",
    "print('Mean order value, returning:', round(by_returning.loc[1, 'order_value'], 2))\n",
    "print('Difference (returning minus new):',\n",
    "      round(by_returning.loc[1, 'order_value'] - by_returning.loc[0, 'order_value'], 2))\n",
    "print('Chat share, new:', round(by_returning.loc[0, 'used_chat'], 2))\n",
    "print('Chat share, returning:', round(by_returning.loc[1, 'used_chat'], 2))\n",
    "# Keep this result in mind: returning customers chat more AND spend more."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17",
   "metadata": {},
   "source": [
    "## A Permutation Test Without Randomization\n",
    "\n",
    "```{index} permutation test; observational data\n",
    "```\n",
    "\n",
    "The test itself is the one from {doc}`1001-ab-testing`. Only the interpretation changes.\n",
    "\n",
    "**The null hypothesis.** The chat label is unrelated to order value. If that is true, the 62 orders marked \"chat\" are just 62 orders that could have been any of the 300, and shuffling the labels among the orders should produce differences like the one we saw.\n",
    "\n",
    "**The alternative hypothesis.** Orders that used chat have a higher average order value. Marketing's hunch points one way, so we use a one-sided test and choose the direction before looking at the shuffles.\n",
    "\n",
    "**The test statistic.** The mean order value of chat orders minus the mean of no-chat orders.\n",
    "\n",
    "The method is the same as before: keep every order value, shuffle which orders carry the chat label, and recompute the difference. One shuffle looks like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c18",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.806959Z",
     "iopub.status.busy": "2026-10-24T04:20:03.806885Z",
     "iopub.status.idle": "2026-10-24T04:20:03.809274Z",
     "shell.execute_reply": "2026-10-24T04:20:03.808960Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chat orders after shuffling: 62 (same as before: 62 )\n",
      "Difference after one shuffle: $-14.85\n"
     ]
    }
   ],
   "source": [
    "rng = np.random.default_rng(1002)\n",
    "\n",
    "values = orders['order_value'].to_numpy()\n",
    "chat_labels = orders['used_chat'].to_numpy()\n",
    "\n",
    "def diff_in_means(values, labels):\n",
    "    return values[labels == 1].mean() - values[labels == 0].mean()\n",
    "\n",
    "shuffled_labels = rng.permutation(chat_labels)\n",
    "print('Chat orders after shuffling:', shuffled_labels.sum(), '(same as before:', chat_labels.sum(), ')')\n",
    "print('Difference after one shuffle: $' + str(round(diff_in_means(values, shuffled_labels), 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c19",
   "metadata": {},
   "source": [
    "Shuffling keeps the group sizes and the order values fixed; only the labels move. Repeat it 5,000 times to build the distribution of differences we would see if the label carried no information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c20",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.810358Z",
     "iopub.status.busy": "2026-10-24T04:20:03.810285Z",
     "iopub.status.idle": "2026-10-24T04:20:03.906105Z",
     "shell.execute_reply": "2026-10-24T04:20:03.905614Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAGHCAYAAADvIhWAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAUqJJREFUeJzt3Qd0VGX6x/EnlNBCEaQpVRARRASUIigoICgqRRQFxUpZGy5FwIYd17VhBRVEBVexseoiKIJIEREERQRFRXqVKp0w//N7939nZyYzyaSRm+T7OWdOMndueW9/7ttuQiAQCBgAAAB8qUBOJwAAAACxEawBAAD4GMEaAACAjxGsAQAA+BjBGgAAgI8RrAEAAPgYwRoAAICPEawBAAD4GMEaAPjE3/72N0tOTs7pZADwGYK1CC1atLCRI0cGv+/evdsGDBhgTZs2tbp169rKlSvTvZEPHTrkpv3ss88yv8eQbe68807r06dPjm7h7777zh0rmzdvztF0+FW07eOH/ZZV1q5da6m9VGbJkiXuelS/fn0X2EUzceJEt40iPx9++GFcaZg5c6Z1797dGjdubJ06dbL//Oc/KcbZtWuXPfHEE3bhhRfaWWedZT179rRvvvkmHWuav8VzzMZ7Lbj11lvdMYG8jWAtgoKx0JPj0UcftWnTptlzzz1nkydPturVq6d7Ix89etR+/vlnF/hlBZ3AH3/8cZbMC/+zYcMGd7NMj0GDBsW8aWbEvn373LFy+PDhY7rc3CLa9snIfvObTz75xLp162azZ8+2tm3b2rPPPuse8kK9/vrrdt1119lJJ51khQsXjrnOO3bscNcwXa9CP+eff36a6VCgd8EFF9iZZ55pL7/8sl100UV2+eWX27/+9a+w8fTwOn78eLvxxhvtxRdftKpVq7oH3UmTJmVyS+QP8Ryz8V4L1q1bZ+vXr8/iFMJvCuV0Avxu8eLFdu6551qzZs3ML3QC68kWOU8Xyb/++ivfLNeP/vnPf+bqosN33nnHBWFPPvmkbdu2ze666y6bMmWKjRgxIiyXv0ePHnbttde6/999991U51mwYEH3UJdeQ4cOdblkw4YNc98VtK1Zs8Y9HFxxxRVuvtK6dWt75plnrHjx4u67ctd++OEH93Cr8QBkLXLWUnmq0cVOT7q6MOr/rl27ppp7NmbMGOvQoYM1atTIevfubb/88kvYOAqwhgwZYuecc477RLvJtGnTxi3r1FNPdUHiwIEDbevWrcHfTzvttGA2ule8EeupavDgwda/f//g999++82NrzR4li5d6oZ5T3laj9GjR1v79u3t9NNPd0/73377bYoi3bfffttuv/12dzG/8sor3W9btmxx81ZgqyKUO+64w7Zv326peeONN9y2iKSndD3le5STqOUqeO7Vq5dbropoZsyYETadN97ChQutS5cubh1ee+0199vXX3/t0nrGGWe4bav9lZrHH388uI2bNGnibpaah0f7QDfVL7/8MjjeK6+8Evw9nuUpt0S/aZzrr7/eNm7cmGqa4lmuiru0bbTuyiXRNo4nR/mmm25yOSbnnXeePfzww3bgwIGwohbtbx0bym3RMa5jK/TYjHe9vWIb7ZeLL77Y7Uvl0Ci3ISPb5+mnn7YHHnggQ/N/9dVXw+avHK6MFkOndbzEonW86qqr7Oabb7ZSpUq5nLVRo0bZgw8+GDZe0aJFLTvp+qRribZXKK2LtrvOKY+OAy9Q86jU4c8//4xrWfFs93i2Z3qOy1C6btSrVy/sgUfXC6XnyJEjwWE6f0KP37TOLV2bdO6EWrBggVsHBeKpyci1wKOi86w4N6PxziddY3R/0nbTcRLv/ok2re4zyrn17jO6t2o7IRUBhClXrlxgwIABgaNHjwaWL18eOOusswI9evRw/69atSrm1urevXvg+OOPD4wePTrw7bffBt58881A69at3W/79+9XJZRAiRIlAv/4xz+Cv+v7U089FTafX3/91S1Lny+++CLQsWPHQL169QKHDh1yv69YscLN6/HHHw+O5/0WSWkpXbp04MiRI+77iy++6JZ54oknBsfRfKpUqRL8ftVVVwVKlSoVeOmllwILFiwI3HXXXYHixYsHli1bFrYuxYoVCzz88MOBRYsWBf7444/Axo0bA1WrVg00atQoMGXKlMCsWbMC559/fuDMM88MJCcnx9xuTz/9dKBixYpR98Nzzz0X/K7tpeXWrVs38N577wUWLlzo9lNiYmJg5cqVKcarVatW4J133gn8+OOPgT///DPw/vvvBwoWLBjo169fYN68ee638uXLBx599NHgtL169Qp06NAh+H3Lli3Bbax99uCDDwYKFSrktots2LAhcNFFFwXatGkTHE/LkniW9/LLLweKFCkSeOaZZ9w8H3nkkUDRokVd+teuXRtzm6W23PHjx7s0ajma56hRo9w8ddzFcvjw4cAJJ5wQ6NmzZ+Drr78OzJkzJzBixIjAnXfeGRync+fObr46pmfOnBn47LPPAo0bNw6cfvrpweMr3vX25tWtWzd3nHz11VfuPGvatGlYuuLdPpH7Ld75P//8825+2kaa/2OPPeaO67S2fyxpHS+xaF+2b9/e/d+pUye3P9LSsmVLN240Om90Xpx99tnu/NM5PXv27DTnqfNU6z98+PCw4dr+2iavv/56zGn37t3rjqGuXbumuZx4t3s82zPe4zKSzpcCBQoEPvnkE/d927Zt7ruuj962WrNmjUuTzod4zy3tl1tuuSVsWUqX5qNrZKxjNqPXgqw+N1Ob/8UXX+zm791z0rN/Iqe95pprAiVLlgy88MILbvx7773XHQM//PBDqmnJzwjWYgRrHp0AOrhTM336dHdSff7552HDvYuuF+AMHjw47PebbropGNDFsm/fPnfhnTZt2v92mpkLStLy22+/uXF1AxZdSIcMGeIuCj/99JMbppvEdddd5/7/8ssv3fhvvPFG2Hw0nYLR0HWJ3Ca33nqrO/k2b94cHLZjxw4X+L399ttZFqzNnz8/7OZStmzZqOOF7guNpwBVN8VQulgpzUpntAtoNNoO119/ffC7AvnIm2Y8y9M4FSpUCNx9991h4/Tv3z+uC3S05eoBQ/MMPX5FF1HdhHbv3h11Xgq2tczFixeHDQ99CNBFNykpKbB9+/bgsNWrVwcKFy4cmDBhQtzr7c2rRo0aYUGJdw55N7T0bJ9owVo889fDlW4SoXSjzWiwFs/xEs3UqVPdDa158+aBmjVrurTqvM9osKaHtDvuuCMwY8YMF6jqeEhISHDnWlqUVm0X7+FMx4bSpG0Sep5F6t27t7uuLFmyJNX5Z3a7R27PeI7LWBTAe+eKjtFTTjklcPnllwfuu+8+N2zs2LHuuNVxFO+5lZFgLbPXgqw8N2PNv1KlSu7an5Zo+ydyWgW/Wq9x48aFTatt36VLlzSXkV9RDJoFvvrqK0tKSrJ27dqFDS9UKLxKoLKKQ5144okpilvU2kvFUS1btnRFocrSVlHp77//nu50qSKyPtOnT3fzUCuvzp07u3l//vnnrphrzpw5LitaVKymNKslWCgV7ao4ONTZZ58d9l3TqgJzhQoVgsPKlCnjitUip82M0G1YoEABq1y5ctQiq9D0qYhPWe8qagrVsWNH27Nnj9vm0ezcudMeeught18bNGjgsvu/+OKLNPdFPMtTPSAVG3vb3qOilYzy5qn9FUrf9+7da8uXL486nSqHe8XZqjel4ggdL6rEHkrjHHfcccHv1apVszp16gSLx9KznRs2bBh2fuhcEG9fZnb7pDX/1atXu2IpFTmGijyH0yOjx4v2j+p7qTqAiuX0vUqVKikq9cdLRb4qGlZxtuapumUadvfdd9v+/ftTnVbjqphP1x0dFyou86pSlChRIuo09957r02YMMHGjRvntntq0rPd492eaR2XsejY0rVRdD3Ud330vzdMRXc6jjJ6bsUjK64FWXluRqPjIbIYPt79Ezmt7hW6dkfWbYx2n8H/0MAgCyjoiay/EU1k8CahzfRVx61Vq1Z2zTXXuHoP5cuXd9OoDlho/aH08C4+OqFUT0Dz8oap/oDm6104VU9P6dHJFUo3EN3kDh48GBwWub6adtasWSkqNaueULFixSyrpLUNo6VPaRPdrCLrk3gXy2guueQSt32GDx9uNWrUcPNU3ahVq1almsZ4lucFtZEXwHiOo1h004g2D++793skXTgVtKtupm5eavmsm/oLL7wQFrhHqzOlfevNNz3bOdp+DN2X3rwyun3Smr8XtEQem5k5VjN6vIgezFQH6KeffnKtLFUvUY0OdK4ef/zx6UpHtHXXOa56Yn/88YdbViyqM6flq46Rbu56GPLqEul6EUnB/SOPPOLqS6lhQlrSs93j3Z5pHZexaNuqQYTqhul6qONeQYdaWatFrQKPe+65J1PnVjwye6xHmzYz52Y00dIS7/6Jdq9ISEhIkXmh+4zqPCrNsR4M8jOCtSygSv96MtJTY0a69vCoHzadYC+99FJYs+zILj90c02tL6bIC5IqWqtirJ60dSHXMF1gTz75ZHdxqlixohtXfTcVKVLE3n///WCrr1CJiYlhAVsoTSu64URSrmMsJUuWTNGqUd91scwqtWvXdmnXDTDyiV50Q4qkp38FMApeQqeJbG6v7RS5L+JZnraz9oVyVEJbGqf1hJvacmvVquXmqXmotZ7n+++/d39POeWUmPNTeq6++mr38RqnKIf3sssucxdWWbZsmctx844N3Xj1xO41MMnIdo5FOcKZ2T5pqVmzplsPNbBR7q9H3zMi3uMlHsqR1jbUeaubaHqDtWh0A9V+jHdeylXVDVjUHYeua2rxGUoBnRoUKTeub9++Wbrd07M90zouY1Huu4ICdT2ih0rloul6pDQqZ1IV9L3crnjPrWjXs7QCoaw41o/luZnZ4133Ch1fejiMzL2XrHy4z0soBs0Cyn3QCXfDDTcEi1nUsur+++9P13x0wihredGiRcF5ROs4UUU6v/76a1zzVNGkWjcpl8TLVlfOmW7Oao0TWvyg/pQ0bz0l669yyXQRUotOBXveTTsaXQRUzKqLhqbRtLrYz5s3z12EUsu+15OUui8QBYNqAatcwKyiC+htt93m1leBrteCSbkIuuFoWCT9pifC0Nam6vtq7ty5YeNpO+lGGJreeJanJ2EFQ+qawXsS1bbTTSIe0ZarfapcAbUyXrFihRumBwg97arI4YQTTog6L+2ff/zjH8HuYDRPPf0qvaH7XBdiBfkKEjWOunfQ/2r5nNHtHEtmt09adEPQ+frYY48Fb6bqEifa/HVT0nqk1lot3uMlVjGiOpT1gm+dr8oFU+5rRrrf0H4Jzd3Qean9qxZ3yq33qCVjZAvMTz/91I0vSo+KYrXvtC6h+0+twVU8qu2n1phZvd3Tsz3TOi5jUfCiFpFadvPmzd3xKwrQnnrqKXeOebmQ8Z5bCminTp0abKGvddS6ZvexfizPzcwe73oAVPCvTAltN+8+o+vPBx98kO605BdslSygC5CyzPVUo7om+uhJJrT+Vjx0MdXJpQuHDmYdyJpXuXLlUlzc1Xu4ngBT67pDVI/BC4i8p0SdDHoaCh3mrYdOPuXmVapUyQVbpUuXdn0vqZ5balR8+95777kAVTkDmlY3BjXljixWDaU6LrrIaL1VR0YXSE2vT1bSzUoXCT1NKyjW+ukpVts3Wpa7LuSqg6OnbuU8ajuqab16bA+lYFpFARontAuNeJancZQrqxxOrbdyPiPrlMQSa7m64GvfartqWZq3lhGaWxtJx5EeEvTAobouOt4UoHgBtEf1nxR8K3dG+0dPxhonNABI73ZOTWa2TzyUC6ybhNZb6dP6ed3zhD7x62FFAUW03Ob0Hi/RaLnKydQNUIGSplc9WHVnEVqEpCDBu8mqd/vQrltCAy6db6p3pv2ifaV6qsr5iuxmwluv0E5Xtb213XXt0vS61igwu/TSS8Om7devnwvktc6Rb0rIiu2enu0Zz3EZi65/kddBb1hkPbp4zi3tR/3unUsKBuM5BjJ7rB/rczMzx7sCX91ndMwqHbrX6T6jh/TIutD4nwS1Mgj5nu8px0oXTS/Q0pORDi6vqDAtKrLUjU+Bh5croU2si6JOQu/pzctKVuVO3SxDKRtdxao6kBVAKU26gYZWIFUOlOpa6IatLPpo2ckejad0hRaDadn6xJpWOSubNm0KpsETa11Cab6aXheBeJ+S1H+bTl4FqLopap29C4+3XVVUEXkz0JOo0uEV78Qaz6OnTgWjuglGFglpO6koQen2KJdD46soV+Nrv6iIIbS4W/NU2rVsXRzLli0b1/JCt5f2uY4ZbQc9sSvYj1XvKp7l6maj37RMHc/x0L711jX0WPP6oFJ6FJBr/TVvpTdWAJPaeuvhQudGaE6f1lsVk3Uu6HxLz/aJ3G+Zmb+CXt00tE29+atz2n//+99xFUvFc7zEonNZQZa2ceT29875WPXfoh0vCsYUiMW6dnlvOog2rX5TemIVj6l+bazc73hzA9Pa7vFsz/Qel5F0nihXKvR65h0rugeEnlPpObd0D9D+0rbXtVD3EQViXrqiXWsyei0IPd4ze26mNf9Iae2f1KZN7T6DlAjWAKQp9KaYV6ielB481PGzHirUok85A6qTpFwCj3JT1EmpV58vO6kTX70aKq1APT9s9/x6XALRUAwKIF/S079u8so90ZO/uqnw3ssZSrk+WVn8mhrVF83LgVp6tjsAn+WsKTtYT1fKllVZfzTKqlb2rvqOiZX1HM84ANIvreKM3EzFVSq+8YrgkXu2e14+LgFf5ayp9YfK7VUJVh9V7gx9h5rqTahSpOpbqf8xVYpUfzih4hkHQMapTk9evSGqDlt66jnBP9s9Lx+XQKgczW9Xiyd1F6FiBjXnFrVw8ipvippGq8m8XkKuSoiqz6GWQ+qjx+uTJp5xAAAAcqMcLQZVH2BqiTht2rSY4yiXTP3bqEWWRx25qisJ1e+IdxwAAIDcqEBO1ldQX056ZYW6r1BHsGrOHEpdMCiXLfK1FMoxW7x4cdzjxFq+mol7H3XIpx6rfVCFDwAAIOeDNdVLUx9A6j1ddc3Uyaf+6mWuej+Y10+QRHYKq+/eb/GME406OFRHfN5H/XmpXx0FjgAA+MW+RYtsed1Tgx99R/6SY8Ga1xGrXqCrzibVI7c6fNRHvUCHjhP5EnN1uqcelOMdJxq9fFa5ad4nI+/wAwAAyLMNDNSAQK+4UMeI3tsClBumBgfea27UUkidJka+Tknf9SqPeMeJ1RIpsidzAAAAv8mxnDUFWOqmIzLI0qsrvPeZqd81vSvso48+CnvVx/Tp04PvcotnHAAAgNwqR7vuePDBB12LTbXi1N9vvvnG3nrrLZs0aVJwnIcfftgFXSq2bNGihes/TTlxejFxesYBAADIjXK0U1x1rzFv3jyXm/b444+7+mqzZs1yfaR59L64mTNnuhfajho1yurXr+9akerFsekZBwAAIDfyxeum/EDdd6hVqBob8KoqAIBfqPXn6l5XB79XnzjBikd0V4W8LcdfNwUAAIDYCNYAAAB8jGANAADAxwjWAAAAfIxgDQAAwMcI1gAAAHyMYA0AAMDHCNYAAAB8jGANAADAxwjWAAAAfIxgDQAAwMcI1gAAAHyMYA0AkOvs+fJL2790aU4nAzgmCh2bxQBA/nZkxw7b9eFkO/DTTxY4fNgSq1WzUhdfbEVPqRM23l9ffWU73//Aqox6xvKqdQPusDKXX25JrVpmeB6H1603CwRi/q5tvGfmTNszdZodPXDAqr74Qsp5bNlim+69L8Xw4medaeVuuinNNCT/tdd2vvO2HVi+whIKF7aS7dtZyfPPTzGe9vnuqdPs0No1VrhCBSvZoYMVb9w4rvUEhJw1AMhme+d/Y7916OgCsRItWlipjh0sefdu+6N7d9s2enTYuIc3bLC/Zs/O0/tE63d444YMTZu8a5dtG/Oy7Zo82baPe812vP22HT10KGycI9u22a9t29nujz+2owcP2l+zZkWdV2D/fvdbibNbWJkrewQ/Jc4+O67ge9Wll9qe6V9Y0jmtrEidOrZx+F22ZdSosPG2Pv+Crbqihx396y8r1b69JSQm2prrrretL6QMHoFYyFkDgGx0eNMmW3fbbZbUurWd+MQ/g8NLXXihFWt0hm0cNtwSa9R0ARxSl7xzp/3epasVOaWOFal1khUsU8blau2+8Sar/uYbwfEKlCxpNSd/aIXKlrVtL71kf335ZarzLda4iRVrcFq6Nv+2F1+yo/v3W7VXX7ECJUq4YYlVq/w317BzZ0usUcMNO7pvn1UbO9ZKNGv63/1+0UWWULSom/64nj2t0HHHsduRJoI1AMhGOyZMsMCBA1Zh0MAUv5Xp0sW2v/GGCygig7VDa9bYznffs0Nr11rhSpWs7LW9rXDlysHfA8nJtvuTT2zvNwsscOiQFT31VDvuqiutQPHiwXEUTKhIdf/ixRY4mmwlmrewMpd1s4RChcKKXCsMGWw73nzTDq1f74rx9kz7zCoMHmRFTj45LE0bhg6z4s2aWZluXeOav5dTuP2NN+3w+nVWqFJltx7RrO3X30p27GhlunaJuS13T51qgYMHrerzz9uOdyZZ4RNPsJLnnedyuUIVKFLEfbLTgR9/tKKnnRYM1KR406Zmycm2Z/r0YDFq+dtutQLFioVNq32l8Y5s2UqwhrhQDAoA2Wjv1/Nd0BMaaIVKanWOHfz557CAQwHJHz17uUCg5Hlt7MDy5baq++V2ZOvW4Dhb/vmEbXn6GStav56VbHu+Je/c4QKe0OLCP664wra/9poVa9LYFe39OXasbbhzaFggpcBiTe9rrVD58lamWzcr0by5HVq92nZMejcsnfu+W2y7/v1vK3Jy7fjnv2WLS/eBn1dYyfbtrVC5svZHjyvd+kVSceSh339LdVse3X/A1VMLRNRVy2zu1LYxo23d7QNs88jHXPAbjwJJSXZ0z57w9P31l/t7cOXK/40XEajJ3jlzXHFoYo3qmUo38g9y1gAgGx3ZssWKndEw5u+FTzjh/8cLyWVJTnY5W8p5k1KdOtlvHS90OXCV7rsvmCtW9upeVrZXr/+Oc+GFLoDyqE7UoXXrrdanU1zOnJRo2tR+u6iTHfddr/9VcE9OtkoP3G9J55wTnLZ0164uCKs4ZLALKmTXhx+4elnFGjSIe/5Kb8GSJa3aq69aQsGCbpxC5SvYxrvvTrEdqrz0oiVWTz14KdmurW197jlbc01vK1j+eEs651wXuCUkJFhGJdauZcUbNbZClSra/iXf29qbbrIyl3cPbudYks5rY5sffsT2Lljg1lvp+PPVse635F27Y06nYFC5guVuuD7bc/+Qd5CzBgDZSMHO0b37Yv6uOk3/Ha9w2PCS7dr/bx6FClnSeefZvoWLgsOKnXGGbX/rLdsxaZIrKpWCpUsHf9876ysXgHmBlCgYUhFcWAOGAgVcblqo0l26WPKePbZnxsz/pnH/fts95VNXxJme+e9fuMiSzj8/GKi59eoQvW6eijOLnHRSzO3k5l+1qp30wftW5JRT7MDSH23TiBG28uyWtv3NCZYRhSpVspM++MDK3XiDle7UySrdfZdVvPtu2/HWv2zf4sWpTnvclVdamR5X2Nqb+rhc0N8v6mSH/ljl6qolxAjClOO2/vbbXf218rffnqE0I38iZw0AslHiSSfZod9/j/m7ihx1c0888cSQK3MhK1Dif3XPRDlUqmDvqfzA/bbzgw/trxkzbcuTT7nxj+/b1wURomBLxaehRaNyeONGO7x+ffC7Krur24lQhStWsBKtWtrOD953denU7YTqxZW69NLgOPHMXzl9BUuVDPvdrVdInbb0UjBU+cEHbPuEiS7dh9ets82PPOKGq1VmekTL2SrZ4QLbdP/9dmDpUiveqFHMaRMKFLDKI0ZY+ZtvtoO/r7KCJZOsyKmn2soWZ7tuWSIpoF5zw42WWKuWVXn++RTbHEgNwRoAZCMVT2686y5XbJl07rlhv6n7jt1Tprg6Z15xo3PkiAtClJMUGtQVDvmum/1xPa5wn8DRo7bzvfds030jrFijRlb0lFOscNUqVui4si73J1QZ6+GKItNSpttltn7gQDu8eYvt+uADl0MWWjcsnvkrvYf++CPs98Nr1rj1ywqFKpR367/rk0/swLIf0x2sReMFxKENB1JNQ/ny7uPV69P0ygUNdXjzZtddh3Lyqr48JqwRCBAPikEBIBuV7nypFT/zTNv00MMuBya0Q9UNd93lcmgqDBqUYrqtzz7ngjDZv2yZ7fnss2ArTFEr0mARaoECLkCTwKHD7q/qsqmHfwV8KmL0Pi43KfDf+aam5PnnuWLVrc89a/sWLrQy3S8L+z2e+atlp3LlDqxY8d+0JSfb1lHPRl2ecuh2fjg57bcW/LgsbNiBn3+x5G3brEjdupZeu6d95hpZeI7u3Wtb/vG4C6ZCA2t1v6L0/TVnbnDYke3bXQAe+n3Tww9ZUru2Vrxxo7Dha66/wQqUKuW6+SiYlJTudALkrAFANlJ9rapjRtumRx+1VV26uDpdCcWKua4f1Eq0+lsTrXBoEaimKVLEju7fZ79d0MEKVajgxi15YUdX8d+j1qO/dujgGigUKJzoiiTLXnttsL+w0pde6gKFP67q6QIqBQvK5SpSq5ZrUJBmugsXttKXXGzbX3/D5QiVaBn+toF45q/0qgK+WoAWrV/fjmzebEXr1Ytap0utQYvUCe8qJJJy9jY//LBrFeu6B0lIcC1Oy95wg5Vs0yZs3PV33mlHd+12XaDY0aPB4trjrrkm+OYE5Z6t7ddPa+v6bDv4yy//zf165eVgbpkoKFb6FIiFtvLc8fY7tvmxf7hxtY+S2rSxyg8/FJYOBX8qBi/WsGFYS1k5/vbbrFj9+mnuCyAhENkGOp/avXu3lS5d2nbt2mWlSpXK6eQAyINU7KluOvQqpMLVqltilfAgTZTTc/C331zlfbW2PLx2jQsgitSsmWJcBREKMJRjpbpx0bqw0KuWDvy03NU5S6xZ09VHi7asaPQmAOWeFa5Y0QVZ0aQ2f8/BVavs8PoNVrhSRStSu7ZrgKC/od2Z6NVQaqCQViMDl+5Nm+zPceOscOUTXMvNaLlVe+fNc28viFS0bt3w/uqOHrVDq1bZka3brPAJlV3RbWTrUm3nvd98k2JaUYCqenqJJ9WKuu7KCTyydUvU9VAAp45707Jv0SJb3evq4PfqEydY8SZN0pwOeQfB2v8jWAOA3EMNDLxOcfM6gjVQDAoAyHVU5FigWNGcTgZwTBCsAQBynWhFyEBeRWtQAAAAHyNYAwAA8DGCNQAAAB8jWAMAAPAxgjUAAAAfI1gDAADwMYI1AAAAHyNYAwAA8DGCNQAAAB8jWAMAAPAxgjUAAAAfI1gDAADwsRwN1oYPH25VqlQJ+7Rr1y7FeJMmTbKWLVta7dq1rWvXrrZ8+fIMjQMAAJDb5GiwtmPHDmvYsKHNnz8/+Jk4cWLYOB9++KH16tXLrrnmGnv//fetZMmS1rp1a9u6dWu6xgEAAMiNcrwYtFixYmE5axUrVgz7/aGHHrJrr73W+vfv7wK7cePGWSAQsJdeeild4wAAAORGOR6szZo1y+rUqWNnnXWWDRw40OW2eXbv3m2LFy+29u3bB4cVKlTI2rZta1999VXc4wAAAORWORqslS9f3h588EGbPHmyPfrooy64Ur2zAwcOuN/Xr1/v/laqVClsOuW+eb/FM040Bw8edIFe6AcAAMBvCuXkwhWoJSQkuP/r1atnp512mlWvXt3efvttu+666+zo0aP/TWSh8GQWLlzYkpOT3f/xjBPNyJEj7YEHHsjydQIAAMgzOWteoOapXLmyC9a8lpzKeZNt27aFjafv3m/xjBOrJequXbuCn7Vr12bRWgEAAOShOmuh9u/fbxs3brSyZcu67xUqVHDB29y5c8PGmzNnjjVt2jTucaIpUqSIlSpVKuwDAADgNzkWrKnO2G233WYbNmxw35W71adPH/f/lVdeGRzvlltusbFjx9r333/vijyfffZZW7NmjfXt2zdd4wAAAORGOVZnTTlbqqPWqlUr1wJUuWrNmjVzrUOVU+YZNGiQy21r3ry5FSxY0OWAqU7bqaeemq5xAAAAcqOEgDoky2HKVUtKSnKBViyHDh1y4x1//PEp6rqlZ5xY1Bq0dOnSbnqKRAEAfrFv0SJb3evq4PfqEydY8SZNcjRNyEetQT0KktKSmJiYaoOBeMcBAADITXzVwAAAAADhCNYAAAB8jGANAADAxwjWAAAAfIxgDQAAwMcI1gAAAHzMF113AMi/xowZk6np+/Xrl2VpAQA/ImcNAADAx8hZA5CrkTMHIK8jZw0AAMDHCNYAAAB8jGANAADAxwjWAAAAfIxgDQAAwMcI1gAAAHyMYA0AAMDHCNYAAAB8jE5xAeRYh7QAgLSRswYAAOBjBGsAAAA+RrAGAADgYwRrAAAAPkawBgAA4GMEawAAAD5GsAYAAOBjBGsAAAA+Rqe4QD5Gp7YA4H/krAEAAOSHYG3Pnj0WCASyanYAAADIaLC2fPlyGzp0aPD7gAEDrFSpUnbCCSfYkiVL2LAAAAA5WWdt4MCBNmjQIPf/L7/8YmPHjrVPPvnEvvjiC7vzzjvts88+y6r0AYDv6+3169cvS9ICAFkWrM2bN8/ee+899/+0adOsc+fO1qlTJ2vZsqXVrFmTLQ0AAJCTxaBFixa1TZs2uf8//fRTO//8893/hw4dssTExKxKGwAAQL6XoZw15aJ169bNmjRpYrNnz7bx48e74SoGbd++fb7fqAAAADmas/b888/bpZdeakeOHLGPP/7YKlSo4IYrcBsxYkSWJQ4AACC/y1DOWvHixe2hhx5KMfzFF1/MijQBAAAgs/2sHT582ObOnWtvvvlmcNiff/6Z0dkBAAAgq4K1NWvWWOPGja1t27bWu3fv4PA+ffq4YlEAAADkYLD297//3Vq0aGG7d+8OGz548GB77LHHsihpAAAAyFCdtVmzZtmKFStSdNPRoEEDW7RoUYa26oIFC+ytt96y1q1bW9euXcN+27x5s/tNf7WMHj16WKFChdI9DgAAQL7IWTtw4IAVKPDfSRMSEoLDN27c6BofpNeOHTvsqquustdff90FgqFWrlzpgq+pU6da4cKFXWvTjh07WnJycrrGAQAAyDfBWps2bWz06NFhwdrevXttyJAh1q5du3TP78Ybb7TrrrvOqlevnuI3vYO0bt26rvNdtUCdMWOGffXVVy4XLT3jAAAA5Jtg7YknnrBnnnnGWrVqZYFAwC6//HKrVauWffPNNzZy5Mh0zeuFF16wLVu22F133RW1xemUKVOsZ8+ewZy8atWquWBx8uTJcY8DAACQW2WoUpdysX788Uf3AuRy5cq5YlG9yPjmm2+2ihUrxj2fH374wR588EGbP3++FSxYMGqr04MHD7pAMJS+q9uQeMeJRtPo44lsLAEAAOAHGa6Br7cW3HvvvRlesIpN1QhAuXSxXv6+b98+97dkyZJhw0uVKhX8LZ5xolEO4AMPPJDh9AMAAPi6U9zMmjhxomuQoNajd9xxh/usX7/eNTDQ/0ePHg0GYDt37kzRIMH7LZ5xohk+fLjt2rUr+Fm7dm02rCUAAMAxyllT68p4qVVmWpo1a2b3339/2DB1BaIcsRo1ariGC6p7lpSU5LoJCV2+vterV8/9H8840RQpUsR9AAAA8kSwNm3aNBs0aFCWLbhhw4buE2r8+PHWqFEjl7MmCtguu+wyN7x///5WtGhRV89NddE++OADN44aFaQ1DgAAQL6os6b6ZbJt2zY7/vjj7VjQGxHUUe5ZZ53lAjm1/Lz66qutc+fO6RoHAAAg3zQwKF++vOuyI6upn7YqVaqEDatUqZItWbLE9aGmtxP07dvXdRmS3nEAAADydLCmgGjOnDnWsmXLbEtMr169og4vVqyYdevWLdVp4xkHAAAgzwZrenn7ueeeG8xRC33NVKTsyHUDAADIj+IO1u6880678sor7ffff7fzzjvPPv/88+xNGQAAAOIP1i6++GL75JNPXFcZkpF3gAIAACAbu+5ITk6O+looAMjP9Oq9zNDr+gAg08HaySefbPfdd581b97cfVcuW2q5cAAAADiGwdqLL77oOqsdNWqU+969e/eY4+rF7gAAADiGwVqbNm1cX2ZeS1ACMgAAAJ++yH3p0qVZnxIAAABkzRsMypQpY+vWrYv5e+RbCAAAAHAMg7WqVaum+jud4gIAAORgsLZ8+fKw70ePHrWVK1fasGHDbMCAAVmUNADZ2V0EACAPB2t169ZNMaxevXoux+1vf/ub9e/fPyvSBgAAkO9lqIFBLHXq1LGffvop329UAAAA3wVrBw8etCeffJLGBQAAADldDJqUlJRi2L59+6xkyZL21ltvZUW6AAAAkNFgbcKECSmGHXfccdawYUPXrQcAAAByMFjr0qVLFi0eAAAAWR6sed11qGPc7du3p/jtjDPOyOhsAQAAkNlgbd68edazZ09bvXp11N/pFBcAACAHg7VbbrnFOnToYIMHD3Z11QAAAOCjYO2XX36xWbNmWalSpbI+RQAAAMhcP2u1a9e2LVu2ZGRSAAAAZEewduDAgeBHxZ833XSTLV261Pbv3x/2mz4AAAA4xsWgxYoVSzHs9NNPjzouDQwAAACOcbA2e/bsLFokAAAAsjxYa9WqVaq/79mzx72GKiEhIe6FAwAAIBsaGCxfvtyGDh0a/D5gwADXMvSEE06wJUuWZGSWAAAAyKquOwYOHGiDBg0KduMxduxY++STT+yLL76wO++80z777LOMzBYA8qUxY8Zkeh79+vXLkrQAyENvMHjvvffc/9OmTbPOnTtbp06drGXLllazZs2sTiMAAEC+laFi0KJFi9qmTZvc/59++qmdf/757v9Dhw5ZYmJi1qYQAAAgH8tQzppy0bp162ZNmjRxrUTHjx/vhqsYtH379lmdRgAAgHwrQzlrzz//vF166aV25MgR+/jjj61ChQpuuAK3ESNGZHUaAQAA8q0M5awVL17cHnrooRTDX3zxxaxIEwAAADKTswYAAIBjg2ANAADAxwjWAAAA8kKwdvHFFwf/nzBhQnalBwAAABkJ1qZOnWrJycnu/2uuuSbeyQAAAHAsWoPWqVPH7rvvPmvevLn7rtdLxZMLBwAAgGMQrKlbjjvuuMNGjRrlvnfv3j3muAcOHEhXIlatWmV//fWXe1VVUlJS1HHWrl1rmzdvdkGjXhqf0XEAAADyZDFomzZtbMmSJS6o8gKyWJ94ffTRR3bqqadax44d7corr7SKFSu63LtQmt9ll11mp5xyiit+rVSpkj333HPpHgcAACDfdIq7dOnSLFm43i+qF8FXq1bNfZ8xY4a1bdvWzj33XGvXrp0b9sADD9iCBQvst99+s8qVK9vkyZOta9eu1rRpU2vWrFnc4wAAAOSbrjtOO+009/fPP/+0r7/+2ubNm+f+T6++ffsGAzU555xzrFChQrZ+/frgsNdee81uuukmF4RJly5d3PI1PD3jAAAA5Juctf3797v6a2PHjg22EC1YsKDdeOON9swzz1ixYsXintfOnTvtxx9/tN27d9u4ceOsYcOGrkhTNmzY4Oqg6YXxoZRjtnjx4rjHiebgwYPu49HyAQAA8kTO2pAhQ1yR5XvvvWcbN250xZn6/4svvnC/pcfPP/9sw4YNs4EDB7p53nzzzcFGBtu3b3d/y5UrFzaNvnu/xTNONCNHjrTSpUsHP1WrVk1XugEAAHybs/bOO+/YZ599Zo0aNQoOU9Fj9erVrUOHDvb888/HPS/VKZszZ477/8svv3TTlyhRwnr06GGFCxd2wyMbLShnLzEx0f0fzzjRDB8+3AWIoTlrBGwAACBP5Kzt2bPHBWaRNCwzxYlqcariy08//dR9V/BUoECBsDpsou9eXbd4xommSJEirnuP0A8AAECeCNYaN25sjz/+uAUCgeAw/f/YY4+lqDsWy9GjR1Pkhh0+fNjWrFkTLNIsXry4nX322a6LD8/evXtt+vTp1r59+7jHAQAAyFfFoE888YTrG+3999+3s846yw379ttvXUV/vZYqHqrcr2nVirNevXquocHLL79s+/bts1tvvTU43sMPP+yCLhVbtmjRwvWfVqFCBdeSND3jAAAA5JucNeVkrVy50nr16uVyw44cOeL+1zD9Fg+1GFW9ty1bttjTTz9tkyZNsvPOO89WrFjh3mTgad26tc2cOdNWr17t3p5Qv359V8ct9E0H8YwDAACQGyUEQssy8zHVtVOr0F27dlF/DbnCmDFjcjoJ8JF+/frldBKQTfYtWmSre10d/F594gQrHmeVI+TjnDUAAAAcGwRrAAAAPkawBgAAkNeCNXVeCwAAAJ8Ga+3atQvrYw0AAAA+CtZq1Khhv/zyS9anBgAAAJkP1u655x677rrrbP78+a7LC72JIPQDAACAHHyDwfXXX+/+6m0B0VBECgAAkIPB2uzZs7No8QAAAMjyYK1Vq1YZmQwAAADHqp81vRN07ty59uabbwaH/fnnnxmdHQAAALIqWFuzZo01btzY2rZta7179w4O79Onj3388ccZmSUAAACyKlj7+9//7hoXqCVoqMGDB9tjjz2WkVkCAAAgq+qszZo1y1asWGGJiYlhwxs0aGCLFi3KyCwBAACQVTlr6kutQIH/TpqQkBAcvnHjRitevHhGZgkAAICsCtbatGljo0ePDgvW9u7da0OGDHGvogIAAEAOFoM+8cQTdu6559qUKVNcB7iXX355sO81tRAFkLYxY8awmQAA2ROs1a1b13788Ud3sylXrpwrFu3Xr5/dfPPNVrFixYzMEgCQg8G/ruEA8lCwJhUqVLB77703a1MDAACArAnWVEftrbfesuXLl7vv9erVs549e9LAAAAAIKcbGCxcuNBOOukkGz58uC1evNh9hg0bZrVq1bLvvvsuK9MHAACQr2UoWOvfv7917tzZ1q1bZzNnznQf/X/JJZdQ7wEAACCni0GXLVtmU6dOtaJFiwaH6f9HH33UqlWrlpXpAwAAyNcylLOm4s5NmzalGK5hKh4FAADAMQ7W1D2H9xk4cKD16NHD9bO2detW27Jli/v/iiuusEGDBmVR0gAAABB3MWixYsVSDOvUqVOKYTfccINdf/31bFkAAIBjGax5bygAAACAD4O1Vq1aZW9KAAAAkDUNDAAAAODjYE19ql111VVWtWpVS0pKSvEBAABADvaz1rt3bzty5Ig99NBDVqZMmSxKCgAAALIkWJs/f76tXr3aypcvn5HJAQAAkJ3FoJUrV7b9+/dnZFIAAABkd7A2dOhQ69u3r/36668WCAQyMgsAAABkV7DWpEkTW7BggZ188slWoEABS0hICPsAAAAgB+us6S0FZ511lvXp04cGBgAAAH4L1n755RebOXOmlS1bNutTBAAAgMwVg9asWdN27dqVkUkBAACQ3cHa3/72N1cUunTpUtcq9MCBA2EfAAAA5GAx6O233+7+nn766VF/p4Uo8roxY8bkdBIAAPlEhoK12bNnZ8nCVfft6aeftnnz5lmhQoXcy+Lvvvtuq1ChQth4kyZNslGjRtnmzZutQYMG9uijj9qpp56a7nEAAADyRbCmoCqzkpOTrWvXri6XTsWq+/bts8GDB1vbtm3t22+/taJFi7rxPvzwQ+vVq5c999xz1qJFC3vyySetdevWtmzZsuAbFOIZBwAAIDdKCGSgzFIvck9NlSpV4prP0aNHXT9tnpUrV1qdOnVcS9M2bdq4YY0bN3afV1991X3XO0n1BoXbbrvN7rvvvrjHScvu3butdOnSruFEqVKl4poG+RfFoMhr+vXrl9NJQAz7Fi2y1b2uDn6vPnGCFW/ShO2Vj2QoZ61q1aqp/h5v/BcaqMnhw4fd38KFCwcDqMWLF7s3JnhUXKrct6+++irucQAAAPJVsLZ8+fIUOWTKFRs2bJgNGDAgQwlRgHfXXXdZ7dq1XYe7sn79eve3UqVKYeNWrFjRvv/++7jHiebgwYPu41HQBwAAkCeCtbp166YYVq9ePZfjpvpn/fv3T/c8FajNmDHDvvzyS0tMTAwGgS6RhcKTqZw31XmLd5xoRo4caQ888EC60wkAAOD7ftZiUX2zn376Kd3T3X///fb888/blClTXN0zj9c4YNu2bWHj67v3WzzjRDN8+HBXP837rF27Nt3pBgAA8GXOWjQqUlQrzHgbF3iUu6XpFKhFtjJVFx7Vq1e3uXPnWufOnYPD58yZY5dccknc40RTpEgR9wEAZE2jGRopAD7KWUtKSkrxKVasmD311FP2xBNPxD2fRx55xI2vQO2cc86JOs4tt9xiY8eOdfXPVOT57LPP2po1a6xv377pGgcAACDf5KxNmDAhxbDjjjvOGjZsaGXKlIlrHtu3b7d77rnHihcvbldddVXYb//85z+DwwYNGmQbN2605s2bW8GCBV23Gm+//XZYh7fxjAMAAJBv+lnLCsoB27BhQ9TfypYt64K4UIcOHXJ1y44//nhLSEiIOl0848RCP2tID/pZA1KiGDR70M8a0pWzFm8rz9GjR8fVx1p66rephWhabyOIZxwAAIA8G6z9+uuvMX9TBp3e8XngwIG4gjUAAABkcbA2ffr0qMP1eqg777zT/T9kyJD0zBIAAADZ1c/aDz/8YBdddJG1b9/eGjRo4N5i8Pjjj2dmlgAAAMhssKZuMa699lpr1KiRa32pLjPGjRuX7j7WAAAAkIXB2o4dO1wxp95UoFy0WbNm2ccff2z169dPz2wAAACQHXXWTjrpJNeA4Pbbb7du3bq5YfPnz08xnvo7AwAAwDEO1nbu3BnstFafWHKo6zYAAID8HaytWrUq+1ICAACAzAVrNWrUSM/oAAAAyMmuOwAAAJC9CNYAAAB8jGANAADAxwjWAAAAfIxgDQAAwMcI1gAAAHyMYA0AAMDHCNYAAAB8jGANAAAgr7zBAMgrxowZk9NJAAAgLuSsAQAA+BjBGgAAgI8RrAEAAPgYwRoAAICPEawBAAD4GMEaAACAjxGsAQAA+BjBGgAAgI8RrAEAAPgYwRoAAICPEawBAAD4GMEaAACAj/EidwBAlhgzZkympu/Xrx97AoiCnDUAAAAfI1gDAADwMYI1AAAAHyNYAwAA8DGCNQAAAB8jWAMAAPAxgjUAAAAfy/F+1vbs2WMTJ060FStW2K233mq1a9dOMc7mzZvtrbfecn8bNGhgPXr0sEKFCqV7HAAAgNwmR3PWXn31VTvllFNs5syZNmrUKFu3bl2KcVauXOmCr6lTp1rhwoVtxIgR1rFjR0tOTk7XOAAAALlRjmY9nXXWWfbzzz/brl27bNKkSVHHGTp0qNWtW9c+/fRTK1CggPXp08flvikX7Zprrol7HAAAgNwoR3PWGjZsaCVLloz5++HDh23KlCnWs2dPF4RJtWrVrE2bNjZ58uS4xwEAAMitfN3AYM2aNXbw4EGrVatW2HB9V9FnvONEo2l2794d9gEAAPAbXwdr+/btc38jc99KlSoV/C2ecaIZOXKklS5dOvipWrVqNqwBAABAHg7WvABs586dYcN37NgR/C2ecaIZPny4qyvnfdauXZsNawAAAJCHgzXVPUtKSnLdeoTS93r16sU9TjRFihRxuW+hHwAAAL/xdbCmBgOXXXaZjR8/3g4cOOCG/fDDDzZ37ly74oor4h4HAAAgt0oIBAKBnFr4ggULXPcaf/31l40dO9YFXVWqVHF9pOkjmzZtstatW1tiYqI1atTItfzs1KmTvf7668H5xDNOWtTAQHXXVCRKLpu/jRkzJqeTAMCn+vXrZ3nNvkWLbHWvq4Pfq0+cYMWbNMnRNCEf9bOm4ssaNWq4/59++ung8DJlygT/r1Spki1ZssT1oaa3E/Tt29datWoVNp94xgEAAMiNcjRYU52y1OqVeYoVK2bdunXL9DgAAAC5ja/rrAEAAOR3BGsAAAA+RrAGAADgYwRrAAAAPkawBgAA4GMEawAAAD5GsAYAAOBjBGsAAAA+RrAGAADgYwRrAAAAPkawBgAA4GMEawAAAD6Woy9yR/40ZsyYnE4CAAC5BjlrAAAAPkawBgAA4GMUgwIA8ozMVrPo169flqUFyCrkrAEAAPgYwRoAAICPEawBAAD4GMEaAACAjxGsAQAA+BjBGgAAgI8RrAEAAPgY/awBAPD/6KcNfkTOGgAAgI8RrAEAAPgYwRoAAICPEawBAAD4GMEaAACAjxGsAQAA+Bhdd+CYN20HAADxI2cNAADAx8hZy2fIFQMAIHchZw0AAMDHyFkDAMBHpRf9+vXLkrQg7yBnDQAAwMcI1gAAAHyMYA0AAMDHqLMGAICP672VWr/ezgj5/u9//9t2L1wYc3rqvOU9eSpYW7t2rW3evNnq1KljpUqVyunkAABwzNHIIe/JE8HagQMHrFevXvbpp59a9erVbfXq1faPf/zDbrvtNstr6CcNAID8JU8Eaw888IAtWLDAfvvtN6tcubJNnjzZunbtak2bNrVmzZqZXxBoAQCAfNnA4LXXXrObbrrJBWrSpUsXO+2009xwAACA3CzX56xt2LDB1VNr0qRJ2HDlqi1evDjmdAcPHnQfz65du9zf3bt3Z1ta9+/fn23zBgDkTYUOHrS/kpOD3/cdPJjt95NnnnnGctoNN9yQqenHjRuXo8uPV8mSJS0hISH1kQK53NKlSwNajXnz5oUNHzJkSKB27doxpxsxYoSbjg/bgGOAY4BjgGOAY4BjwHJoG+zatSvNWCfX56wVLlw42MgglJ46EhMTY043fPhwGzhwYPD70aNHbfv27VauXLm0I1zkGsoprVq1qmspTAvhvIP9mnexb/Mm9mvqOWtpyfXBmm7EBQoUsPXr14cN1/dq1arFnK5IkSLuE6pMmTLZlk7kLAVqBGt5D/s172Lf5k3s13zawKB48eJ29tln20cffRQctnfvXps+fbq1b98+R9MGAACQWbk+Z00efvhhF5ipaLNFixb23HPPWYUKFaxv3745nTQAAID8nbMmrVu3tpkzZ7rOcEeNGmX169e3OXPmWFJSUk4nDTlMRd0jRoxIUeSN3I39mnexb/Mm9mvmJKiVQSbnAQAAgGySJ3LWAAAA8iqCNQAAAB8jWAMAAPAxgjXkecuXL7e5c+fG/F3VNn/66SdbsmSJHTly5JimDemn/bVs2TL7/vvv2V+5nPbj119/Hde+Tg553RL8bdWqVW6/pfZKrG3bttm3337rXheJtBGsIc965513rHnz5tayZUvXYjiaX3/91U477TRr06aNde7c2XWknFpgh5z1yy+/WL169ez888+3Sy65xGrUqJHqzR7+NGHCBDvzzDPtnHPOidkf5ooVK6xu3brWtm1bu/jii92+XrBgwTFPK+I3ceJEO/nkk935ecUVV1ilSpXs2WefTTHesGHDrEqVKnbddddZ9erV7ZZbbnGBOWIjWEOezlF7+umn7amnnoo5Ts+ePd1NYOPGja7rl27duln37t1t3759xzStiM+VV17pbgYbNmywNWvWuJu49lfk6+bgbwrEXnzxRXv00Uej/q4bd48ePVxgrn2t18V16NDB7euDBw8e8/QiPuvWrbNp06YFc9bGjx9vAwYMsFmzZoU9ROsl8bNnz3bjLFy40F5//XV75ZVX2MypyYqXqQN+9tprrwUKFiyYYvgPP/zgXqI7Z86c4LANGzYEChQoEHj33XePcSqRlu+++87tr/nz5weHrV27NpCQkBD48MMP2YC50EsvvRQoUaJEiuELFixw+3rhwoXBYX/88Ycb9vHHHx/jVCIzKlWqFHjkkUeC3y+44IJAly5dwsa5+uqrA82aNWNDp4KcNeRbixcvdn+bNGkSHFa5cmWXPe/9Bv/w9knjxo2Dw7SvtM/YX3mL9qfe+dyoUaPgMBWX6c007OvcQ6UVW7dutdq1aweHaf+FXnOladOmrs4wRaF5/HVTyB9UGTW1IpCiRYu6ejDx2r59u3u3rKYLVa5cOfcb/EX7RC+BLly4cNhw9lfe3NdlypRxAVso9nXucfjwYVcnTUXZXbp0Cdu32o+h9F3XdlU/KVGiRA6k1v8I1pBrjBw50rZs2RLzd+WwvPvuu3HPTzd9XSD0NJeQkBAcrhZMiYmJmU4vspb2V7S6aeyvvId9nbup5e4111zjGnB99dVXYdfTaPvWazXKdTc2gjXkGh988EGWzk/FKrqoqOm4Wi3J0aNHbdOmTa5VKPxF++vQoUOuyf/xxx/vhnn7j/2V9/a1cll27tzpcthE3eroYY197W86J3v37u3ez/3ll19azZo1U+zb9evXhw3Td12DI3PN8T/UWUO+pW4D9HLhjz76KDhMrZZ0g4jVnQByzrnnnuuevEP314wZM2zPnj3srzxGXenoxh26rz///HMXwLVr1y5H04bY9LCrok9dR2fOnBlWV82ja+t//vMfN65H+5lrburIWUOe7pNLT+IrV6503/WkJ6effrqr+1S6dGm76667bMiQIa4YNCkpyfX/o+48NA78pWzZsjZ06FAbNGiQu9CrvqG+6yle9WKQu7ruUA7pb7/95vald26eccYZ7jxUzungwYPtjjvucDlqeqjSvr7hhhvslFNOyenkI4b+/fvbpEmTbMyYMS7H2+vw9oQTTrCTTjrJ/X/nnXfav/71L7v66qvtqquussmTJ7trtfpoQ2wJahKayu9ArvXQQw+5Pn8ivfTSS9agQYPgd/UF9P7777sitgsuuMBuu+026k74lC5X3v7STVx9b916660Un+Qy9957r8t5iTR27NhgMKZ9re+6mWtfd+zY0e3rQoXIY/Ar9VMZrV7xZZddZn//+9+D31WX7fHHH3fBuopF9QBWv379Y5za3IVgDQAAwMeoswYAAOBjBGsAAAA+RrAGAADgYwRrAAAAPkawBgAA4GMEawAAAD5GsAYAAOBjBGs4ZtSx5dtvv21//vln2HB1jKiOL6dMmRIc9s0337iXsi9ZsiTP7qHVq1fbv//975xORq63ePHiYA/4fvTtt9/a119/bX6jNCltfubXbZdX98/06dNt7dq1KYbHOr/0kvbff//9GKQMdAWNTNPrRfTKGL2yqVixYla5cmU77bTT3P+hDhw44F4vootQuXLl3DC9YuTmm2+2tm3buhf+XnTRRe41JHPnzrWmTZvapZde6l5BkxfNnj3bvVKnc+fOOZ2UXO3111+3P/74w1q1amV+pFfv/PXXX9aiRQvzk1GjRrlXO5111lmZmo8erPS2gebNm1t+2XZ5cf/88MMPds0117hXP0V64oknop5fCuz0+ihd03X9R/YhWEOm6V2ajRs3du9+O3jwoHvSUm6Z3uM3cuRIK1GihBtPL2bu0aOHe++fR6+T0fv/HnjgAfd97969LoBbsGBBpi9SflejRg3r0qVLTicDyBS9vk255tkRrOmBTQ95yP79o/ck68G5ZMmSwWGHDx+2hQsX2vr1610Oth6cQ4MyPXzffffdrmSka9eu7KZsRLCGLNG3b1+76aabgt91Yuvk1UvUP/30UzesYMGCLjg57rjjXE6ccuT0jriqVau64lHRy51l3rx5LuBr3759MBdOT9ganpyc7C4aysHz6IKuC8aFF17oLix6UbTG8V4evGbNGpcmvQxcgaUXQIqWo/H17kE9XWp6TVulSpUU67lu3Tr77rvvrEKFCi6Y1DqFSm05kbTeSm9G0hE5jd6RqWmUviZNmtiJJ57oLrTaXgqAmzVrFtyO8aZXL2H23t+oXNI6derYqaeeGnX56Umz58cff3THR8WKFV36Qrdl6HrpqX3jxo3u2ElMTHT7etasWe7/Ro0axZz/999/b6tWrbJq1aq5NBUo8L9aH5qn3jGp91Dqf+U8aB1CKYdBaVDubqiff/7ZfTR86dKltmzZMje8TJkydvrpp7uXVqfmyy+/dNtb43p0TO3bty9F7kVq6xAvpVHbs27duu4TSceHjgEtX/uhdOnSYdOmtn6qpqD06Xz2zuF27dqFPZCFbm8tX9Ps2bPHWrdu7Y63Xbt2ueO0aNGidvbZZ7uXtnu0fxVoRM6nXr16MdMc7/ZVurVuOpe1HD1MRhPPMkXXJY27detWd1xp/NyyfzSOrtMvvvhicJjSpPNP+2PHjh3Wu3dvd+3We3nLly/vxtHx2KtXL3vhhRcI1rKbXuQOZEbBggUDr7zySorh06ZNC+gQmzVrlvu+Z88e9/3rr78OHDp0KNCjR49A6dKlA40aNXL/63PxxRe7cTp16uS+r1y50k370UcfBcqWLRto1apVoGPHjm66J598MrisjRs3Bqc7+eSTA927dw/MmDHD/fb3v/89cNxxxwUuuuiiQPPmzQMnnnhiYP78+cFpX3rppUD58uUDLVq0CJx77rmBNm3aBIoWLRr44IMPguMcPXo0cMcddwSKFSsWaN26tRu3ZcuWgZ07dwbHSWs5kd58881AxYoV05WOSN40p512WuC8885zyy1SpIjbHxrWtm3bwJlnnhk4/vjjA8uXLw+bNq30Ll26NGy/aPvrf22LzKT5yJEjbv+UKVPG7csqVaoEzjjjjMCmTZtSzLdp06Zun2u5u3fvDvzxxx+BmjVrBmrVqhXo0KFDoGrVqoFmzZoFOnfuHJxW+0TrXa1atcAll1wSqF27tlu/rVu3BsfR/Jo0aeLmozQMHjw4RToXLlzojqnff/89bPill14auOqqq9z/77zzTnAbtWvXLlC8ePHA008/HTb+jTfe6H73KG1Dhw4NG+eWW25J9zqkZfv27W4e2m9ax1NPPTXQr1+/FNugTp067ncdL5UrVw78+uuvwXHSWr8JEyYEatSoEahevXpwvBUrVqRIi4Y3bNjQrc+FF17otruOt/Hjx7vpdQzqb+PGjQMHDx6Mue3iSXM82/eee+4JlCpVyl0vdB5ruTq2oolnmevXr3fDtR00jubdq1evsHPFz/vnueeec8dYqN69e7vto3Xwtt1XX30VWL16ddh406dPDxQqVMidn8g+BGvItmBNN2VdPB588MEUwZqnfv36YReXVatWuXG8IE3Wrl0bSEpKcsGf5/vvv3dBgf6GBmu6qGi5nnHjxrmL1ZYtW4LDHnvssbALkwIDTfv2228Hhw0bNixQr1694PcxY8a4QO27774LDlNgo+XGu5x4grW00hHJm+bdd98NDtM20LCPP/44OEwX8j59+qRru0Tatm2bC6xC05eRNI8ePdoFatrX3nGhgP36669PMd/I4+ryyy93Qal3Q1dApeMv9Easm0yXLl3cA4HoeNBNRzd+j25aiYmJgWXLlgVSU7du3cDDDz8c/P7nn3+66f7zn/9EHV/HhH7XMZuZYC2edUiLAkrd4EMDvMmTJwf/V5p0XnnnWnJysgtcbr755pjzjLZ+1157rQtMUuMtywt89+/fH6hUqZI7DtasWeOG6Wavh4eJEyemGqyllea0tu/evXvdMaPAw/PTTz+leJhJzzKvvPJK98Cyb98+9/3nn392177XX389V+wfnXt6CAmlB7T+/fu7/0OPzUibN29252ro9kTWoxgU2UbFWsouV7FAZqi4VPUodu/e7VqIih40VHym4rDQ4g7VuQgtTnvttdfc7xrv/x9OXNGLil9VOVZFkaLsfdWn87Rp08aefPJJN77qaKgSu7L7Q4vdVCSR3uWkJa10RKMin+7duwe/qzK2WpFdfPHFwWGqr6IGDelNr4qgVGdlw4YNdujQITdc9QlD05jeNKs4RvUcVWdPVIl6wIABbt+NGzcuOJ7Sc+ONNwa/Ky0ffvihvffee64IVFTkq8YpHhWRav6DBg1yLW29dVMx4rRp08LSoenSKqrSPlcdStXLER1/Km664IILguPs3LnTFVeq2FjFTSoyVrFVPEXB0aRnHWJRkZm2k/ZzaJFXZGMWVTOoXbt2sEjrnHPOSdECMavWT9tMjYhERZ4q1tWx6x1rOsfr168ftYJ7etOcGl0fVOSpIkQVi+oYjSzeT88yVfypokEdJ16jKlUZuOKKK9x+VPGh3/ePqp9oX0RWbdF5rXmqGkKs65jOf28eyD4Ea8hWqmeWWr2teKilny6IuriFUgCigC1UaD02b1qlIXJaXYRUp8sTeaFSPQ39ruWqvorqdqXWGCDe5aQlrXRE410sQ6eJNiy0onY86VVdGNWp07SqS6P9qIv2li1bMpVmdVkSGkhKrVq13A1Mgb1XH0b7NjTYU8CogM0L8jwKADZt2hQcR0Gl6ihFdimgG11qx0qsYO3ee+91Nz8F6rohaxt56/Wvf/3L+vfv727Oujl66x65jdIjPesQi/aT0qF0pSbavgs9TrJy/TJynGYkzWnR+G+++aZria2GTao7p4eH1M7v1JapepraJl792NBjWg82uWH/6IFJdeNCXXLJJe64V8D5wQcfuBb+DRo0sAkTJoSdg950oQ0TkPUI1pBt9ISsPtUy2/VGqVKlXKDgVZBNTWROjqZVTs9TTz2VqTQoNyWyf7jsWM6xEk96FaQo5+Gtt94KDlOQpVyezFBOwvbt28OG6btyPLSdY+1Lr4GEKjuHCv2u9RLl0kU2DIgUT1cDCgRV6V1Bmpav/qbUjYHn1ltvdd/79OkTHKabVmrbSDkkyq0IFXoDTs86xOJVQk/tmI1HRtYvp6W1fUU50fqooch//vMfl/v12GOPuW2eXl7OWLRjOrIiv1/3j4K9Tz75JMVwNZQYMWKEC9oUGKrRmHJ8lZMY2jjBmweyD53iIlvo6W7IkCGuVWJGbzgetdLTBSGyCEhPdGpJlta0ushEjqen4fQW4ag4Vl2TeJQT5D1VZtVyjpV40qvcqtDWaXpaDy1KzSgFgCreC72hqnhRxcqxWuSJAnYF/mr161Grws8//zz4XTdHFY2OHj06xfQZ3RfKXdO2Um7MySef7LqTEB0LChR1Q/NMnTrV5VimRueEips9yi0M7XQ03nXQ/tEDjLZBJM1DrXvfeOONsOHpqZIQ7/opV8ZP3WuktX2Vfu+417oNHDjQ5SDPnz8/Q8srXry4y3VV7lPo9U/HeKy+//y2f1QlQEWpodOqZXkoFa/quIws7lSfmArUVEyP7EPOGrKE6lHooqDiGxXdvPPOO+4C8dFHH6XoHDe9dMG7/fbbrVu3bu5JUhcn5drp4qj5RzahD3XPPfe4m7m62VAdDN3wVTShJ2p1FxAv1VnSGxZU9Hr99de7C6Vu4Fq+5plVyzlW4kmvioWU26CbkYpXnn/++Qx1HRGtPycdHwoYlbuhTjtVHDtjxow0p1V6lLunQE/1zdRPX2Sa1JGq6vroo2NGQbXWVUU4//znP9OdXhV7qi9ALVsPIB5tE63DLbfc4o5PFW1pG6V1vKvjUXWJMGzYMFdUFu2tHvGsw6JFi1w/VzrfohVBqRsGTa/cEAUjKn5WsK1e5+MR7/qdeeaZLrdFwaVyRqN1DXEspbV9Faip6FPHt+rIabsoVyk0Bzm9lEOtZSpHS4Gbrg0KEkOPFz/vH11j9SCi8/C6665zw4YOHeoeRs8991wXuOm8ffbZZ+3ll18Om1bncmi3TcgeBGvINN3MdAHUk6QuIKoLpBubLkBeRfBYneJqnNAnQwUNGify5qPevFWHQhdVPSWrQrAubF6dNV2gNF1oMZrou4IBFWOpDySl7/zzzw+74KgCb2QdKvW9pPl5gYDqjiggVYVg1SXSOuoi5VXijWc5aXWKG086IkWbRtsztP820U0pNFcwnvTqRqM+m9RvlfajGg2oknFojlhG0qz9r2IVLUv7slKlSi7wCK3sH22+ohui+n5TvZmffvrJ7rvvPpe7EJojoad//TZ+/Hi3jkqP6ifpJuVR0aYqucdDxZ8K1pcvX56isrhyW3XT1Xqorp1e1/PKK6+4ICFWx67azp999pkrSlI6dVNUjlloMVo866CcIG0jr9J+JOVUqi87BbQKwLV9Q4u6om0D1UtK7/opOFJOkoJ95fIpWIkMBqItS/XvIutknXfeeWF1vyK3XTxpTmv7KudNadW21TVE+1fHlJdjGimeZapKga4PyilTTpNy4vv165fieuTX/aMqAXqAU5Hqtdde677r2qC+1/QQpVx1DVODJB2bHp3H6itRD3zIXglqEprNywAAZLEHH3zQBcWhDztAZigHWUFmZOtYPVSGVj8IfTuCHpiV+4vsRbAGAABiUq66ilKRcwjWAAAAfIzWoAAAAD5GsAYAAOBjBGsAAAA+RrAGAADgYwRrAAAAPkawBgAA4GMEawAAAD5GsAYAAOBjBGsAAAA+RrAGAABg/vV/zMgywBmM2nIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 700x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of simulated differences: 5000\n",
      "Largest simulated difference: $23.74\n",
      "Simulated differences at least as large as observed: 22\n"
     ]
    }
   ],
   "source": [
    "repetitions = 5000\n",
    "simulated_diffs = np.zeros(repetitions)\n",
    "for i in range(repetitions):\n",
    "    simulated_diffs[i] = diff_in_means(values, rng.permutation(chat_labels))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 4))\n",
    "ax.hist(simulated_diffs, bins=30, color='tab:gray', alpha=0.8)\n",
    "ax.axvline(observed_diff, color='tab:red', lw=2.5)\n",
    "ax.set_ylim(0, 600)\n",
    "ax.text(observed_diff - 0.8, 540, f'Observed: ${observed_diff:.2f}', color='tab:red', ha='right', fontsize=11)\n",
    "ax.set_xlabel('Difference in mean order value, chat minus no chat ($)')\n",
    "ax.set_ylabel('Number of shuffles')\n",
    "ax.set_title('If chat were unrelated to spending, a $15.92 gap would be rare', loc='left', fontsize=11)\n",
    "plt.show()\n",
    "\n",
    "print('Number of simulated differences:', len(simulated_diffs))\n",
    "print('Largest simulated difference: $' + str(round(simulated_diffs.max(), 2)))\n",
    "print('Simulated differences at least as large as observed:', np.count_nonzero(simulated_diffs >= observed_diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c21",
   "metadata": {},
   "source": [
    "The simulated differences cluster around zero, and only a few stretch past \\$15. The observed difference sits in the far right tail: 22 of the 5,000 shuffles reached it, so the estimated p-value is about 0.004."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c22",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.907197Z",
     "iopub.status.busy": "2026-10-24T04:20:03.907125Z",
     "iopub.status.idle": "2026-10-24T04:20:03.909361Z",
     "shell.execute_reply": "2026-10-24T04:20:03.908698Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated p-value: 0.0044\n"
     ]
    }
   ],
   "source": [
    "p_value = np.count_nonzero(simulated_diffs >= observed_diff) / len(simulated_diffs)\n",
    "print('Estimated p-value:', p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c23",
   "metadata": {},
   "source": [
    "A p-value of 0.004 is well below the usual 5% cutoff, so we reject the null hypothesis. Because the p-value comes from 5,000 shuffles, it is an estimate; a different seed would give a slightly different small number. (If no shuffle had reached the observed value, the right report would be \"fewer than 1 in 5,000\", not \"zero\", as in {doc}`1001-ab-testing`.)\n",
    "\n",
    "**What the test shows.** It rejects the null hypothesis that the chat label is unrelated to order value. Orders that used chat really do tend to be larger than orders that did not, and chance in the shuffling is not a believable explanation.\n",
    "\n",
    "**What the test does not show.** It does not say that chat made customers spend more. The null hypothesis describes a world where the labels could be swapped freely among customers. In an experiment, that is true by design under the null, because the researcher dealt out the labels. Here the customers dealt out their own labels, so rejecting the null tells us only that the groups differ. The test measures an association. The next part shows why that is not enough to justify the proposal.\n",
    "\n",
    "Notice also the denominator. The p-value is the count divided by the number of simulated results actually collected, which is why the code uses `len(simulated_diffs)` rather than a number typed in elsewhere. The next exercise practices exactly that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c24",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.911092Z",
     "iopub.status.busy": "2026-10-24T04:20:03.910843Z",
     "iopub.status.idle": "2026-10-24T04:20:03.915492Z",
     "shell.execute_reply": "2026-10-24T04:20:03.914652Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Use the Actual Number of Repetitions\n",
    "# In this small practice simulation, larger differences favor the alternative.\n",
    "# Compute the tail fraction using the number of results actually present.\n",
    "# Do not divide by a planned repetition count stored elsewhere.\n",
    "#   1. Count the simulated values that are at least as large as observed.\n",
    "#   2. Divide by the number of simulated values (use len).\n",
    "#   3. Print the result.\n",
    "# Print exactly:\n",
    "# Tail fraction: 0.4\n",
    "import numpy as np\n",
    "observed = 0.7\n",
    "simulated = np.array([0.1, 0.8, -0.2, 0.7, 0.3])\n",
    "### Your code starts here.\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c25",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.917438Z",
     "iopub.status.busy": "2026-10-24T04:20:03.917170Z",
     "iopub.status.idle": "2026-10-24T04:20:03.920798Z",
     "shell.execute_reply": "2026-10-24T04:20:03.920237Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tail fraction: 0.4\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import numpy as np\n",
    "observed = 0.7\n",
    "simulated = np.array([0.1, 0.8, -0.2, 0.7, 0.3])\n",
    "print('Tail fraction:', np.count_nonzero(simulated >= observed) / len(simulated))\n",
    "# np.mean(simulated >= observed) gives the same fraction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c26",
   "metadata": {},
   "source": [
    "## Confounding\n",
    "\n",
    "```{index} confounding\n",
    "```\n",
    "\n",
    "```{index} confounder\n",
    "```\n",
    "\n",
    "Marketing's hunch is that chat causes bigger orders. Another story fits the same data equally well: some kinds of customers both use chat more and spend more, regardless of chat. One candidate is whether the customer has ordered before. Returning customers know the store, may buy more per visit, and may also be more comfortable asking a question. The exercise above already hinted at this. Here are the numbers in one table: for each combination of customer type and chat use, the number of orders and the mean order value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.922499Z",
     "iopub.status.busy": "2026-10-24T04:20:03.922424Z",
     "iopub.status.idle": "2026-10-24T04:20:03.927201Z",
     "shell.execute_reply": "2026-10-24T04:20:03.926837Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>orders</th>\n",
       "      <th>mean_order_value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>returning_customer</th>\n",
       "      <th>used_chat</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>158</td>\n",
       "      <td>59.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15</td>\n",
       "      <td>64.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>0</th>\n",
       "      <td>80</td>\n",
       "      <td>90.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>47</td>\n",
       "      <td>92.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              orders  mean_order_value\n",
       "returning_customer used_chat                          \n",
       "0                  0             158             59.29\n",
       "                   1              15             64.03\n",
       "1                  0              80             90.20\n",
       "                   1              47             92.49"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = orders.groupby(['returning_customer', 'used_chat'])['order_value'].agg(['count', 'mean']).round(2)\n",
    "summary.columns = ['orders', 'mean_order_value']\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c28",
   "metadata": {},
   "source": [
    "Two patterns stand out. First, the groups are not alike. Among new customers, 15 of 173 orders used chat. Among returning customers, 47 of 127 did. Second, returning customers spend much more whether or not they chat.\n",
    "\n",
    "Check how the chat group is made up compared with the no-chat group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c29",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.933109Z",
     "iopub.status.busy": "2026-10-24T04:20:03.932874Z",
     "iopub.status.idle": "2026-10-24T04:20:03.941649Z",
     "shell.execute_reply": "2026-10-24T04:20:03.940827Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Share of no-chat orders from returning customers: 0.34\n",
      "Share of chat orders from returning customers:    0.76\n",
      "\n",
      "Share of new customers who used chat:       0.09\n",
      "Share of returning customers who used chat: 0.37\n"
     ]
    }
   ],
   "source": [
    "returning_share = orders.groupby('used_chat')['returning_customer'].mean()\n",
    "print('Share of no-chat orders from returning customers:', round(returning_share[0], 2))\n",
    "print('Share of chat orders from returning customers:   ', round(returning_share[1], 2))\n",
    "\n",
    "chat_rate = orders.groupby('returning_customer')['used_chat'].mean()\n",
    "print()\n",
    "print('Share of new customers who used chat:      ', round(chat_rate[0], 2))\n",
    "print('Share of returning customers who used chat:', round(chat_rate[1], 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c30",
   "metadata": {},
   "source": [
    "About three-quarters of the chat orders came from returning customers, compared with about one-third of the no-chat orders. Returning customers used chat 37% of the time, new customers only 9%. So the \\$15.92 gap compares a chat group full of high spenders with a no-chat group that has far fewer of them. Part of the gap, perhaps most of it, comes from who the customers are rather than from chat.\n",
    "\n",
    "To compare like with like, compute the chat gap separately within each type of customer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.943207Z",
     "iopub.status.busy": "2026-10-24T04:20:03.943110Z",
     "iopub.status.idle": "2026-10-24T04:20:03.950144Z",
     "shell.execute_reply": "2026-10-24T04:20:03.949509Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chat gap among new customers:       $4.73\n",
      "Chat gap among returning customers: $2.28\n",
      "Overall chat gap:                   $15.92\n"
     ]
    }
   ],
   "source": [
    "means = orders.groupby(['returning_customer', 'used_chat'])['order_value'].mean().unstack()\n",
    "gap_new = means.loc[0, 1] - means.loc[0, 0]\n",
    "gap_returning = means.loc[1, 1] - means.loc[1, 0]\n",
    "\n",
    "print('Chat gap among new customers:       $' + str(round(gap_new, 2)))\n",
    "print('Chat gap among returning customers: $' + str(round(gap_returning, 2)))\n",
    "print('Overall chat gap:                   $' + str(round(observed_diff, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c32",
   "metadata": {},
   "source": [
    "Within each type of customer, the chat gap is much smaller: about \\$4.73 among new customers and about \\$2.28 among returning customers, compared with \\$15.92 overall. Most of the overall gap came from the mix of customers, not from chat.\n",
    "\n",
    "This is the pattern called confounding. A **confounder** is a variable that is related to both the group a unit is in and the outcome, and so produces or hides a difference between the groups that the group itself does not cause. Here `returning_customer` is a confounder: it is related to chat use (returning customers chat more) and to order value (returning customers spend more).\n",
    "\n",
    "The small gaps that remain do not prove that chat has a small positive effect either. With only 15 new customers who chatted, the \\$4.73 could easily be chance, and other differences between customers who chat and customers who do not (their budgets, their interest in the product) are not in the data at all.\n",
    "\n",
    "```{index} opt-in comparison\n",
    "```\n",
    "\n",
    "Chat is an example of an **opt-in comparison**: the customers who opt in to a feature are compared with those who do not. Opt-in groups almost always differ from the rest in ways that also affect the outcome. People who join a loyalty program, open a marketing email, or try a beta feature tend to be the more engaged customers to begin with. A raw comparison between opt-in users and everyone else therefore overstates what the feature does. This is why a statement such as \"customers who try the new design buy more, so the design works\" cannot be accepted from the comparison alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c33",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.952281Z",
     "iopub.status.busy": "2026-10-24T04:20:03.952129Z",
     "iopub.status.idle": "2026-10-24T04:20:03.955716Z",
     "shell.execute_reply": "2026-10-24T04:20:03.955332Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Compute the Within-Group Gaps\n",
    "# A store compares customers who joined its loyalty program with those who did not.\n",
    "# 'segment' is the customer's usual shopping segment (Home or Office).\n",
    "#   1. Use groupby on ['segment', 'member'] to compute the mean spend of each cell.\n",
    "#   2. Compute the member gap (members minus non-members) within each segment.\n",
    "#   3. Compute the overall member gap, ignoring segment.\n",
    "#   4. Print the three gaps rounded to 1 decimal. In a comment, say whether\n",
    "#      segment looks like a confounder.\n",
    "# Print exactly:\n",
    "# Gap in Home segment: 5.0\n",
    "# Gap in Office segment: 5.0\n",
    "# Overall gap: 15.7\n",
    "import pandas as pd\n",
    "shoppers = pd.DataFrame({\n",
    "    'segment': ['Home'] * 8 + ['Office'] * 8,\n",
    "    'member': [0, 0, 0, 0, 0, 1, 1, 1] + [0, 1, 1, 1, 1, 1, 1, 1],\n",
    "    'spend': [20, 22, 18, 20, 20, 25, 27, 23] + [40, 45, 47, 43, 45, 45, 47, 43],\n",
    "})\n",
    "### Your code starts here.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c34",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.957745Z",
     "iopub.status.busy": "2026-10-24T04:20:03.957581Z",
     "iopub.status.idle": "2026-10-24T04:20:03.967259Z",
     "shell.execute_reply": "2026-10-24T04:20:03.966424Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gap in Home segment: 5.0\n",
      "Gap in Office segment: 5.0\n",
      "Overall gap: 15.7\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import pandas as pd\n",
    "shoppers = pd.DataFrame({\n",
    "    'segment': ['Home'] * 8 + ['Office'] * 8,\n",
    "    'member': [0, 0, 0, 0, 0, 1, 1, 1] + [0, 1, 1, 1, 1, 1, 1, 1],\n",
    "    'spend': [20, 22, 18, 20, 20, 25, 27, 23] + [40, 45, 47, 43, 45, 45, 47, 43],\n",
    "})\n",
    "\n",
    "means = shoppers.groupby(['segment', 'member'])['spend'].mean().unstack()\n",
    "for segment in ['Home', 'Office']:\n",
    "    print(f'Gap in {segment} segment:', round(means.loc[segment, 1] - means.loc[segment, 0], 1))\n",
    "\n",
    "overall = shoppers.groupby('member')['spend'].mean()\n",
    "print('Overall gap:', round(overall[1] - overall[0], 1))\n",
    "# Segment is a confounder: Office shoppers spend more and are far more likely\n",
    "# to be members, so the overall gap is much larger than the gap within a segment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c35",
   "metadata": {},
   "source": [
    "## What Observational Data Can Support\n",
    "\n",
    "```{index} observational data; what it supports\n",
    "```\n",
    "\n",
    "None of this means observational data is useless. It means each claim needs the right kind of evidence.\n",
    "\n",
    "| Claim | Can observational data support it? |\n",
    "|---|---|\n",
    "| \"Chat users spend more than other customers.\" | Yes. This is a description of the data, and the permutation test shows it is not a fluke. |\n",
    "| \"If a customer uses chat, we should expect a larger order.\" | Yes, as a prediction. Knowing that an order used chat helps predict its size, even if chat is not the reason. |\n",
    "| \"Customers who open chat are good targets for upselling.\" | Yes. Targeting needs only that the group is different, not that chat made it so. |\n",
    "| \"Chat raises order value, so putting it on every page will raise sales.\" | Not from this comparison. This is a causal claim about what would happen if the store changed what customers see. |\n",
    "\n",
    "A causal claim is a claim about a change: what would happen to the same customers if the store acted differently. To support it, you need one of two things.\n",
    "\n",
    "1. **Random assignment.** The store shows the chat button to a randomly chosen half of visitors and hides it from the rest. Then returning customers, big spenders, and every other trait are split by chance between the groups, so a difference in order value can be credited to chat. This is the design of {doc}`1001-ab-testing`.\n",
    "2. **Careful adjustment.** Compare like with like, as the table above did for new and returning customers. This helps only for the confounders you thought of and recorded. A trait that was not measured can still distort the comparison.\n",
    "\n",
    "For a decision this size, the cheaper path is usually to run the experiment: turn chat on for a random half of sessions for a few weeks and compare order value. {doc}`1003-causality` explains why random assignment licenses a causal claim and how to read the result. For another way to ask how much an estimate could vary, see {doc}`../11-estimation/1100-estimation`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36",
   "metadata": {},
   "source": [
    "## Section Practice\n",
    "\n",
    "```{index} exercises; comparing two groups\n",
    "```\n",
    "\n",
    "Practice comparing two groups with `groupby` before using a permutation test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c37",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.970369Z",
     "iopub.status.busy": "2026-10-24T04:20:03.970115Z",
     "iopub.status.idle": "2026-10-24T04:20:03.973498Z",
     "shell.execute_reply": "2026-10-24T04:20:03.972896Z"
    },
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Practice Comparing Two Groups\n",
    "# A warehouse weighs packages from two suppliers. Each value is the drop in weight\n",
    "# (grams) from the label weight.\n",
    "#   1. Use groupby to compute the mean drop for each supplier.\n",
    "#   2. Compute the test statistic: mean drop of supplier B minus supplier A.\n",
    "#   3. Print the two means and the difference, rounded to 2 decimals.\n",
    "# Print exactly:\n",
    "# Supplier A: 1.12\n",
    "# Supplier B: 2.06\n",
    "# Difference (B - A): 0.94\n",
    "import pandas as pd\n",
    "\n",
    "packages = pd.DataFrame({\n",
    "    'supplier': ['A'] * 6 + ['B'] * 5,\n",
    "    'drop': [1.2, 0.8, 1.5, 1.0, 0.9, 1.3, 2.1, 1.9, 2.4, 1.7, 2.2],\n",
    "})\n",
    "### Your code starts here\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c38",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-24T04:20:03.975102Z",
     "iopub.status.busy": "2026-10-24T04:20:03.975008Z",
     "iopub.status.idle": "2026-10-24T04:20:03.978683Z",
     "shell.execute_reply": "2026-10-24T04:20:03.978019Z"
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Supplier A: 1.12\n",
      "Supplier B: 2.06\n",
      "Difference (B - A): 0.94\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import pandas as pd\n",
    "\n",
    "packages = pd.DataFrame({\n",
    "    'supplier': ['A'] * 6 + ['B'] * 5,\n",
    "    'drop': [1.2, 0.8, 1.5, 1.0, 0.9, 1.3, 2.1, 1.9, 2.4, 1.7, 2.2],\n",
    "})\n",
    "\n",
    "means = packages.groupby('supplier')['drop'].mean()\n",
    "\n",
    "print('Supplier A:', round(means['A'], 2))\n",
    "print('Supplier B:', round(means['B'], 2))\n",
    "print('Difference (B - A):', round(means['B'] - means['A'], 2))\n",
    "# A permutation test, as in this section, would tell whether a difference this\n",
    "# large could be due to chance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}