{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lab\n",
"\n",
"- Use this assignment to practice the chapter's concepts and coding skills in runnable Python code.\n",
"- Sign in from the account menu before you start working on the assignment.\n",
"- Write and run your code directly in the coding cells.\n",
"- Multiple submissions are allowed. The highest score is kept.\n",
"\n",
"\n",
"\n",
"Follow one store's loyalty-program investigation. First assess the membership rate, then the regional mix, then the decisions made from test results. The simulation uses a fixed seed so everyone can check the same output. Twenty repetitions keep the arithmetic visible here; an actual analysis would use many more, especially near a cutoff.\n",
"\n",
"For each result, identify the model being tested and the direction of evidence. A result about the membership rate does not automatically answer a separate question about regional representation.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"thebe-interactive"
]
},
"outputs": [],
"source": [
"### Question 1: Observed Test Statistic\n",
"# A store's model says 20% of customers join its loyalty program.\n",
"# In a random sample of 400 customers, 98 joined.\n",
"# 1. Compute the observed percent of program members.\n",
"# 2. Compute the test statistic: the distance between the observed percent\n",
"# and the model's 20%, | observed percent - 20 |, rounded to 2 decimals.\n",
"# Print exactly:\n",
"# Observed percent: 24.5\n",
"# Test statistic: 4.5\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"members = 98\n",
"sample_size = 400\n",
"model_percent = 20\n",
"### Your code starts here.\n",
"\n",
"\n",
"\n",
"### Your code ends here.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"tags": [
"hide-input",
"lab-answer"
]
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "Observed percent: 24.5\nTest statistic: 4.5\n"
}
],
"source": [
"### Solution\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"members = 98\n",
"sample_size = 400\n",
"model_percent = 20\n",
"\n",
"observed_percent = round(100 * members / sample_size, 2)\n",
"test_statistic = round(abs(observed_percent - model_percent), 2)\n",
"\n",
"print(\"Observed percent:\", observed_percent)\n",
"print(\"Test statistic:\", test_statistic)\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"tags": [
"thebe-interactive"
]
},
"outputs": [],
"source": [
"### Question 2: Total Variation Distance\n",
"# A loyalty program should reflect the store's customer base. The table shows the\n",
"# share of all customers in each region and the share in a separate sample\n",
"# of 100 program members.\n",
"# 1. Compute the absolute difference between the two distributions for each region.\n",
"# 2. Compute the total variation distance (TVD): the sum of the absolute\n",
"# differences divided by 2, rounded to 2 decimals.\n",
"# Print exactly:\n",
"# TVD: 0.1\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"regions = pd.DataFrame({\n",
" \"region\": [\"North\", \"Central\", \"South\"],\n",
" \"all_customers\": [0.40, 0.35, 0.25],\n",
" \"members\": [0.30, 0.40, 0.30],\n",
"})\n",
"### Your code starts here.\n",
"\n",
"\n",
"\n",
"### Your code ends here.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"tags": [
"hide-input",
"lab-answer"
]
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "TVD: 0.1\n"
}
],
"source": [
"### Solution\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"regions = pd.DataFrame({\n",
" \"region\": [\"North\", \"Central\", \"South\"],\n",
" \"all_customers\": [0.40, 0.35, 0.25],\n",
" \"members\": [0.30, 0.40, 0.30],\n",
"})\n",
"\n",
"abs_diff = (regions['members'] - regions['all_customers']).abs()\n",
"tvd = round(abs_diff.sum() / 2, 2)\n",
"\n",
"print(\"TVD:\", tvd)\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"tags": [
"thebe-interactive"
]
},
"outputs": [],
"source": [
"### Question 3: P-value From a Simulation\n",
"# Continue the regional-mix investigation from Question 2.\n",
"# Null: members are selected independently with the population's regional shares.\n",
"# Large TVDs are evidence against that model. Simulate 20 samples of 100 members.\n",
"# 1. Start an empty list. In each repetition, generate new counts with\n",
"# rng.multinomial(sample_size, population).\n",
"# 2. Compute each TVD as the sum of absolute differences between simulated\n",
"# counts and expected_counts, divided by (2 * sample_size). Append it.\n",
"# This is the same TVD formula, expressed in counts rather than proportions.\n",
"# 3. Convert the list to a pandas Series. Count TVDs >= observed_tvd,\n",
"# then divide that count by the number of simulations for the p-value.\n",
"# Print exactly:\n",
"# At least as large: 4\n",
"# P-value: 0.2\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"rng = np.random.default_rng(17)\n",
"sample_size = 100\n",
"population = np.array([0.40, 0.35, 0.25])\n",
"expected_counts = sample_size * population\n",
"observed_tvd = 0.10\n",
"repetitions = 20\n",
"### Your code starts here.\n",
"\n",
"\n",
"\n",
"### Your code ends here.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"tags": [
"hide-input",
"lab-answer"
]
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "At least as large: 4\nP-value: 0.2\n"
}
],
"source": [
"### Solution\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"rng = np.random.default_rng(17)\n",
"sample_size = 100\n",
"population = np.array([0.40, 0.35, 0.25])\n",
"expected_counts = sample_size * population\n",
"observed_tvd = 0.10\n",
"repetitions = 20\n",
"\n",
"simulated_tvds = []\n",
"for _ in range(repetitions):\n",
" counts = rng.multinomial(sample_size, population)\n",
" tvd = np.abs(counts - expected_counts).sum() / (2 * sample_size)\n",
" simulated_tvds.append(tvd)\n",
"simulated_tvds = pd.Series(simulated_tvds)\n",
"at_least_as_large = (simulated_tvds >= observed_tvd).sum()\n",
"p_value = at_least_as_large / len(simulated_tvds)\n",
"print('At least as large:', at_least_as_large)\n",
"print('P-value:', p_value)\n",
"# At a preselected 5% level, do not reject this regional-mix model.\n",
"# This is insufficient evidence against the model, not proof of random selection.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"tags": [
"thebe-interactive"
]
},
"outputs": [],
"source": [
"### Question 4: Applying the Conventions\n",
"# The same store tests three policies intended to support the loyalty program.\n",
"# Apply these conventional evidence labels, then consider the business question below:\n",
"# p-value below 0.01 -> \"highly significant\"\n",
"# p-value below 0.05 -> \"significant\"\n",
"# otherwise -> \"not significant\"\n",
"# Before seeing results, the team selected a 5% significance level.\n",
"# A label summarizes evidence against the null, not the size or value of a gain.\n",
"# For discussion: free shipping adds an estimated $200 monthly contribution\n",
"# before its $1,000 monthly operating cost. Does p = 0.003 justify rollout?\n",
"# Explain in a comment; print one line per test, in table order.\n",
"# Print exactly:\n",
"# Free shipping: highly significant\n",
"# Loyalty email: significant\n",
"# New layout: not significant\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"tests = pd.DataFrame({\n",
" \"policy\": [\"Free shipping\", \"Loyalty email\", \"New layout\"],\n",
" \"p_value\": [0.003, 0.04, 0.21],\n",
"})\n",
"### Your code starts here.\n",
"\n",
"\n",
"\n",
"### Your code ends here.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"tags": [
"hide-input",
"lab-answer"
]
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "Free shipping: highly significant\nLoyalty email: significant\nNew layout: not significant\n"
}
],
"source": [
"### Solution\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"tests = pd.DataFrame({\n",
" \"policy\": [\"Free shipping\", \"Loyalty email\", \"New layout\"],\n",
" \"p_value\": [0.003, 0.04, 0.21],\n",
"})\n",
"\n",
"for policy, p_value in zip(tests['policy'], tests['p_value']):\n",
" if p_value < 0.01:\n",
" label = \"highly significant\"\n",
" elif p_value < 0.05:\n",
" label = \"significant\"\n",
" else:\n",
" label = \"not significant\"\n",
" print(policy + \":\", label)\n",
"\n",
"# Free shipping has strong statistical evidence, but its estimated net monthly\n",
"# contribution is $200 - $1,000 = -$800. The p-value alone does not justify rollout.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"tags": [
"thebe-interactive"
]
},
"outputs": [],
"source": [
"### Question 5: False Alarms\n",
"# The store checks its test process with 20 A/A tests of its loyalty signup page.\n",
"# Each compares the page with an identical copy of itself,\n",
"# so the null hypothesis (no difference) is true in every test.\n",
"# 1. Count the tests whose p-value is below the 0.05 cutoff. Each one is a\n",
"# Type I error (a false alarm).\n",
"# 2. Compute the false-alarm rate: that count divided by the number of tests.\n",
"# Print exactly:\n",
"# False alarms: 1\n",
"# False-alarm rate: 0.05\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"p_values = pd.Series([0.280, 0.783, 0.988, 0.986, 0.883, 0.913, 0.708, 0.554, 0.923, 0.031,\n",
" 0.370, 0.606, 0.478, 0.793, 0.195, 0.062, 0.127, 0.963, 0.133, 0.226])\n",
"### Your code starts here.\n",
"\n",
"\n",
"\n",
"### Your code ends here.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"tags": [
"hide-input",
"lab-answer"
]
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "False alarms: 1\nFalse-alarm rate: 0.05\n"
}
],
"source": [
"### Solution\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"p_values = pd.Series([0.280, 0.783, 0.988, 0.986, 0.883, 0.913, 0.708, 0.554, 0.923, 0.031,\n",
" 0.370, 0.606, 0.478, 0.793, 0.195, 0.062, 0.127, 0.963, 0.133, 0.226])\n",
"\n",
"false_alarms = (p_values < 0.05).sum()\n",
"\n",
"print(\"False alarms:\", false_alarms)\n",
"print(\"False-alarm rate:\", false_alarms / len(p_values))\n",
"# With a 5% cutoff, about 5% of tests of a true null hypothesis come out significant.\n",
"\n",
"# One false alarm is an observed result in this batch, not a guarantee that\n",
"# every batch of 20 will contain exactly one false alarm.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"