Lab

9.4.2. Lab#

  • Use this assignment to practice the chapter’s concepts and coding skills in runnable Python code.

  • Sign in from the account menu before you start working on the assignment.

  • Write and run your code directly in the coding cells.

  • Multiple submissions are allowed. The highest score is kept.

Follow one store’s loyalty-program investigation. First assess the membership rate, then the regional mix, then the decisions made from test results. The simulation uses a fixed seed so everyone can check the same output. Twenty repetitions keep the arithmetic visible here; an actual analysis would use many more, especially near a cutoff.

For each result, identify the model being tested and the direction of evidence. A result about the membership rate does not automatically answer a separate question about regional representation.

### Question 1: Observed Test Statistic
# A store's model says 20% of customers join its loyalty program.
# In a random sample of 400 customers, 98 joined.
#   1. Compute the observed percent of program members.
#   2. Compute the test statistic: the distance between the observed percent
#      and the model's 20%, | observed percent - 20 |, rounded to 2 decimals.
# Print exactly:
# Observed percent: 24.5
# Test statistic: 4.5
import numpy as np
import pandas as pd

members = 98
sample_size = 400
model_percent = 20
### Your code starts here.



### Your code ends here.

Hide code cell source

 
### Solution
import numpy as np
import pandas as pd

members = 98
sample_size = 400
model_percent = 20

observed_percent = round(100 * members / sample_size, 2)
test_statistic = round(abs(observed_percent - model_percent), 2)

print("Observed percent:", observed_percent)
print("Test statistic:", test_statistic)

Observed percent: 24.5
Test statistic: 4.5

### Question 2: Total Variation Distance
# A loyalty program should reflect the store's customer base. The table shows the
# share of all customers in each region and the share in a separate sample
# of 100 program members.
#   1. Compute the absolute difference between the two distributions for each region.
#   2. Compute the total variation distance (TVD): the sum of the absolute
#      differences divided by 2, rounded to 2 decimals.
# Print exactly:
# TVD: 0.1
import numpy as np
import pandas as pd

regions = pd.DataFrame({
    "region": ["North", "Central", "South"],
    "all_customers": [0.40, 0.35, 0.25],
    "members": [0.30, 0.40, 0.30],
})
### Your code starts here.



### Your code ends here.

Hide code cell source

 
### Solution
import numpy as np
import pandas as pd

regions = pd.DataFrame({
    "region": ["North", "Central", "South"],
    "all_customers": [0.40, 0.35, 0.25],
    "members": [0.30, 0.40, 0.30],
})

abs_diff = (regions['members'] - regions['all_customers']).abs()
tvd = round(abs_diff.sum() / 2, 2)

print("TVD:", tvd)

TVD: 0.1

### Question 3: P-value From a Simulation
# Continue the regional-mix investigation from Question 2.
# Null: members are selected independently with the population's regional shares.
# Large TVDs are evidence against that model. Simulate 20 samples of 100 members.
#   1. Start an empty list. In each repetition, generate new counts with
#      rng.multinomial(sample_size, population).
#   2. Compute each TVD as the sum of absolute differences between simulated
#      counts and expected_counts, divided by (2 * sample_size). Append it.
#      This is the same TVD formula, expressed in counts rather than proportions.
#   3. Convert the list to a pandas Series. Count TVDs >= observed_tvd,
#      then divide that count by the number of simulations for the p-value.
# Print exactly:
# At least as large: 4
# P-value: 0.2
import numpy as np
import pandas as pd

rng = np.random.default_rng(17)
sample_size = 100
population = np.array([0.40, 0.35, 0.25])
expected_counts = sample_size * population
observed_tvd = 0.10
repetitions = 20
### Your code starts here.



### Your code ends here.

Hide code cell source

 
### Solution
import numpy as np
import pandas as pd

rng = np.random.default_rng(17)
sample_size = 100
population = np.array([0.40, 0.35, 0.25])
expected_counts = sample_size * population
observed_tvd = 0.10
repetitions = 20

simulated_tvds = []
for _ in range(repetitions):
    counts = rng.multinomial(sample_size, population)
    tvd = np.abs(counts - expected_counts).sum() / (2 * sample_size)
    simulated_tvds.append(tvd)
simulated_tvds = pd.Series(simulated_tvds)
at_least_as_large = (simulated_tvds >= observed_tvd).sum()
p_value = at_least_as_large / len(simulated_tvds)
print('At least as large:', at_least_as_large)
print('P-value:', p_value)
# At a preselected 5% level, do not reject this regional-mix model.
# This is insufficient evidence against the model, not proof of random selection.

At least as large: 4
P-value: 0.2

### Question 4: Applying the Conventions
# The same store tests three policies intended to support the loyalty program.
# Apply these conventional evidence labels, then consider the business question below:
#   p-value below 0.01 -> "highly significant"
#   p-value below 0.05 -> "significant"
#   otherwise          -> "not significant"
# Before seeing results, the team selected a 5% significance level.
# A label summarizes evidence against the null, not the size or value of a gain.
# For discussion: free shipping adds an estimated $200 monthly contribution
# before its $1,000 monthly operating cost. Does p = 0.003 justify rollout?
# Explain in a comment; print one line per test, in table order.
# Print exactly:
# Free shipping: highly significant
# Loyalty email: significant
# New layout: not significant
import numpy as np
import pandas as pd

tests = pd.DataFrame({
    "policy": ["Free shipping", "Loyalty email", "New layout"],
    "p_value": [0.003, 0.04, 0.21],
})
### Your code starts here.



### Your code ends here.

Hide code cell source

 
### Solution
import numpy as np
import pandas as pd

tests = pd.DataFrame({
    "policy": ["Free shipping", "Loyalty email", "New layout"],
    "p_value": [0.003, 0.04, 0.21],
})

for policy, p_value in zip(tests['policy'], tests['p_value']):
    if p_value < 0.01:
        label = "highly significant"
    elif p_value < 0.05:
        label = "significant"
    else:
        label = "not significant"
    print(policy + ":", label)

# Free shipping has strong statistical evidence, but its estimated net monthly
# contribution is $200 - $1,000 = -$800. The p-value alone does not justify rollout.

Free shipping: highly significant
Loyalty email: significant
New layout: not significant

### Question 5: False Alarms
# The store checks its test process with 20 A/A tests of its loyalty signup page.
# Each compares the page with an identical copy of itself,
# so the null hypothesis (no difference) is true in every test.
#   1. Count the tests whose p-value is below the 0.05 cutoff. Each one is a
#      Type I error (a false alarm).
#   2. Compute the false-alarm rate: that count divided by the number of tests.
# Print exactly:
# False alarms: 1
# False-alarm rate: 0.05
import numpy as np
import pandas as pd

p_values = pd.Series([0.280, 0.783, 0.988, 0.986, 0.883, 0.913, 0.708, 0.554, 0.923, 0.031,
                      0.370, 0.606, 0.478, 0.793, 0.195, 0.062, 0.127, 0.963, 0.133, 0.226])
### Your code starts here.



### Your code ends here.

Hide code cell source

 
### Solution
import numpy as np
import pandas as pd

p_values = pd.Series([0.280, 0.783, 0.988, 0.986, 0.883, 0.913, 0.708, 0.554, 0.923, 0.031,
                      0.370, 0.606, 0.478, 0.793, 0.195, 0.062, 0.127, 0.963, 0.133, 0.226])

false_alarms = (p_values < 0.05).sum()

print("False alarms:", false_alarms)
print("False-alarm rate:", false_alarms / len(p_values))
# With a 5% cutoff, about 5% of tests of a true null hypothesis come out significant.

# One false alarm is an observed result in this batch, not a guarantee that
# every batch of 20 will contain exactly one false alarm.

False alarms: 1
False-alarm rate: 0.05