10.4.2. Lab

• Use this assignment to practice the chapter’s concepts and coding skills in runnable Python code.

• Sign in from the account menu before you start working on the assignment.

• Write and run your code directly in the coding cells.

• Multiple submissions are allowed. The highest score is kept.

Follow an online store’s experiment from group summaries to a simulated comparison and a qualified business conclusion. Choose the direction of the test and significance level before examining results. Each question includes its own setup.

The seeded simulations make your results reproducible. Use unrounded values for comparisons and decisions. Question 4 retains a two-decimal display for its submitted output: a displayed 0.0 is a rounded estimate, not a claim that the true probability is zero. In an analysis report, retain more decimal places and state the repetition count.

### Question 1: Comparing Two Groups
# An online store showed checkout page A or B to customers at random and recorded
# each order's value in dollars.
#   1. Use groupby to compute the mean order value for each version.
#      Keep full precision for calculations; round to 2 decimals when printing.
#   2. Compute the observed test statistic: mean of B minus mean of A, rounded to 2 decimals.
# Print exactly:
# Mean A: 41.67
# Mean B: 48.17
# Observed difference: 6.5
import numpy as np
import pandas as pd

ab = pd.DataFrame({
    "version": ["A", "B", "A", "B", "A", "B", "A", "B", "A", "B", "A", "B"],
    "order_value": [42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47],
})
### Your code starts here.



### Your code ends here.

Show code cell source

Hide code cell source

### Solution
import numpy as np
import pandas as pd

ab = pd.DataFrame({
    "version": ["A", "B", "A", "B", "A", "B", "A", "B", "A", "B", "A", "B"],
    "order_value": [42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47],
})

means = ab.groupby('version')['order_value'].mean()
observed = round(means['B'] - means['A'], 2)

print("Mean A:", round(means['A'], 2))
print("Mean B:", round(means['B'], 2))
print("Observed difference:", observed)

Mean A: 41.67
Mean B: 48.17
Observed difference: 6.5

### Question 2: Conversion Rates
# A second test tracked whether each visitor bought something (1) or not (0).
#   1. Use groupby to compute the conversion rate (mean of converted) for each version.
#   2. Compute the difference in rates, B minus A, rounded to 2 decimals.
# Print exactly:
# Rate A: 0.2
# Rate B: 0.3
# Difference: 0.1
import numpy as np
import pandas as pd

visits = pd.DataFrame({
    "version": ["A"] * 10 + ["B"] * 10,
    "converted": [0, 1, 0, 0, 0, 1, 0, 0, 0, 0,
                  1, 0, 0, 1, 0, 0, 1, 0, 0, 0],
})
### Your code starts here.



### Your code ends here.

Show code cell source

Hide code cell source

### Solution
import numpy as np
import pandas as pd

visits = pd.DataFrame({
    "version": ["A"] * 10 + ["B"] * 10,
    "converted": [0, 1, 0, 0, 0, 1, 0, 0, 0, 0,
                  1, 0, 0, 1, 0, 0, 1, 0, 0, 0],
})

rates = visits.groupby('version')['converted'].mean()

print("Rate A:", rates['A'])
print("Rate B:", rates['B'])
print("Difference:", round(rates['B'] - rates['A'], 2))

Rate A: 0.2
Rate B: 0.3
Difference: 0.1

### Question 3: One Shuffle of the Labels
# Continue the order-value comparison from Question 1.
# Under the null model, labels can be rearranged while outcomes and group sizes stay fixed.
#   1. Use rng.permutation(ab['version'].to_numpy()) to generate shuffled labels.
#   2. Add them as a new column named 'shuffled'.
#   3. Compute the difference in means, B minus A, using the shuffled labels,
#      rounded to 2 decimals for printing.
# Print exactly:
# Shuffled difference: -3.17
import numpy as np
import pandas as pd

ab = pd.DataFrame({
    "version": ["A", "B", "A", "B", "A", "B", "A", "B", "A", "B", "A", "B"],
    "order_value": [42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47],
})
rng = np.random.default_rng(10)
### Your code starts here.



### Your code ends here.

Show code cell source

Hide code cell source

### Solution
import numpy as np
import pandas as pd

ab = pd.DataFrame({
    "version": ["A", "B", "A", "B", "A", "B", "A", "B", "A", "B", "A", "B"],
    "order_value": [42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47],
})
rng = np.random.default_rng(10)
shuffled_labels = rng.permutation(ab['version'].to_numpy())

ab['shuffled'] = shuffled_labels
shuffled_means = ab.groupby('shuffled')['order_value'].mean()

print("Shuffled difference:", round(shuffled_means['B'] - shuffled_means['A'], 2))

Shuffled difference: -3.17

### Question 4: Permutation Test Decision
# Complete the store's test. The alternative says B raises order values.
# Equivalently to shuffling six A and six B labels, randomly rearrange the values
# and assign the first six to B and the remaining six to A.
#   1. Start an empty list. Repeat repetitions times: use rng.permutation(values),
#      compute (sum of first six - sum of remaining six) / group_size, and append it.
#   2. Convert the list to a pandas Series. Compute the proportion >= observed.
#   3. Compare the UNROUNDED estimate with the preselected 5% cutoff.
#      Print the estimate rounded to 2 decimals to match this exercise's format.
# The displayed 0.0 is rounded: it does not mean zero extreme results or zero
# probability. Report more digits and repetitions outside this formatting exercise.
# Print exactly:
# P-value: 0.0
# Decision: reject the null
import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
values = np.array([42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47])
group_size = 6
observed = 6.5
repetitions = 5000
### Your code starts here.



### Your code ends here.

Show code cell source

Hide code cell source

### Solution
import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
values = np.array([42, 46, 38, 50, 45, 49, 40, 45, 44, 52, 41, 47])
group_size = 6
observed = 6.5
repetitions = 5000

simulated = []
for _ in range(repetitions):
    shuffled = rng.permutation(values)
    difference = (shuffled[:group_size].sum() - shuffled[group_size:].sum()) / group_size
    simulated.append(difference)
simulated = pd.Series(simulated)
p_value = (simulated >= observed).mean()
if p_value < 0.05:
    decision = 'reject the null'
else:
    decision = 'do not reject the null'
print('P-value:', round(p_value, 2))
print('Decision:', decision)
# There are 11 extreme results in these 5,000 seeded repetitions: p = 0.0022.
# The rounded display is 0.0; the unrounded estimate drives the decision.
# As in any simulation, another seed produces a slightly different estimate.

P-value: 0.0
Decision: reject the null

### Question 5: What Can Each Study Claim?
# The store reviews three studies with consistent outcome measurement and
# comparable follow-up. Assume no interference between participants, and
# that the randomized studies were conducted as planned.
# Classify each study:
#   - p-value of 0.05 or more             -> "no evidence of a difference"
#     (meaning insufficient evidence at this cutoff, not proof of no effect)
#   - p-value below 0.05 and randomized   -> "evidence of a causal effect"
#   - p-value below 0.05, not randomized  -> "association only"
# Print one line per study, in table order.
# Print exactly:
# Coupon email: evidence of a causal effect
# Loyalty members: association only
# New layout: no evidence of a difference
import numpy as np
import pandas as pd

studies = pd.DataFrame({
    "study": ["Coupon email", "Loyalty members", "New layout"],
    "randomized": [True, False, True],
    "p_value": [0.01, 0.002, 0.34],
})
### Your code starts here.



### Your code ends here.

Show code cell source

Hide code cell source

### Solution
import numpy as np
import pandas as pd

studies = pd.DataFrame({
    "study": ["Coupon email", "Loyalty members", "New layout"],
    "randomized": [True, False, True],
    "p_value": [0.01, 0.002, 0.34],
})

for study, randomized, p_value in zip(studies['study'], studies['randomized'], studies['p_value']):
    if p_value >= 0.05:
        claim = "no evidence of a difference"
    elif randomized:
        claim = "evidence of a causal effect"
    else:
        claim = "association only"
    print(study + ":", claim)
# Customers who chose to join the loyalty program may differ in many ways from
# those who did not, so even a tiny p-value shows association, not causation.

# These labels assume the design and follow-up conditions stated in the question.
# A causal effect concerns the study participants; generalizing needs justification.
# A statistically detectable effect can still be too small to justify its costs.

Coupon email: evidence of a causal effect
Loyalty members: association only
New layout: no evidence of a difference

Submit Clear Results