Chapter 09: Hypothesis Testing

Compare observed evidence with a chance model.

Which outcomes would be unusual if the stated model were true?

Learning Goals

  1. State the null model and the alternative before testing.
  2. Choose a statistic and identify its direction of evidence.
  3. Simulate the model and compare the observed statistic.
  4. Report the p-value, limitations, and practical consequences.

Chapter Flow

Case Statistic Evidence against the model
Swain: 26% eligible share Count among 100 Small counts
Alameda: eligible shares Total variation distance Large distances
Mendel: 75% purple Absolute distance from 75% Large distances

A two-sided alternative can use a right-tail distance statistic.

What To Watch For

  • Keep the model and sample size fixed. Draw a new sample each time.
  • Count simulated statistics at least as extreme as observed.
  • Divide by repetitions to estimate the p-value.
  • Apply the significance level chosen before seeing the result.

Zero extreme simulations means none were seen in that run. It does not establish zero probability.

Carry Forward

A checkout change has p = 0.003 and an estimated gain of 0.02 percentage points.

  • 100,000 monthly visits imply 20 extra purchases.
  • At $10 contribution each, the estimated gain is $200.
  • A $1,000 monthly operating cost exceeds that gain.

Statistical evidence alone does not establish business value. Report the estimated effect and its uncertainty too.