Chapter 10: Two Samples

Compare two groups and reason about experiments.

Which differences could the stated null model produce?

Learning Goals

  1. State the groups, outcome, and subtraction order.
  2. Generate and interpret a permutation distribution.
  3. Distinguish sampling, assignment, and causal evidence.
  4. Report effect size, uncertainty, and business consequences.

Chapter Flow

Case Direction of evidence Design and interpretation
Smoking groups Smaller signed differences Observational association
Football teams Larger signed differences Exchangeable-label model check
Randomized trial Larger absolute distances Causal evidence under trial assumptions

The same computation can serve different study designs.

What To Watch For

  • Keep outcomes and group sizes fixed. Rearrange labels.
  • Compute the same statistic for each new shuffle.
  • Include ties in the tail selected before testing.
  • Divide by the number of results actually generated.

Zero extreme draws and a rounded display of 0.0 do not establish zero probability. Decide with the unrounded estimate.

Carry Forward

A conversion increase from 8% to 10% is:

  • Two percentage points on the absolute scale.
  • 25% relative lift, using 8% as the baseline.
  • An estimated 200 extra purchases per 10,000 visits.

At $5 contribution per purchase, that is $1,000 before costs. Statistical significance alone does not establish profitability.