10. Two Samples#
This chapter compares two groups. You will use A/B testing, randomized experiments, and causal reasoning to decide whether an observed difference is meaningful.
Chapter Video
Video credit: CrashCourse. (2018, August 15). T-Tests: A Matched Pair Made in Heaven: Crash Course Statistics #27. YouTube. Watch on YouTube.
This embedded video is provided for educational reference only. All rights are retained by the respective copyright holder.
The video introduces t-tests. This chapter develops a simulation-based permutation approach; focus on the shared question of how to interpret a difference between groups.
Learning Objectives
By the end of this chapter, you will be able to:
Frame two-sample questions in terms of groups, outcomes, and differences.
Distinguish random sampling from random assignment and use permutations to simulate a null comparison.
Run and interpret A/B tests for management and product decisions.
Explain how experimental design supports causal claims.
Separate association, causation, statistical evidence, and practical impact.
Chapter Flow
Glossary
# |
Term |
Explanation |
|---|---|---|
1 |
Two-sample comparison |
An analysis that compares outcomes from two groups. |
2 |
A/B test |
A randomized comparison of two conditions or versions. |
3 |
Treatment group |
The group receiving the intervention or new condition. |
4 |
Control group |
The comparison group receiving the baseline condition. |
5 |
Outcome |
The measured result used to compare groups. |
6 |
Difference in means |
The mean of one group minus the mean of another. |
7 |
Difference in proportions |
The proportion of one group minus the proportion of another. |
8 |
Random sampling |
Selecting observations from a population by chance. |
9 |
Random assignment |
Assigning units to conditions by chance. |
10 |
Permutation |
A rearrangement used to simulate group labels under a null model. |
11 |
Null model |
A stated reference model describing how group differences could arise through a specified chance mechanism. |
12 |
Test statistic |
A summary used to measure the group difference. |
13 |
P-value |
The probability, under the null model, of a statistic at least as extreme as observed. |
14 |
Causality |
A claim that changing one factor changes an outcome. |
15 |
Confounding |
Mixing of effects that makes causal interpretation difficult. |
16 |
Blinding |
Hiding treatment status to reduce bias. |
17 |
Placebo |
An inactive treatment used as a comparison. |
18 |
Sample size |
The number of observations in a study or group. |
19 |
Effect size |
The magnitude of a difference in practical units. |
20 |
Statistical power |
The chance that a specified test detects an effect of a specified size under a given study design. |
21 |
Business experiment |
A structured test used to support a management decision. |