Hypothesis Testing

"It worked" is a claim that deserves a trial

A marketing director declared victory when email click-through rose from 2.0% to 2.4% after a subject-line change. The test ran four days on 20,000 sends per arm. Finance approved a full rebrand of email templates. Two months later, revenue from email-attributed orders was flat. The team had seen real numbers move, but they never asked how extreme those numbers would be if the subject line actually changed nothing. Hypothesis testing is the formal discipline for that question. It pairs with confidence intervals from Lesson 2: intervals show plausible ranges; tests evaluate whether data are compatible with a null hypothesis of no effect or status quo.

The null hypothesis (H₀, pronounced "H-naught") is the skeptical default you test against. In an A/B test (A/B test, a randomized comparison of control A vs variant B), H₀ often says conversion rates are equal. The alternative hypothesis (H₁) says an effect exists (two-sided: different in either direction; one-sided: better in a pre-specified direction only). You choose a significance level α (alpha, the maximum false-positive rate you tolerate, often 0.05) before seeing results. You compute a test statistic measuring how far data sit from what H₀ predicts. You convert that into a p-value. If p-value < α, you reject H₀ ("incompatible with no effect at this threshold"). If not, you fail to reject H₀ (data do not provide strong evidence against no effect).

Hypothesis testing does not prove H₁ true with certainty. It controls error rates when used with good design. Lesson 1: Samples, Populations, and Sampling Error warned that bias breaks any test; Lesson 4 will show that statistical significance is not business importance. Your job is to demand the full output: H₀, H₁, α, sample sizes, test statistic, p-value, confidence interval on effect size, and dollar read.

The logic of falsification in business language

Hypothesis testing mimics courtroom presumption of innocence. H₀ is "no effect" until evidence contradicts it at a chosen standard. You do not prove variant B works; you ask whether data make "B equals C" implausible. That framing protects organizations from deploying every random wiggle.

The cost is conservatism: truly helpful variants may be rejected when n is small (Type II error). The fix is not to abandon testing; it is to size tests for effects that matter and to use reversible rollouts when evidence is suggestive but not conclusive.

z-test formula reference card (two proportions)

For quick desk checks when software is unavailable:

1. p̂_C = x_C/n_C, p̂_B = x_B/n_B
2. p̂_pool = (x_C + x_B)/(n_C + n_B)
3. SE = √[ p̂_pool(1−p̂_pool) (1/n_C + 1/n_B) ]
4. z = (p̂_B − p̂_C)/SE
5. Two-sided p from normal tail; compare to α

Always write hypotheses before step 1. If you cannot state H₀ in plain English, the test is not ready.

1. State hypotheses in plain language and symbols. Example: H₀: p_B = p_C; H₁: p_B ≠ p_C (two-sided).

2. Choose α (commonly 0.05 for exploratory product tests; stricter for regulated claims). α is the Type I error rate you accept if H₀ is true.

3. Select test matched to data type (two-proportion z-test for conversion A/B; one-sample t-test for mean vs target; etc.).

4. Check assumptions (random assignment, independent observations, sample size rules for proportions).

5. Compute test statistic and p-value from data.

6. Decide and interpret in context: reject or fail to reject H₀; report CI and practical effect size.

Pre-register steps 1 to 4 before data arrive. Changing the primary metric after peeking invites p-hacking (Lesson 5).

p-value in plain English (and what it is not)

The p-value answers: "If the null were true (no difference, mean equals target, etc.), how often would random sampling produce a result at least this far from the null?" A small p-value means "unlikely under H₀." A large p-value means "could easily happen by chance under H₀."

Common misread: "p = 0.03 means 3% chance H₀ is true." That is wrong. The p-value is computed assuming H₀ is true; it is not a probability H₀ is true. Another misread: "p = 0.06 means no effect." Non-rejection is not proof of no effect; you may lack power (sample too small to detect a real lift).

For a two-sided two-proportion test, if |z| = 1.96, p ≈ 0.05. If |z| = 2.58, p ≈ 0.01. Software and z-tables give exact tails.

Type I, Type II errors, and power

Type I error: Reject H₀ when H₀ is true (false positive). Rate controlled by α.

Type II error: Fail to reject H₀ when H₁ is true (false negative).

Power = 1 − P(Type II) = probability of rejecting H₀ when a specified real effect exists.

Lowering α (stricter) reduces Type I but raises Type II if n fixed. Increasing n reduces Type II at fixed α. Underpowered tests (small n) miss lifts that matter; that is why minimum detectable effect (MDE, the smallest lift you need to detect) and sample-size planning precede expensive experiments (Lesson 4).

Numerical intuition: suppose true lift is +0.5 pp but your test only has 20% power. Four times out of five you will fail to reject H₀ even when the variant truly helps. Teams interpret that as "test failed" and kill good ideas. Conversely, with enormous n and true lift +0.01 pp, power nears 100% and you reject H₀ for economically irrelevant lifts. Power and MDE connect statistics to business stakes.

Two-proportion z-test for A/B conversion

When comparing independent proportions with large samples, use a pooled proportion under H₀:

p̂_pool = (x_C + x_B) / (n_C + n_B)

SE = √[ p̂_pool(1 − p̂_pool) × (1/n_C + 1/n_B) ]

z = (p̂_B − p̂_C) / SE

Two-sided p-value from standard normal tails. For confidence interval on difference, use unpooled SE from Lesson 2 (not identical to test SE, but aligned in decision).

One-sided tests use H₁: p_B > p_C (or <) only if worse direction is irrelevant and was pre-specified. Two-sided is default for product work because harm matters.

One-sample tests and means

One-sample proportion test: Compare p̂ to hypothesized p₀. SE under H₀ uses p₀: SE = √[p₀(1−p₀)/n].

One-sample mean test: Compare x̄ to target μ₀ with t = (x̄ − μ₀) / (s/√n), df = n−1.

Choose the test that matches the claim. "Did we beat 20% open rate?" is one-sample proportion. "Did B beat A?" is two-sample difference.

A/B testing governance managers should enforce

• Random assignment and stable exposure window
• Pre-specified primary metric and MDE tied to dollars
• Sample size from power analysis, not "when it looks good"
• Report p-value, CI on lift, and expected profit
• Segmented analysis plan documented upfront (not 20 post-hoc cuts)

Stopping early when results look significant inflates false positives unless specialized sequential methods are pre-approved. Peeking is a governance problem, not only a math problem.

Reading p-values on a spectrum

Treat p-values as a compatibility measure, not a bright line only at 0.05:

A result with p = 0.049 is not materially stronger than p = 0.051 in business terms, even though one crosses α = 0.05. Confidence intervals and MDE (minimum detectable effect, Lesson 4) should drive reversible and irreversible calls more than arbitrary thresholds.

One-sided versus two-sided tests (decision rules)

Two-sided H₁: p_B ≠ p_C. Detects harm or help. Default for product and pricing tests where downside matters.

One-sided H₁: p_B > p_C only. Uses half the tail area; smaller p for same z in the favorable direction. Justified when worse direction is irrelevant and pre-registered (non-inferiority medical trials are specialized cases).

Choosing one-sided after seeing variant ahead is p-hacking (Lesson 5). If you would panic when variant loses, use two-sided.

Power analysis sketch before you run

Power depends on baseline rate, MDE, α, and n. Illustrative two-proportion plan:

• Baseline p = 0.045 (4.5%)
• Target p_B = 0.050 (+0.5 pp MDE)
• α = 0.05 two-sided, power = 0.80

Required n per arm is on the order of 30,000 sessions (exact value from power software). Without that plan, a test with n = 2,000 per arm is underpowered for 0.5 pp: real lifts are often missed (Type II error).

Before approving A/B tests, ask for: "What lift do we need to pay for this?" and "What n gives 80% chance to detect that lift?" If runtime exceeds calendar, widen MDE or accept risk explicitly in the decision memo.

Paired versus independent samples (conceptual)

Independent samples: control and variant are different customers (standard web A/B). Use two-proportion z-test from this lesson.

Paired samples: same stores before and after, or same people twice. Differences are correlated; use paired tests on the difference column. Mis-specifying independence inflates false positives when pairs are treated as independent.

Always ask: "Could the same unit appear in both groups?" If yes, independent two-sample formulas are wrong.

Connecting tests to confidence intervals and dollars

Report a standard table in experiment readouts:

Dollar row forces Lesson 4 integration at the source and prevents "p = 0.04" slack messages without context.

Sequential testing warning (peeking)

Teams that peek daily and stop when p < 0.05 inflate Type I error dramatically. If you peek 20 times, effective α is far above 0.05 even if each test uses α = 0.05. Fixes: fixed horizon (run to planned n), formal sequential designs (outside scope here), or discipline: peek for logistics only, decide once.

Example: true conversion unchanged, daily p wanders. By day 14, one day may show p = 0.04 by luck. Launching on that day is a false positive factory. Pre-commit to decision date and sample size; treat interim peeks as monitoring for bugs, not success.

Guardrail metrics in A/B tests

Primary metric might be conversion; guardrails protect harm: average order value, load time, support tickets, refund rate. Test guardrails with pre-specified limits: "Do not launch if refund CI lower bound exceeds +0.2 pp."

If primary wins but guardrail degrades significantly, business significance fails even when p < 0.05 on conversion. Document guardrail hierarchy before launch.

Full numeric appendix: two-proportion test checklist

Use RidgeRetail data (control 4.50%, variant 5.10%, n = 2,000 each):

Memorize the flow, not every decimal. Software computes; managers verify structure and interpret.

RidgeRetail tested a streamlined checkout (variant B) vs control (C). Control: n_C = 2,000 sessions, 90 purchases (4.50%). Variant B: n_B = 2,000 sessions, 102 purchases (5.10%).

Part A: Hypotheses and setup

H₀: p_B = p_C (no conversion difference)
H₁: p_B ≠ p_C (two-sided)
α = 0.05

p̂_C = 90/2000 = 0.045; p̂_B = 102/2000 = 0.051; observed lift = 0.6 pp.

Part B: Pooled test statistic

p̂_pool = (90 + 102) / 4000 = 192/4000 = 0.048

SE = √[0.048 × 0.952 × (1/2000 + 1/2000)]
= √[0.045696 × 0.001] = √0.000045696 ≈ 0.00676

z = (0.051 − 0.045) / 0.00676 = 0.006 / 0.00676 ≈ 0.888

Part C: p-value and decision

Two-sided p-value ≈ 2 × P(Z > 0.888) ≈ 0.37 (37%).

Since 0.37 > 0.05, fail to reject H₀. Data are consistent with no difference at α = 5%.

Check: |z| = 0.888 < 1.96 critical value ✓

Part D: CI on difference and managerial read

SE_unpooled = √[0.045×0.955/2000 + 0.051×0.949/2000] ≈ 0.00675

95% CI on difference = 0.006 ± 1.96×0.00675 = −0.0072 to +0.0192 (−0.72 pp to +1.92 pp)

Includes zero ✓, aligns with fail to reject.

Managerial read: Do not globalize checkout on +0.6 pp. True lift could be negative. To detect a planned MDE of +0.5 pp with 80% power at α = 0.05 two-sided, rough per-arm n is on the order of tens of thousands (use a power calculator with baseline 4.5%). Either scale the test or accept high Type II risk.

A vendor claims its targeting tool achieves 20% open rate. RidgeRetail sends 400 targeted emails; 85 are opened.

Part A: Hypotheses

H₀: p = 0.20
H₁: p ≠ 0.20 (two-sided)
α = 0.05

p̂ = 85/400 = 0.2125 (21.25%)

Part B: Test under H₀

SE = √[0.20 × 0.80 / 400] = √[0.16/400] = √0.0004 = 0.02

z = (0.2125 − 0.20) / 0.02 = 0.0125 / 0.02 = 0.625

Part C: p-value

Two-sided p ≈ 0.53. Fail to reject H₀.

95% CI for true p (using p̂ SE): 0.2125 ± 1.96×√[0.2125×0.7875/400] ≈ 0.2125 ± 0.040 = 0.173 to 0.252

CI includes 0.20 ✓

Part D: Managerial read

Observed 21.25% does not disprove vendor's 20% claim. Contract renewal should use business thresholds: is 21% vs 20% worth price premium? Statistical compatibility is not endorsement. Combine with cost per acquisition in Lesson 4.

Before rerunning checkout test, RidgeRetail finance sets MDE = +0.5 pp (0.045 to 0.050), α = 0.05 two-sided, desired power = 0.80.

Part A: Why the first test failed business and stats

Observed +0.6 pp had p = 0.37. CI included zero. Even if true lift were +0.5 pp, the test was not designed to detect it reliably.

Part B: Approximate per-arm n (illustrative formula output)

Using standard two-proportion power formulas (software recommended for exact values), required n per arm is roughly 31,000 sessions for this baseline and MDE. At 50,000 sessions per week total split across arms, runtime ≈ 1.2 weeks per arm if 100% traffic in test ( unrealistic); practical split 50/50 on checkout traffic might need several weeks.

Check: doubling MDE to +1.0 pp drops required n dramatically (roughly quarter to fifth), illustrating the cost of demanding fine precision on low baselines.

Part C: Decision charter

Product accepts 3-week test at 50/50 if MDE +0.5 pp yields $1.2M annual profit (precomputed). Kill if CI lower bound < 0 pp or if significant increase in cart abandonment (guardrail metric, separate test with Bonferroni-adjusted α if multiple guardrails).

Part D: Managerial read

Power planning turns " inconclusive" into a scheduled answer date instead of endless peeking. Board gets: "We will know by July 15 whether lift plausibly exceeds +0.5 pp at 80% power, or we stop."

Control: 45 conversions / 1,000 sessions (4.5%). Variant: 58 / 1,000 (5.8%). Two-sided α = 0.05.

1. Compute p̂_pool, SE, z, and p-value. Reject or fail to reject?
2. Compute 95% CI on difference (unpooled).
3. Implementation costs $500,000; each +0.1 pp conversion yields $80,000 annual profit. Should you launch on statistics alone? Explain in a paragraph.

Solution

1. p̂_pool = (45+58)/2000 = 103/2000 = 0.0515. SE = √[0.0515×0.9485×(1/1000+1/1000)] = √[0.048847×0.002] = √0.000097694 ≈ 0.00988. z = (0.058−0.045)/0.00988 = 0.013/0.00988 ≈ 1.32. Two-sided p ≈ 0.19. Fail to reject H₀ at 5%.

2. SE_unpooled = √[0.045×0.955/1000 + 0.058×0.942/1000] ≈ 0.00988. CI = 0.013 ± 1.96×0.00988 = −0.0064 to +0.0324 (−0.64 pp to +3.24 pp).

3. Statistics alone do not justify full launch. p > 0.05 and CI includes zero and negative lifts. Point estimate +1.3 pp might imply 1.3×10×$80k = $1.04M gross upside if real, but lower bound −0.64 pp shows downside. Payback on $500k is uncertain. Treat as pilot: extend test for power, or launch limited rollout with reversal plan, not enterprise bet on significance.

Detailed verification: z = 0.013/0.00988 = 1.316. Two-sided p between 0.18 and 0.20. Unpooled CI includes zero, consistent with fail to reject. Dollar math at lower bound −0.64 pp: −0.0064×10M×$80k/0.001... simplify: 0.64 pp × 10M/100 × $80k per 0.1 pp block = negative six-figure swing, unacceptable without mitigation.

Call center target average handle time is 6.0 minutes. Sample n = 16 calls, x̄ = 6.4 minutes, s = 0.8 minutes. Use t* = 2.131 (df = 15) for two-sided test at α ≈ 0.05.

1. State H₀ and H₁ for "handle time differs from target."
2. Compute t statistic and compare to critical value (decision rule: reject if |t| > 2.131).
3. Compute 95% CI for true mean handle time.

Solution

1. H₀: μ = 6.0. H₁: μ ≠ 6.0.

2. SE = 0.8/√16 = 0.8/4 = 0.2. t = (6.4 − 6.0)/0.2 = 2.0. |t| = 2.0 < 2.131 → fail to reject H₀ at this α (borderline; larger n would clarify).

3. CI = 6.4 ± 2.131×0.2 = 6.4 ± 0.426 = 5.97 to 6.83 minutes. Includes 6.0 ✓, consistent with fail to reject.

Operations note: Even failing to reject, upper bound 6.83 may breach service goals; business significance (Lesson 4) may still trigger process review.

Borderline t statistics illustrate why power matters: with n = 16, SE is large; true mean 6.5 might still fail to reject. Increasing to n = 64 quarters SE, t would exceed critical value if true mean remains 6.4. Always pair hypothesis tests with CI width and business thresholds, not only critical value comparison.

A manager sees: "n = 5,000 per arm, control 3.00%, variant 3.12%, z = 2.02, p = 0.043, CI difference +0.01 pp to +0.23 pp."

Step 1 (Lesson 1): Randomized sessions, large n, sampling error small.

Step 2 (Lesson 2): CI excludes zero narrowly; true lift plausibly as low as +0.01 pp.

Step 3 (Lesson 3): Reject H₀ at α = 0.05; p just below threshold.

Step 4 (Lesson 4): At 10M sessions and $30 profit per conversion, +0.12 pp point → $360k/year. If build is $2M, business kill despite significance.

Step 5 (Lesson 5): Confirm primary metric pre-specified; if twenty metrics tested, p = 0.043 may be noise.

This walkthrough is the unit in one paragraph chain.

Type I and Type II in dollar terms

Type I (false launch): ship a useless feature costing $500k because p < 0.05 by luck. Type II (false kill): abandon a +$2M variant because n was too small. α controls Type I rate; power controls Type II. Finance should ask which error is costlier for this decision class and set α, n, and reversibility accordingly. Irreversible bets: lower α, higher n, stress CI lower bound. Reversible bets: accept higher α with monitoring.

One-sample z-test reference (proportion vs benchmark)

When testing if open rate differs from benchmark p₀, use SE = √[p₀(1−p₀)/n] under H₀, not p̂. Vendor claim example: p₀ = 0.20, n = 400, p̂ = 0.2125, z = 0.625. If instead you wrongly used p̂ in SE, z would differ slightly; using p₀ is correct for the null. Managers rarely compute by hand but should verify analysts used the right SE in vendor audits.

A/B testing decision tree (summary)

1. Was assignment random and population defined? If no, stop; fix design.
2. Compute CI on primary lift and p-value on pre-registered primary.
3. If CI includes zero, default is do not launch nationally unless reversible pilot with guardrails.
4. If CI excludes zero, translate to dollars at lower bound; compare to fully loaded cost.
5. Red-team for p-hacking and segment Simpson issues before board.

This tree is the operational output of Lessons 1 through 5 combined.

Worked p-value interpretation drill

Suppose z = 1.96 exactly for a two-sided test. p ≈ 0.05. Plain language: "If there were truly no difference, data at least this extreme would occur about 5% of the time." That is rare but not impossible; one in twenty true nulls looks this strong by chance. Hence pre-registration and replication matter. If z = 0.5, p ≈ 0.62: "If null were true, we'd often see data this mild or more extreme; nothing alarming."

Never convert p to "62% chance null is true." That inversion is the most common executive misread in Unit 4.

When reporting to executives, pair every p-value with the CI and a plain sentence: "Compatible with no effect" or "Unlikely if null were true," then immediately state dollar bounds. The p-value is one tile in the mosaic, not the picture.

Hypothesis testing completes the inference toolkit alongside confidence intervals. Where CI shows the range of plausible effects, the test summarizes whether zero effect remains plausible at a chosen α. Use both; never either alone.

Pre-registering α = 0.05 does not mean p = 0.06 is "almost significant." Treat borderline p-values as triggers for more data or reversible pilots, not as launch codes.

Document every test in a shared log with date, H₀, result, and decision. Logs prevent relitigating dead experiments and build organizational memory about what "inconclusive" looked like last quarter.

Key takeaways

• Hypothesis tests discipline "did it work?" by measuring compatibility with a null of no effect.
• p-value is probability of data this extreme if H₀ were true; it is not P(H₀ true).
• Always report CIs and effect sizes alongside p-values; fail to reject may mean low power.
• Pre-specify H₀, H₁, α, metric, and sample size; peeking and metric shopping inflate false wins.
• Governance for A/B tests ties statistical rules to dollar MDE and reversibility.

1. Run a two-proportion test on a real or simulated experiment; report z, p-value, and difference CI.
2. What sample size would detect your smallest worthwhile lift at 80% power?
3. Continue to Lesson 4: Statistical Significance versus Business Significance.