Confidence Intervals

A single number hides the risk you are about to take

A growth team reported that checkout conversion was 4.2% after a redesign. The vice president of Product asked whether to roll out globally. "4.2% is up from 4.0%" sounds decisive. It omits the question every executive should ask: how wrong could we be if the true conversion rate for all future visitors is not 4.2%? Lesson 1: Samples, Populations, and Sampling Error taught that sample statistics wiggle because of randomness. A confidence interval (CI) turns that wiggle into a range of plausible values for the unknown population parameter (true conversion, true average order value, true defect rate).

A point estimate is one number from your sample (4.2%, $47 average, 88% satisfaction). A confidence interval pairs that point with a margin of error derived from the standard error (SE, the typical size of sampling error for the statistic). The usual executive translation of a 95% CI is: "Given how we sampled, the true rate plausibly sits in this band." It is not a guarantee that there is a 95% probability the parameter sits in this specific band (that wording is Bayesian, a different interpretation family). The frequentist 95% CI means: if we repeated the same sampling process many times and built an interval each time, about 95% of those intervals would capture the true parameter. For decision makers, the practical read is the same: plan for the range, not the midpoint alone.

Opening hook expansion: why intervals beat thresholds alone

Executives often ask "Are we above 4% conversion?" That binary question throws away information. A 95% CI of 3.7% to 4.3% answers the threshold question (maybe, because 4% is inside) while also showing precision. If next month CI is 3.9% to 4.1%, the same point 4.0% is tighter evidence. Point estimates without intervals cannot show that learning.

Margin of error in media polls (transferable skill)

Political and market polls report "±3 percentage points." That is half-width of a CI for a proportion, usually 95%. If n = 1,067 and p̂ = 0.48, SE ≈ √[0.48×0.52/1067] ≈ 0.0153, margin ≈ 1.96×0.0153 ≈ 3 pp. Managers who decode polls can decode vendor survey claims the same way.

When a vendor says "margin ±2 pp" without stating n or confidence level, request the full calculation. Marketing claims of precision often assume 95% and hide conservative p = 0.5 planning.

Every CI in this lesson follows the skeleton:

Point estimate ± (critical value) × (standard error)

The point estimate comes from data (sample mean x̄, sample proportion p̂). The standard error estimates how much that point would vary across repeated samples. The critical value comes from a reference distribution and reflects how confident you want to be.

Worked micro-example: n = 1,000, x = 50 conversions, p̂ = 0.05. SE = √[0.05×0.95/1000] = √0.0000475 ≈ 0.00689. Margin at 95% = 1.96×0.00689 ≈ 0.0135. CI ≈ 0.0365 to 0.0635 (3.65% to 6.35%). Weekly report "5.0%" without band hides ±1.35 pp uncertainty.

For large samples or proportions where normal approximations are reasonable, the z critical value for 95% confidence is z = 1.96*. That number comes from the standard normal distribution: about 95% of the area lies within 1.96 standard deviations of the center. For 90% confidence, z* ≈ 1.645; for 99%, z* ≈ 2.576.

When the sample is small and you estimate spread from the data itself, use the t distribution with degrees of freedom (df, roughly sample size minus one for a single mean). The t critical value is slightly wider than z* to account for extra uncertainty. With df = 24 and 95% confidence, t* ≈ 2.064.

Always state n, how the sample was drawn, and whether approximations are justified (for proportions, np and n(1−p) at least 5 or 10 is a common check).

Confidence interval for a single proportion

Marketing teams live on proportions: conversion, click-through, return rate, survey top-box. If you observe x successes in n trials, p̂ = x/n. The 95% CI is:

CI = p̂ ± z × √[ p̂(1 − p̂) / n ]*

Example logic in prose: if p̂ = 0.105 and n = 400, the standard error is √[0.105×0.895/400] ≈ 0.0153. Margin = 1.96×0.0153 ≈ 0.030, so the CI spans roughly 0.075 to 0.135. The width is about 6 percentage points. Narrowing that band requires more trials because margin shrinks with 1/√n.

Planning sample size before data collection avoids underpowered pilots. To target margin of error E at confidence level with planning proportion p:

n = z*² × p(1 − p) / E²

If you have no prior estimate, p = 0.5 is conservative (maximizes p(1−p)). That yields the largest required n, which protects budgets from underestimating needed volume.

Confidence interval for a single mean

For average ticket size, handle time, or weekly revenue, the CI for a population mean uses the sample mean x̄ and sample standard deviation s:

CI = x̄ ± t × (s / √n)* (small sample, σ unknown)

CI = x̄ ± z × (s / √n)* (very large n, or σ known)

The Central Limit Theorem from Lesson 1 explains why averages behave well even when individual transactions are skewed. Still check extremes: a mean revenue CI with two whale deals in n = 15 is fragile; report sensitivity or use robust methods outside this lesson's scope.

Operational leaders should default to the lower bound of a CI when downside is costly (inventory shortfalls, service level agreements) and to the upper bound when upside drives capacity (call center staffing for peak). Using only x̄ treats upside and downside as symmetric when finance often is not.

Confidence interval for a difference (A/B and comparisons)

Product and marketing decisions usually compare variant B to control A. The parameter of interest is p_B − p_C or μ_B − μ_C. The CI for a difference in proportions is:

CI = (p̂_B − p̂_C) ± z × SE_diff*

where

SE_diff = √[ p̂_B(1−p̂_B)/n_B + p̂_C(1−p̂_C)/n_C ]

If the interval for the difference includes zero (no difference), the data are consistent with no effect at that confidence level. That aligns with failing to reject the null in hypothesis testing (Lesson 3), though tests and CIs answer slightly different questions.

Overlap of individual CIs for A and B is not equivalent to a CI for the difference. Two intervals can overlap while the difference CI excludes zero, and vice versa. Always compute the difference CI when deciding lift.

Interpreting width, confidence level, and business stakes

Wider intervals mean more uncertainty. Width shrinks with larger n but only with √n. Halving the margin requires roughly four times the sample. Confidence level trades width for assurance: 99% CIs are wider than 95%. Do not raise confidence to 99% without accepting noisier decisions unless regulation demands it.

Distinguish a CI for a population mean from a prediction interval for a single future observation. Mean CI answers "what is average demand?" Prediction interval answers "what will one store sell next week?" Prediction intervals are always wider. Supplying one store is not the same as estimating the network average.

Choosing z versus t in practice

When σ (population standard deviation) is unknown and n is moderate, the sample standard deviation s substitutes for σ. That substitution adds uncertainty, so we widen intervals using the t distribution instead of the normal z. With large n (often 30 or more for roughly symmetric data), t* and z* are close; with n = 25, t* ≈ 2.064 vs z* = 1.96, about 5% wider half-width.

For proportions, z intervals are standard when np and n(1−p) are sufficiently large. When counts are small (e.g., 3 defects in 50 trials), normal approximations fail and exact binomial methods or Bayesian intervals are preferable. Managers should still receive a range; the method choice is the analyst's job, but the existence of a range is non-negotiable.

Report both confidence level and method in footnotes for regulated or investor-facing work. "95% CI using normal approximation" differs from "95% exact binomial CI," especially in tails.

Step-by-step proportion CI template (manager checklist)

When reviewing an analyst's slide, verify these steps exist:

1. Define population and n clearly.
2. Compute p̂ = x/n.
3. Check np and n(1−p) (both at least 5 as a starter rule).
4. SE = √[p̂(1−p̂)/n].
5. Pick z* (1.96 for 95%) or justify alternative level.
6. Margin = z* × SE.
7. CI = p̂ ± margin.
8. State width in percentage points and compare to decision threshold.

If any step is missing, the interval may be wrong or non-replicable. Replication matters when the same decision will recur next quarter.

Difference CIs and hypothesis test alignment

Lesson 3 shows that when a 95% CI for a difference includes zero, a two-sided hypothesis test at α = 0.05 typically fails to reject the null of no difference (with minor exceptions when test uses pooled SE and CI uses unpooled SE). The CI is often more informative because it shows how large a difference remains plausible, not only whether zero is excluded.

Example: difference in conversion 1.3 pp with CI −0.6 pp to +3.2 pp. Test p ≈ 0.19. Fail to reject. CI adds: true lift could plausibly be −0.6 pp (harm) or +3.2 pp (strong win). A p-value alone hides that asymmetry unless paired with bounds.

For capital decisions, always request the difference CI unpooled for two independent samples:

SE_diff = √[ p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂ ]

Using pooled SE for CI when proportions differ slightly is a common analyst shortcut; unpooled is the safer default for interpretation.

Sample size planning: conservative versus informed

Lesson 1 introduced n = z*² p(1−p) / E². Three planning scenarios appear in practice:

Conservative (p = 0.5): Maximizes required n when you have no clue. Protects budgets.

Pilot-informed (p from early data): Lowers n when p is far from 0.5 (defect rates near 2%, conversion near 1%).

Fixed budget (n given, solve for E): E = z* × √[p(1−p)/n]. Leadership picks n = 2,000 sessions; analyst reports achievable margin, not fantasy ±0.1 pp.

Always round n up to integers. Fractional customers do not exist.

Prediction intervals versus confidence intervals (operational example)

Store weekly sales: x̄ = $500k, s = $80k, n = 52 weeks. 95% CI for mean weekly sales (using t* ≈ 2.0):

SE_mean = 80/√52 ≈ 11.1 → CI ≈ $478k to $522k.

95% prediction interval for one future week adds individual variation:

Rough form: x̄ ± t* × s × √(1 + 1/n) ≈ 500 ± 2.0 × 80 × 1.01 ≈ $339k to $661k.

Inventory for a single high-stakes week should use prediction logic, not the mean CI. Network staffing for average load uses mean CI. Mixing them understocks or overstaffs.

Executive briefing: translating CIs without jargon

When presenting to a board, replace "95% CI 3.65% to 6.35%" with: "If we reran this measurement many times the same way, a band like this would usually contain the true conversion rate. Given n = 1,000, true rate could plausibly be as low as 3.7% or as high as 6.4%. Last week's move from 4.8% to 5.2% is inside that band, so we should not reallocate budget yet."

For means: "Average handle time is 8.2 minutes in our sample; plausibly 7.6 to 8.8 for the true average among all calls this quarter." Operators hear action; investors hear risk bounds.

Always pair CI with n and sampling note: "Based on 900 random audits" beats "based on data we had."

Worked comparison: same data at 90%, 95%, and 99% confidence

Use LarkCommerce trial: p̂ = 0.105, n = 400, SE = 0.01533.

Check: 1.96×0.01533 ≈ 0.030 ✓

Higher confidence widens the band. A break-even threshold at 9% conversion sits inside all three intervals, so even 99% CI does not prove break-even is met. Decision: extend trial or accept risk; confidence level alone does not create certainty.

Sensitivity to confidence level

Same data, different confidence levels:

Regulated industries may mandate 99% for safety limits. Growth teams often plan at 95% but stress 99% upper bound for downside risk metrics (chargeback rate, error rate). Document which level governs the decision to avoid moving goalposts after results.

LarkCommerce ran a limited trial of a one-click checkout on 400 eligible sessions. 42 completed purchase. Leadership wants a 95% CI for true conversion before a board review.

Part A: Setup

n = 400 trials, x = 42 conversions, point estimate p̂ = 42/400 = 0.105 (10.5%).

Check: np = 42 and n(1−p) = 358, both ≥ 5 ✓

Population: all eligible sessions if one-click were shown under current traffic mix.

Part B: Standard error and CI

SE = √[0.105 × 0.895 / 400] = √[0.093975 / 400] = √0.00023494 ≈ 0.01533

Margin = z* × SE = 1.96 × 0.01533 ≈ 0.03005 (3.0 pp)

95% CI = 0.105 ± 0.03005 = 0.0750 to 0.1350 (7.5% to 13.5%)

Check: half-width 0.030; 0.105 − 0.030 = 0.075; 0.105 + 0.030 = 0.135 ✓

Part C: Precision planning

Current width ≈ 6.0 pp. To halve width to 3.0 pp, need n × 4:

n_new ≈ 400 × 4 = 1,600 trials

Verify: SE_new ≈ 0.01533/2 ≈ 0.00767; margin ≈ 0.015 ✓

Part D: Managerial read

Board question: "If true conversion might be 7.5%, does one-click pay for engineering?" Finance models should stress the lower bound, not 10.5%. If the initiative needs 9% conversion to break even, the CI shows upside potential but not proof. Extend the trial or accept risk explicitly.

Northwind Foods tracks two metrics: average weekly store sales for a small region, and a promotional banner test on checkout conversion.

Scenario 1 (mean): n = 25 weeks, x̄ = $520,000, s = $80,000. Use 95% CI with t* = 2.064 (df = 24).

SE = 80,000 / √25 = 80,000 / 5 = $16,000

CI = 520,000 ± 2.064 × 16,000 = 520,000 ± 33,024 = $486,976 to $553,024

Check: 2.064 × 16 = 33.024 ✓

Inventory planning using $520k ignores a plausible $487k floor.

Scenario 2 (difference in proportions): Control 1,000 sessions, 45 conversions (4.5%). Variant 1,000 sessions, 58 conversions (5.8%).

Point difference = 0.058 − 0.045 = 0.013 (1.3 pp)

SE_diff = √[0.045×0.955/1000 + 0.058×0.942/1000]
= √[0.000042975 + 0.000054636] = √0.000097611 ≈ 0.00988

95% CI = 0.013 ± 1.96 × 0.00988 = 0.013 ± 0.01936 = −0.0064 to +0.0324 (−0.64 pp to +3.24 pp)

The interval includes zero ✓. Observed +1.3 pp lift is compatible with no true difference at 95% confidence. Matches hypothesis test intuition in Lesson 3.

Managerial read: Do not nationalize the banner on 1.3 pp alone. Either increase sample to narrow the band or pair with dollar thresholds (Lesson 4).

Coastal Bank redesigned branch queuing. They timed n = 20 random customer visits during peak hours: x̄ = 11.4 minutes, s = 3.2 minutes. Leadership promised regulators "average wait under 12 minutes." Use 95% CI with t* = 2.093 (df = 19).

Part A: Setup

Population: peak-hour visits at pilot branches during June 2026. Sample: 20 timed visits, assumed random within peak window. Parameter: true mean wait μ.

Part B: Calculation

SE = s/√n = 3.2/√20 = 3.2/4.472 ≈ 0.716 minutes

CI = 11.4 ± 2.093 × 0.716 = 11.4 ± 1.498 = 9.90 to 12.90 minutes

Check: 2.093 × 0.716 ≈ 1.498 ✓

Part C: Interpretation vs target

The interval includes 12.0 and extends above it. Data are consistent with true mean wait at or below 12 minutes, but also consistent with mean near 12.9. A hypothesis test of H₀: μ = 12 vs H₁: μ ≠ 12 would use t = (11.4−12)/0.716 ≈ −0.84, p large, fail to reject.

Part D: Managerial read

Do not claim victory to regulators based on x̄ = 11.4 alone. Either extend timing study to narrow CI below 12 upper bound, or operationalize "95% upper bound must be ≤ 12" as success criterion before national rollout.

A logistics team samples 900 deliveries; 72 were late.

1. Compute p̂ and a 95% CI for true late-delivery proportion.
2. Leadership wants margin ±2 pp at 95% confidence. How many deliveries must be audited?
3. Explain in a paragraph why the lower bound of the CI matters for a service-level penalty clause.

Solution

1. p̂ = 72/900 = 0.08. SE = √[0.08×0.92/900] = √[0.0736/900] = √0.00008178 ≈ 0.00904. Margin = 1.96×0.00904 ≈ 0.0177. CI = 0.08 ± 0.0177 = 0.0623 to 0.0977 (6.2% to 9.8%). Check: 0.08 − 0.0177 ≈ 0.062 ✓

2. n = 1.96² × 0.08 × 0.92 / 0.02² = 3.8416 × 0.0736 / 0.0004 ≈ 707. (If using conservative p = 0.5, n ≈ 2,401.) Current n = 900 already beats 707 for ±2 pp if p ≈ 0.08.

3. Penalties trigger on breaching a late-rate cap (say 10%). Point estimate 8% looks safe, but true rate could plausibly be 9.8% or even higher if bias understates lateness. Contract risk lives in the upper tail for the vendor, lower tail for the buyer seeking refunds. A manager defending service levels should stress the upper bound near 10%; a buyer auditing performance should not assume 8% is guaranteed.

Step-by-step audit of part 1: x = 72 late, n = 900, p̂ = 72/900 exactly 0.08. Variance estimate p̂(1−p̂) = 0.0736. SE = √(0.0736/900). Square root of 900 is 30; 0.0736/900 = 0.000081777...; sqrt ≈ 0.009043. Double-check margin: 1.96×0.009043 = 0.01772. Lower 0.08−0.01772 = 0.06228; upper 0.09772. All steps reconcile ✓

For part 2, solving n = z² p(1−p)/E² with p = 0.08 gives 707.4, round to 708. At n = 900, achieved margin ≈ 1.96×√(0.0736/900) ≈ 1.77 pp, better than required 2 pp ✓

Two training programs: Program A, n = 30, average post-test score x̄_A = 78, s_A = 10. Program B, n = 35, x̄_B = 82, s_B = 12. Use z* = 1.96 for an approximate 95% CI on μ_B − μ_A (large-sample approximation).

1. Compute SE_diff and CI for the difference in means.
2. Does the CI include zero? What would you tell HR about "B beats A"?
3. If minimum meaningful improvement is 5 points, is the CI sufficient for a rollout decision?

Solution

1. SE_diff = √(10²/30 + 12²/35) = √(100/30 + 144/35) = √(3.333 + 4.114) = √7.447 ≈ 2.728. Difference = 82 − 78 = 4. CI = 4 ± 1.96×2.728 = 4 ± 5.35 = −1.35 to 9.35 points.

2. Interval includes zero. Data are consistent with no true difference at 95% confidence despite sample difference of 4 points.

3. Even the point estimate 4 < 5 point business threshold. Upper bound 9.35 suggests possible meaningful gain, but lower bound −1.35 shows downside. HR should extend evaluation with more trainees or accept uncertainty. Statistical and business significance diverge (Lesson 4).

Required n to detect 5-point difference with similar SDs at 80% power would be far larger than 30 per arm; HR's pilot is underpowered for MDE = 5. Planning note: if they must decide now, use lower bound −1.35: training could harm scores slightly, unacceptable for compliance training goals.

Northwind banner: control 4.5% (45/1,000), variant 5.8% (58/1,000).

95% CI on difference (this lesson): 1.3 pp ± 1.96×0.00988 = −0.64 pp to +3.24 pp.

Two-proportion z-test (Lesson 3 preview): p̂_pool = 0.0515, z ≈ 1.32, p ≈ 0.19, fail to reject H₀ at 5%.

Alignment rule: when CI for difference includes zero, two-sided test at same α usually fails to reject. Both say "noise plausible."

Business overlay (Lesson 4 preview): 5M annual sessions, $50 profit per order. Point +1.3 pp → $3.25M if real; CI lower −0.64 pp → −$1.6M downside. Statistical and business calls diverge: wide CI demands caution even when point lift looks exciting.

Managers should request all three numbers in one slide: CI, p-value, dollar bounds.

Opening the lesson: why intervals beat binary thresholds

Binary "above or below target" rules discard information. If target conversion is 4.0% and you observe 4.2% with CI 3.5% to 4.9%, you are above target in the sample but not proven above in the population. If next month CI narrows to 3.9% to 4.3%, the same 4.2% point becomes stronger evidence. Intervals encode learning over time; points alone do not.

Key takeaways

• Pair every point estimate with a CI so decision makers see plausible parameter ranges.
• Margin shrinks with √n; halving width requires about four times the sample.
• Use difference CIs for A/B comparisons; individual interval overlap misleads.
• Pick confidence level and n before data collection; stress lower or upper bound by downside direction.
• CIs quantify sampling error, not bias; bad samples yield precise wrong answers.

Confidence intervals are the language of honest forecasting in data-driven firms. When your team adopts a rule that no KPI slide ships without a 95% band and stated n, you eliminate a class of board-level surprises. Pair that rule with Lesson 3 tests when comparing arms, and Lesson 4 dollars when deciding launch.

The 95% confidence level is a convention, not a law. Some teams use 90% for low-risk reversible tests and 99% for pricing that affects millions. Document the level in the experiment charter so results are not re-argued after the fact. When in doubt, report both 95% and 99% bands for metrics with safety or regulatory implications.

1. Compute a 95% CI for a KPI you track and state whether last week's move exceeds the margin.
2. Size a study: what n would hit ±1 pp on a conversion metric you care about?
3. Continue to Lesson 3: Hypothesis Testing.