A/B Testing

Randomization is how marketing earns causal claims

BrightBrew's onboarding A/B test assigns new signups randomly to email sequence A or B. If 30-day churn is lower in B, the team can attribute the difference to the sequence rather than channel mix or seasonality, within random chance limits. A/B testing (split testing, comparing two versions with randomized assignment) is the workhorse of causal marketing analytics when sample sizes and execution discipline exist.

Without randomization, Sam would compare subscribers who "happened to" get B because they signed up on a weekend when a different campaign ran. That is not a test; it is a biased comparison. At 142,000 subscribers and continuous growth tests, BrightBrew's experiment platform is as important as its ad creative.

BrightBrew is a direct-to-consumer (DTC) specialty coffee subscription company and the anchor company for MKT 202. As of the latest reporting period, BrightBrew serves 142,000 active subscribers with 4.2% monthly churn, average revenue per user (ARPU, average monthly subscription revenue per active subscriber) of $28, and customer acquisition cost (CAC, marketing and sales spend to win one new paying subscriber) near $42. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair run active-subscriber and churned-subscriber survey panels refreshed quarterly, A/B tests on onboarding (welcome email sequence, first-shipment timing, grinder add-on offer), and cohort retention dashboards by signup month, acquisition channel, and plan type.

You met BrightBrew in MKT 201 (Marketing Management) positioning and STP work on BrightBrew's value proposition. This course adds the evidence layer: how to translate marketing decisions into research questions, collect valid qualitative and quantitative data, analyze cohorts and funnels, run experiments, and present recommendations leaders can act on.

This lesson covers unit of randomization, primary metrics, guardrails, and the operational checklist before launch. Statistical power and interpretation continue in Lessons 3 and 4.

What A/B tests can and cannot prove

A/B tests estimate the average treatment effect (ATE, difference in outcome between treatment and control) for the eligible population during the test window. They prove "B beat A on metric M for this population in this period," not "B will beat A forever in all markets."

Tests cannot replace exploratory research on why B won; pair winning variants with interviews.

Ethical guardrails: no deceptive harm; limit holdout duration; support scripts documented.

Randomization unit and contamination

Randomize at the level treatment applies: user ID for onboarding emails, household for mailed inserts, geo for podcast markets. Contamination (spillover, control users exposed to treatment) biases results toward zero. BrightBrew assigns at account ID; shared household accounts flagged to same cell.

Primary, secondary, and guardrail metrics

Pre-register primary metric (30-day churn). Secondary (first brew within 7 days). Guardrails (support tickets, refund rate) ensure wins are not pyrrhic. Changing primary after peeking destroys inferential credibility.

Intent-to-treat and adherence

Analyze all assigned users (intent-to-treat, ITT, includes non-compliers). Per-protocol subsets are biased. If many never open emails, effect dilutes; that dilution is real-world impact.

Experiment ops checklist

Hypothesis, sample size, runtime, assignment hash, logging, QA on variants, stop rules, single-threaded owner. BrightBrew experiment doc template lives with Priya's research plan.

50/50 split new U.S. signups; primary 30-day churn.

Part A: Design

A: legacy 5-email sequence. B: 3-email + SMS nudge. n target 10,000 per arm.

Part B: Results ITT

| Arm | n | 30-day churn | | A | 10,124 | 5.0% | | B | 10,088 | 4.3% | | Diff | n/a | -0.7pp |

Part C: Guardrails

Support tickets per 100 subs: A 2.1, B 2.0. Refunds stable. Check: n totals ✓

Part D: Managerial read

Roll out B if p-value and CI support; monitor grinder attach not degraded.

ShopNow stopped test early at p=0.04 with n=800; effect vanished at n=5,000. BrightBrew uses pre-registered runtime.

A: churn 5.0% n=10,124; B: 4.3% n=10,088. Approximate absolute churners saved per 10k assigns if rollout B.

Solution

Diff 0.7pp → ~70 fewer churners per 10k signups at ITT scale. Check: 0.007×10,000=70 ✓

Why analyze ITT when 30% never open emails?

Solution

ITT reflects real-world policy effect including non-openers; per-protocol would overstate impact for users who would not receive emails in production mix.

Key takeaways

• A/B tests require randomization at the correct unit with contamination controls.
• Pre-register primary, secondary, and guardrail metrics before launch.
• BrightBrew onboarding tests use 30-day churn as primary decision metric.
• Report intent-to-treat effects for rollout decisions.
• Pair winning variants with qualitative why follow-ups.

1. Draft primary and guardrail metrics for BrightBrew SMS onboarding test.
2. Identify contamination risk in a geo podcast experiment.
3. Continue to Lesson 2: Experimental Design.

Applying A/B Testing at BrightBrew scale

When BrightBrew evaluates a/b testing, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align experiments, power, and causal inference with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why a/b testing is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The experiments, power, and causal inference workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for a/b testing. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak experiments, power, and causal inference answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. A/B Testing gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.

Technical mechanics and checks (worked patterns)

For a/b testing, BrightBrew analysts show work the way finance shows reconciliations. A cohort retention table prints signup month, eligible n, month-zero through month-three retention, and a check that weighted plan mix matches the dashboard within one percentage point. A funnel table multiplies step conversions and compares the product to observed month-two actives within rounding tolerance. A survey proportion reports weighted point estimate, ninety-five percent confidence interval, effective n, and exclusion counts from speeders or straight-liners. An experiment appendix lists assignment counts per arm, sample ratio mismatch p-value, intent-to-treat churn with interval, and support ticket guardrail delta.

Use plain-language hypothesis statements before formulas. Example for onboarding: null hypothesis states sequence B does not change 30-day churn versus sequence A; alternative states churn differs. Randomization creates comparable arms so differences after large n are plausibly treatment-related rather than channel mix artifacts. Still verify seasonality with year-over-year cohort comparisons and document concurrent campaigns that could violate independence assumptions.

For spreadsheet or SQL replication, write the grain first. Customer-month tables suit retention. Customer-level tables suit funnel conversion if timestamps exist for each stage. Survey tables suit respondent weights. Experiment tables suit assignment at signup with outcome flags thirty days later. BrightBrew forbids ambiguous one-word metrics like engagement without operational definition. Engagement might mean logged brew events, email opens, or app sessions; each definition implies different SQL joins and different managerial meaning.

Common executive questions (and disciplined answers)

Executives ask short questions that require long disciplined answers. "How sure are we?" maps to confidence intervals, power, and replication plans, not bravado. "What is the dollar impact?" maps to logos saved times contribution margin with explicit stationarity assumptions. "Can we ship faster?" maps to risk of rolling out biased assignment or underpowered tests that will reverse after holiday spend commits. "Why trust panel data?" maps to sampling frame, weighting, response quality rules, and consent governance. "Why not just ask sales?" maps to selection bias and absent counterfactuals in anecdote.

BrightBrew's credible answer format for a/b testing is three bullets: decision recommendation, evidence strength label exploratory descriptive or causal, and next study if limitations matter. A fourth bullet lists what would falsify the recommendation within sixty days. That discipline prevents the analytics team from becoming either a bottleneck or a rubber stamp.

Practice the translation loop until it is habit. Business question to research questions to design to analysis plan to dashboard tile to memo ask. When the loop is complete, BrightBrew scales what survives skepticism. When the loop is broken, the company buys false confidence cheaply and pays for it in subscriber logos later.

Practice extension: self-check without peeking

Before reading any solution in this lesson again, open a blank document and complete four rows. Row one: write BrightBrew's business question that a/b testing helps answer. Row two: list population inclusion and exclusion rules for that question. Row three: name primary metric, one secondary metric, and one guardrail metric. Row four: state the decision you would make if the metric moves favorably versus unfavorably. Compare your rows to the worked example and practice problem. Gaps indicate what to re-read.

If you are studying outside coffee subscriptions, substitute your company but keep numeric discipline. A B2B SaaS team might replace churn with logo retention and ARPU with average recurring revenue per account. A marketplace might replace funnel steps with search, booking, and repeat purchase. The structural habits from MKT 202 remain: define terms, show checks, label evidence mode, and tie results to decisions with explicit limitations.

Connection to MKT 201 and OMBA 102

MKT 201 positioned BrightBrew's value proposition, segments, and channel strategy. MKT 202 adds evidence standards for those strategic choices. OMBA 102 deepens inference, confidence intervals, and decision analysis that underpin experiment readouts and survey precision. Treat the three courses as a stack: strategy names where to play, analytics names how to validate, statistics names how much certainty the data earns.

When you present to executives, integrate the stack in one narrative arc rather than three jargon layers. Example: MKT 201 chose referral emphasis for Ritualist personas; MKT 202 shows referral cohort three-month retention ninety-one percent versus Starter promo seventy-two percent; OMBA 102 quantifies uncertainty on the difference with an interval and sample size plan for continued monitoring. That integrated story is what capstone memos in Unit six require.

Deep dive: metric definitions BrightBrew reuses every week

Active subscriber means a paid account in good standing on the measurement date, excluding paused accounts unless the lesson explicitly includes them. Logo churn counts accounts canceled after the grace window divided by active logos at period start. Revenue churn compares lost monthly recurring revenue to starting MRR, capturing plan downgrades separately when finance requires. 30-day churn for onboarding experiments counts cancels within thirty days of signup payment among intent-to-treat assigns. First brew requires a logged brew event or a shipment consumption signal according to the experiment charter. CAC divides fully loaded acquisition spend by new paying subscribers in the window. LTV in marketing memos uses contribution margin divided by churn only when stationarity is plausible; promo cohorts get cohort-based LTV instead.

These definitions appear boring until someone changes them silently. A five-basis-point definitional shift can fake a win. A/B Testing training includes insisting on definition links in footers. When BrightBrew compares MKT 201 positioning tests to MKT 202 retention outcomes, shared definitions are the chain between strategy and proof.

For experiments, power, and causal inference, also document data sources and refresh cadence. Billing warehouse updates nightly; app events stream hourly; panel surveys batch weekly; experiment assignments log at signup. A dashboard tile without timestamp and owner is a rumor. Priya's team rejects tiles that lack both.

Walk through a numerical reconciliation each month. Subscribers start plus signups minus churned should approximate ending subscribers within known timing differences. Funnel signup counts should match billing new paid within agreed lag. Panel weighted plan mix should match active population within one point. Experiment assigns should match traffic splits within sample ratio mismatch tolerance. Reconciliation does not guarantee truth, but it catches join bugs before executives do.

Managerial judgment prompts for A/B Testing

1. If evidence is descriptive only, what is the cheapest causal next step BrightBrew could run in two weeks?
2. If Sam wants to scale spend now and Priya wants another week of data, what pre-registered rule breaks the tie?
3. Which stakeholder loses most if BrightBrew accepts a false positive on a/b testing?
4. What would a smart skeptic ask about seasonality, selection, or instrumentation?
5. What single guardrail metric would convince you to pause a winning primary metric?

Write ninety-word answers as a memo appendix. Use BrightBrew numbers wherever possible. This exercise converts lesson prose into decision reflexes you will use under time pressure.

Additional study path: compare this lesson's worked example to the practice problem. Identify one assumption that changed and explain how that change alters the decision. Then compare to Unit six capstone memo structure: decision ask, labeled evidence, limitations, next study. Capstone integration is intentional; courses compound when you reuse the same company, metrics, and vocabulary across units.

For international or B2B readers, map BrightBrew's DTC subscription metrics to your domain explicitly rather than mentally translating on the fly. Poor translation at the metric layer causes most "analytics did not help" complaints in organizations. Invest fifteen minutes writing a mapping table once; reuse it across lessons.

Applying A/B Testing at BrightBrew scale

When BrightBrew evaluates a/b testing, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align experiments, power, and causal inference with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why a/b testing is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The experiments, power, and causal inference workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for a/b testing. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak experiments, power, and causal inference answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. A/B Testing gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.