Sampling Methods

Your sample is your argument

BrightBrew posted a survey link on Instagram Stories. Sixty-eight percent of completes were under 25, mostly Explorer promo signups. Conclusions about "all subscribers" would fail. Sampling defines who has a chance to be included and whether results generalize to the 142,000 subscriber population.

BrightBrew is a direct-to-consumer (DTC) specialty coffee subscription company and the anchor company for MKT 202. As of the latest reporting period, BrightBrew serves 142,000 active subscribers with 4.2% monthly churn, average revenue per user (ARPU, average monthly subscription revenue per active subscriber) of $28, and customer acquisition cost (CAC, marketing and sales spend to win one new paying subscriber) near $42. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair run active-subscriber and churned-subscriber survey panels refreshed quarterly, A/B tests on onboarding (welcome email sequence, first-shipment timing, grinder add-on offer), and cohort retention dashboards by signup month, acquisition channel, and plan type.

You met BrightBrew in MKT 201 (Marketing Management) positioning and STP work on BrightBrew's value proposition. This course adds the evidence layer: how to translate marketing decisions into research questions, collect valid qualitative and quantitative data, analyze cohorts and funnels, run experiments, and present recommendations leaders can act on.

Probability versus non-probability samples

Probability sampling gives every population member a known chance (random panel draw from billing IDs). Enables margin of error calculations. Non-probability (volunteer, social link) is faster but requires caution generalizing.

BrightBrew's quarterly panel uses stratified random sample from active subscribers with known inclusion probabilities.

Stratification and quotas

Stratify by plan, tenure, channel to guarantee representation. Weight responses to match population totals if completion differs by stratum.

Sample size basics

For descriptive proportions, margin of error shrinks with sqrt(n). Rule of thumb for 95% CI on proportions: MOE ≈ 1.96 × sqrt(p(1-p)/n). At n=1,000 and p=0.5, MOE ≈ 3.1%.

Subgroups need their own n; national survey useless for pod-specific claims if pod n=40.

Coverage and nonresponse

Coverage error: frame excludes groups (corporate gift accounts). Nonresponse error: completers differ from decliners. BrightBrew tracks response rate by tenure; reweights or follows up low-response strata.

Sampling for experiments vs surveys

A/B tests sample future signups with randomization; surveys sample current base. Do not confuse assignment units (user vs household).

Population 142,000 active; target n=2,000 completes.

Part A: Frame

Active paid U.S., email valid, not unsubscribed from research.

Part B: Stratified invite

Classic 60%, Pod 25%, Starter promo 15% targets. Invite 6,000 expecting 33% response.

Part C: Weighting check

Raw pod 28% vs target 25%; apply weight 0.89 on pod completes. Check: weighted plan mix within 1pp of population ✓

Part D: Managerial read

Report weighted estimates with unweighted n per stratum for transparency.

FitApp Twitter poll skewed elite athletes; stratified email draw fixed representation.

BrightBrew wants MOE ≤ 3% on a yes/no question at 95% confidence. Approximate required n using p=0.5. If response rate is 30%, how many invites?

Solution

n ≈ (1.96² × 0.25) / 0.03² ≈ 1,067 completes. Invites ≈ 1,067 / 0.30 ≈ 3,557. Check: round up for dropout ✓

Key takeaways

• Sampling frame and method determine whether you can generalize.
• BrightBrew panel uses stratified probability sampling with weights.
• Compute MOE from completes, not invites.
• Subgroup claims require adequate stratum n.
• Separate sampling logic for surveys vs experiments.

1. Define sampling frame for BrightBrew churned-subscriber survey.
2. Calculate MOE for n=400 and p=0.2.
3. Continue to Lesson 3: Measurement Reliability and Validity.

Applying Sampling Methods at BrightBrew scale

When BrightBrew evaluates sampling methods, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align survey design, sampling, and measurement with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why sampling methods is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The survey design, sampling, and measurement workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for sampling methods. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak survey design, sampling, and measurement answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. Sampling Methods gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.

Technical mechanics and checks (worked patterns)

For sampling methods, BrightBrew analysts show work the way finance shows reconciliations. A cohort retention table prints signup month, eligible n, month-zero through month-three retention, and a check that weighted plan mix matches the dashboard within one percentage point. A funnel table multiplies step conversions and compares the product to observed month-two actives within rounding tolerance. A survey proportion reports weighted point estimate, ninety-five percent confidence interval, effective n, and exclusion counts from speeders or straight-liners. An experiment appendix lists assignment counts per arm, sample ratio mismatch p-value, intent-to-treat churn with interval, and support ticket guardrail delta.

Use plain-language hypothesis statements before formulas. Example for onboarding: null hypothesis states sequence B does not change 30-day churn versus sequence A; alternative states churn differs. Randomization creates comparable arms so differences after large n are plausibly treatment-related rather than channel mix artifacts. Still verify seasonality with year-over-year cohort comparisons and document concurrent campaigns that could violate independence assumptions.

For spreadsheet or SQL replication, write the grain first. Customer-month tables suit retention. Customer-level tables suit funnel conversion if timestamps exist for each stage. Survey tables suit respondent weights. Experiment tables suit assignment at signup with outcome flags thirty days later. BrightBrew forbids ambiguous one-word metrics like engagement without operational definition. Engagement might mean logged brew events, email opens, or app sessions; each definition implies different SQL joins and different managerial meaning.

Common executive questions (and disciplined answers)

Executives ask short questions that require long disciplined answers. "How sure are we?" maps to confidence intervals, power, and replication plans, not bravado. "What is the dollar impact?" maps to logos saved times contribution margin with explicit stationarity assumptions. "Can we ship faster?" maps to risk of rolling out biased assignment or underpowered tests that will reverse after holiday spend commits. "Why trust panel data?" maps to sampling frame, weighting, response quality rules, and consent governance. "Why not just ask sales?" maps to selection bias and absent counterfactuals in anecdote.

BrightBrew's credible answer format for sampling methods is three bullets: decision recommendation, evidence strength label exploratory descriptive or causal, and next study if limitations matter. A fourth bullet lists what would falsify the recommendation within sixty days. That discipline prevents the analytics team from becoming either a bottleneck or a rubber stamp.

Practice the translation loop until it is habit. Business question to research questions to design to analysis plan to dashboard tile to memo ask. When the loop is complete, BrightBrew scales what survives skepticism. When the loop is broken, the company buys false confidence cheaply and pays for it in subscriber logos later.

Practice extension: self-check without peeking

Before reading any solution in this lesson again, open a blank document and complete four rows. Row one: write BrightBrew's business question that sampling methods helps answer. Row two: list population inclusion and exclusion rules for that question. Row three: name primary metric, one secondary metric, and one guardrail metric. Row four: state the decision you would make if the metric moves favorably versus unfavorably. Compare your rows to the worked example and practice problem. Gaps indicate what to re-read.

If you are studying outside coffee subscriptions, substitute your company but keep numeric discipline. A B2B SaaS team might replace churn with logo retention and ARPU with average recurring revenue per account. A marketplace might replace funnel steps with search, booking, and repeat purchase. The structural habits from MKT 202 remain: define terms, show checks, label evidence mode, and tie results to decisions with explicit limitations.

Connection to MKT 201 and OMBA 102

MKT 201 positioned BrightBrew's value proposition, segments, and channel strategy. MKT 202 adds evidence standards for those strategic choices. OMBA 102 deepens inference, confidence intervals, and decision analysis that underpin experiment readouts and survey precision. Treat the three courses as a stack: strategy names where to play, analytics names how to validate, statistics names how much certainty the data earns.

When you present to executives, integrate the stack in one narrative arc rather than three jargon layers. Example: MKT 201 chose referral emphasis for Ritualist personas; MKT 202 shows referral cohort three-month retention ninety-one percent versus Starter promo seventy-two percent; OMBA 102 quantifies uncertainty on the difference with an interval and sample size plan for continued monitoring. That integrated story is what capstone memos in Unit six require.

Deep dive: metric definitions BrightBrew reuses every week

Active subscriber means a paid account in good standing on the measurement date, excluding paused accounts unless the lesson explicitly includes them. Logo churn counts accounts canceled after the grace window divided by active logos at period start. Revenue churn compares lost monthly recurring revenue to starting MRR, capturing plan downgrades separately when finance requires. 30-day churn for onboarding experiments counts cancels within thirty days of signup payment among intent-to-treat assigns. First brew requires a logged brew event or a shipment consumption signal according to the experiment charter. CAC divides fully loaded acquisition spend by new paying subscribers in the window. LTV in marketing memos uses contribution margin divided by churn only when stationarity is plausible; promo cohorts get cohort-based LTV instead.

These definitions appear boring until someone changes them silently. A five-basis-point definitional shift can fake a win. Sampling Methods training includes insisting on definition links in footers. When BrightBrew compares MKT 201 positioning tests to MKT 202 retention outcomes, shared definitions are the chain between strategy and proof.

For survey design, sampling, and measurement, also document data sources and refresh cadence. Billing warehouse updates nightly; app events stream hourly; panel surveys batch weekly; experiment assignments log at signup. A dashboard tile without timestamp and owner is a rumor. Priya's team rejects tiles that lack both.

Walk through a numerical reconciliation each month. Subscribers start plus signups minus churned should approximate ending subscribers within known timing differences. Funnel signup counts should match billing new paid within agreed lag. Panel weighted plan mix should match active population within one point. Experiment assigns should match traffic splits within sample ratio mismatch tolerance. Reconciliation does not guarantee truth, but it catches join bugs before executives do.

Managerial judgment prompts for Sampling Methods

1. If evidence is descriptive only, what is the cheapest causal next step BrightBrew could run in two weeks?
2. If Sam wants to scale spend now and Priya wants another week of data, what pre-registered rule breaks the tie?
3. Which stakeholder loses most if BrightBrew accepts a false positive on sampling methods?
4. What would a smart skeptic ask about seasonality, selection, or instrumentation?
5. What single guardrail metric would convince you to pause a winning primary metric?

Write ninety-word answers as a memo appendix. Use BrightBrew numbers wherever possible. This exercise converts lesson prose into decision reflexes you will use under time pressure.

Additional study path: compare this lesson's worked example to the practice problem. Identify one assumption that changed and explain how that change alters the decision. Then compare to Unit six capstone memo structure: decision ask, labeled evidence, limitations, next study. Capstone integration is intentional; courses compound when you reuse the same company, metrics, and vocabulary across units.

For international or B2B readers, map BrightBrew's DTC subscription metrics to your domain explicitly rather than mentally translating on the fly. Poor translation at the metric layer causes most "analytics did not help" complaints in organizations. Invest fifteen minutes writing a mapping table once; reuse it across lessons.

Applying Sampling Methods at BrightBrew scale

When BrightBrew evaluates sampling methods, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align survey design, sampling, and measurement with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why sampling methods is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The survey design, sampling, and measurement workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for sampling methods. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak survey design, sampling, and measurement answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. Sampling Methods gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.

Technical mechanics and checks (worked patterns)

For sampling methods, BrightBrew analysts show work the way finance shows reconciliations. A cohort retention table prints signup month, eligible n, month-zero through month-three retention, and a check that weighted plan mix matches the dashboard within one percentage point. A funnel table multiplies step conversions and compares the product to observed month-two actives within rounding tolerance. A survey proportion reports weighted point estimate, ninety-five percent confidence interval, effective n, and exclusion counts from speeders or straight-liners. An experiment appendix lists assignment counts per arm, sample ratio mismatch p-value, intent-to-treat churn with interval, and support ticket guardrail delta.

Use plain-language hypothesis statements before formulas. Example for onboarding: null hypothesis states sequence B does not change 30-day churn versus sequence A; alternative states churn differs. Randomization creates comparable arms so differences after large n are plausibly treatment-related rather than channel mix artifacts. Still verify seasonality with year-over-year cohort comparisons and document concurrent campaigns that could violate independence assumptions.

For spreadsheet or SQL replication, write the grain first. Customer-month tables suit retention. Customer-level tables suit funnel conversion if timestamps exist for each stage. Survey tables suit respondent weights. Experiment tables suit assignment at signup with outcome flags thirty days later. BrightBrew forbids ambiguous one-word metrics like engagement without operational definition. Engagement might mean logged brew events, email opens, or app sessions; each definition implies different SQL joins and different managerial meaning.