Questionnaire Design

Bad questions produce confident wrong answers

BrightBrew's first internal survey asked "Don't you agree BrightBrew tastes amazing?" on a 1-5 scale. Eighty-one percent selected 4 or 5. Sam cited it in a board deck. Priya flagged acquiescence bias (tendency to agree) and double-barreled wording (two ideas in one question). The survey could not inform positioning because the instrument was broken before fielding finished.

Questionnaire design is architecture: question order, scales, routing logic, and cognitive load determine whether data is usable. At 142,000 subscribers, even a 2% panel response mis-measured wastes decisions on millions in LTV.

BrightBrew is a direct-to-consumer (DTC) specialty coffee subscription company and the anchor company for MKT 202. As of the latest reporting period, BrightBrew serves 142,000 active subscribers with 4.2% monthly churn, average revenue per user (ARPU, average monthly subscription revenue per active subscriber) of $28, and customer acquisition cost (CAC, marketing and sales spend to win one new paying subscriber) near $42. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair run active-subscriber and churned-subscriber survey panels refreshed quarterly, A/B tests on onboarding (welcome email sequence, first-shipment timing, grinder add-on offer), and cohort retention dashboards by signup month, acquisition channel, and plan type.

You met BrightBrew in MKT 201 (Marketing Management) positioning and STP work on BrightBrew's value proposition. This course adds the evidence layer: how to translate marketing decisions into research questions, collect valid qualitative and quantitative data, analyze cohorts and funnels, run experiments, and present recommendations leaders can act on.

Question types and when to use them

Closed-ended items (multiple choice, scales) enable aggregation. Open-ended items capture language for coding. Behavioral frequency items ask past actions ("How many times did you brew last week?") rather than hypotheticals. BrightBrew's panel mixes closed scales with one open "what almost made you cancel?"

Avoid double negatives and agree-disagree ladders for sensitive topics; use construct-specific scales ("How easy was compatibility check?").

Scale selection and anchors

Odd vs even scales (neutral option), number of points (5 vs 7 vs 11), and labeled anchors change distributions. Use consistent scales across waves for trend analysis. BrightBrew uses 7-point importance scales with labeled endpoints only.

NPS (Net Promoter Score, percent promoters minus percent detractors on 0-10 recommend likelihood) requires the standard 0-10 recommend question; do not alter wording when benchmarking.

Survey flow and fatigue

Place sensitive or boring items after engagement hooks. Use skip logic: pod owners see pod block; Classic Bag skips. Target 8-12 minutes max for panel surveys. Monitor drop-off by question in pilot n=50.

Pretesting and cognitive interviews

Pilot with 5-10 cognitive interviews: think-aloud while answering. Revise ambiguous items before full field. BrightBrew pilot caught "grind size satisfaction" misunderstood by French press users.

Mobile and accessibility

Majority complete on phone. Short pages, large tap targets, avoid matrix grids that horizontal-scroll badly. Offer language plainness (grade 8-10 reading level).

Goal: estimate demand for compatibility checker among active panel.

Part A: Bad draft

"Don't you agree compatibility is easy and pods always work?" (double-barreled, leading)

Part B: Revised items

Q1 past behavior: Have you returned a pod order for compatibility? Y/N. Q2: How confident are you selecting pods for your machine? 1-7. Q3 open: Describe last compatibility issue.

Part C: Pilot

n=40 pilot, drop-off 4% at Q2, median time 6.2 min ✓

Part D: Managerial read

Field full n=1,800 only after pilot fixes; primary analysis pre-registered on Q1-Q2.

CloudDesk 40-row grid caused 62% breakoff at row 12. BrightBrew splits blocks across pages with progress bar.

Rewrite: "How satisfied are you with BrightBrew's amazing speed and great price?" into two non-leading items with defined scales.

Solution

Q1: How satisfied are you with delivery speed? 1 very dissatisfied to 7 very satisfied. Q2: How satisfied are you with value for money? same scale. Check: two constructs, no "amazing" ✓

Key takeaways

• Question wording and order determine whether survey data is usable.
• Prefer behavioral and construct-specific items over agree-disagree ladders.
• Pilot with cognitive interviews before full BrightBrew panel fielding.
• Design for mobile completion and 8-12 minute target length.
• Pre-register primary survey items tied to decisions.

1. Audit five items from a survey you know for leading language.
2. Draft a three-question block for BrightBrew pause-subscription interest.
3. Continue to Lesson 2: Sampling Methods.

Applying Questionnaire Design at BrightBrew scale

When BrightBrew evaluates questionnaire design, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align survey design, sampling, and measurement with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why questionnaire design is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The survey design, sampling, and measurement workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for questionnaire design. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak survey design, sampling, and measurement answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. Questionnaire Design gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.

Technical mechanics and checks (worked patterns)

For questionnaire design, BrightBrew analysts show work the way finance shows reconciliations. A cohort retention table prints signup month, eligible n, month-zero through month-three retention, and a check that weighted plan mix matches the dashboard within one percentage point. A funnel table multiplies step conversions and compares the product to observed month-two actives within rounding tolerance. A survey proportion reports weighted point estimate, ninety-five percent confidence interval, effective n, and exclusion counts from speeders or straight-liners. An experiment appendix lists assignment counts per arm, sample ratio mismatch p-value, intent-to-treat churn with interval, and support ticket guardrail delta.

Use plain-language hypothesis statements before formulas. Example for onboarding: null hypothesis states sequence B does not change 30-day churn versus sequence A; alternative states churn differs. Randomization creates comparable arms so differences after large n are plausibly treatment-related rather than channel mix artifacts. Still verify seasonality with year-over-year cohort comparisons and document concurrent campaigns that could violate independence assumptions.

For spreadsheet or SQL replication, write the grain first. Customer-month tables suit retention. Customer-level tables suit funnel conversion if timestamps exist for each stage. Survey tables suit respondent weights. Experiment tables suit assignment at signup with outcome flags thirty days later. BrightBrew forbids ambiguous one-word metrics like engagement without operational definition. Engagement might mean logged brew events, email opens, or app sessions; each definition implies different SQL joins and different managerial meaning.

Common executive questions (and disciplined answers)

Executives ask short questions that require long disciplined answers. "How sure are we?" maps to confidence intervals, power, and replication plans, not bravado. "What is the dollar impact?" maps to logos saved times contribution margin with explicit stationarity assumptions. "Can we ship faster?" maps to risk of rolling out biased assignment or underpowered tests that will reverse after holiday spend commits. "Why trust panel data?" maps to sampling frame, weighting, response quality rules, and consent governance. "Why not just ask sales?" maps to selection bias and absent counterfactuals in anecdote.

BrightBrew's credible answer format for questionnaire design is three bullets: decision recommendation, evidence strength label exploratory descriptive or causal, and next study if limitations matter. A fourth bullet lists what would falsify the recommendation within sixty days. That discipline prevents the analytics team from becoming either a bottleneck or a rubber stamp.

Practice the translation loop until it is habit. Business question to research questions to design to analysis plan to dashboard tile to memo ask. When the loop is complete, BrightBrew scales what survives skepticism. When the loop is broken, the company buys false confidence cheaply and pays for it in subscriber logos later.

Practice extension: self-check without peeking

Before reading any solution in this lesson again, open a blank document and complete four rows. Row one: write BrightBrew's business question that questionnaire design helps answer. Row two: list population inclusion and exclusion rules for that question. Row three: name primary metric, one secondary metric, and one guardrail metric. Row four: state the decision you would make if the metric moves favorably versus unfavorably. Compare your rows to the worked example and practice problem. Gaps indicate what to re-read.

If you are studying outside coffee subscriptions, substitute your company but keep numeric discipline. A B2B SaaS team might replace churn with logo retention and ARPU with average recurring revenue per account. A marketplace might replace funnel steps with search, booking, and repeat purchase. The structural habits from MKT 202 remain: define terms, show checks, label evidence mode, and tie results to decisions with explicit limitations.

Connection to MKT 201 and OMBA 102

MKT 201 positioned BrightBrew's value proposition, segments, and channel strategy. MKT 202 adds evidence standards for those strategic choices. OMBA 102 deepens inference, confidence intervals, and decision analysis that underpin experiment readouts and survey precision. Treat the three courses as a stack: strategy names where to play, analytics names how to validate, statistics names how much certainty the data earns.

When you present to executives, integrate the stack in one narrative arc rather than three jargon layers. Example: MKT 201 chose referral emphasis for Ritualist personas; MKT 202 shows referral cohort three-month retention ninety-one percent versus Starter promo seventy-two percent; OMBA 102 quantifies uncertainty on the difference with an interval and sample size plan for continued monitoring. That integrated story is what capstone memos in Unit six require.

Deep dive: metric definitions BrightBrew reuses every week

Active subscriber means a paid account in good standing on the measurement date, excluding paused accounts unless the lesson explicitly includes them. Logo churn counts accounts canceled after the grace window divided by active logos at period start. Revenue churn compares lost monthly recurring revenue to starting MRR, capturing plan downgrades separately when finance requires. 30-day churn for onboarding experiments counts cancels within thirty days of signup payment among intent-to-treat assigns. First brew requires a logged brew event or a shipment consumption signal according to the experiment charter. CAC divides fully loaded acquisition spend by new paying subscribers in the window. LTV in marketing memos uses contribution margin divided by churn only when stationarity is plausible; promo cohorts get cohort-based LTV instead.

These definitions appear boring until someone changes them silently. A five-basis-point definitional shift can fake a win. Questionnaire Design training includes insisting on definition links in footers. When BrightBrew compares MKT 201 positioning tests to MKT 202 retention outcomes, shared definitions are the chain between strategy and proof.

For survey design, sampling, and measurement, also document data sources and refresh cadence. Billing warehouse updates nightly; app events stream hourly; panel surveys batch weekly; experiment assignments log at signup. A dashboard tile without timestamp and owner is a rumor. Priya's team rejects tiles that lack both.

Walk through a numerical reconciliation each month. Subscribers start plus signups minus churned should approximate ending subscribers within known timing differences. Funnel signup counts should match billing new paid within agreed lag. Panel weighted plan mix should match active population within one point. Experiment assigns should match traffic splits within sample ratio mismatch tolerance. Reconciliation does not guarantee truth, but it catches join bugs before executives do.

Managerial judgment prompts for Questionnaire Design

1. If evidence is descriptive only, what is the cheapest causal next step BrightBrew could run in two weeks?
2. If Sam wants to scale spend now and Priya wants another week of data, what pre-registered rule breaks the tie?
3. Which stakeholder loses most if BrightBrew accepts a false positive on questionnaire design?
4. What would a smart skeptic ask about seasonality, selection, or instrumentation?
5. What single guardrail metric would convince you to pause a winning primary metric?

Write ninety-word answers as a memo appendix. Use BrightBrew numbers wherever possible. This exercise converts lesson prose into decision reflexes you will use under time pressure.

Additional study path: compare this lesson's worked example to the practice problem. Identify one assumption that changed and explain how that change alters the decision. Then compare to Unit six capstone memo structure: decision ask, labeled evidence, limitations, next study. Capstone integration is intentional; courses compound when you reuse the same company, metrics, and vocabulary across units.

For international or B2B readers, map BrightBrew's DTC subscription metrics to your domain explicitly rather than mentally translating on the fly. Poor translation at the metric layer causes most "analytics did not help" complaints in organizations. Invest fifteen minutes writing a mapping table once; reuse it across lessons.

Applying Questionnaire Design at BrightBrew scale

When BrightBrew evaluates questionnaire design, the team starts from operational facts: 142,000 active subscribers, 4.2% monthly logo churn, $28 ARPU, and $42 blended CAC. VP Marketing Elena Okonkwo, Head of Growth Sam Rivera, and Director of Customer Insights Priya Nair align survey design, sampling, and measurement with Monday dashboard reviews and pre-written research plans. A lesson concept that sounds abstract becomes concrete when tied to signup cohorts, panel waves, and experiment cells logged in the warehouse.

Consider how a 0.5 percentage point change in monthly churn affects BrightBrew. At current scale, that shift moves roughly 710 subscriber logos per month before accounting for mix effects across Classic Bag, Espresso Pod, and Starter Kit promos. Contribution margin near 16.24 dollars per month per subscriber turns small rate changes into five-figure monthly impact. That is why questionnaire design is not an academic exercise for Elena Okonkwo's marketing org; it is how the company avoids scaling a channel that fills the top of the funnel while leaking high-churn promo cohorts at month three.

The survey design, sampling, and measurement workflow at BrightBrew deliberately separates exploratory, descriptive, and causal claims. Priya Nair's analysts label outputs before they reach Sam Rivera's growth standups. Exploratory interview themes become survey items only after codebook review. Descriptive cohort spikes trigger pre-registered experiments rather than same-day pricing changes. Causal A/B wins still require guardrail checks on support tickets, refunds, and grinder attach rates so a churn win does not hide margin erosion. You should copy that labeling habit even if you work outside subscription coffee: name the mode, name the population, name the comparison, and name the decision date before numbers hit a slide.

Document definitions alongside every metric tile. BrightBrew's churn formula specifies grace days after failed payment, pause versus cancel handling, and exclusion of fraud flags. Funnel steps define eligible denominators for visit, signup, shipment, first brew, and month-two active status. Survey estimates document weighting targets by plan mix. Experiment readouts specify intent-to-treat estimands and pre-registered minimum detectable effects on 30-day churn. When definitions live in a shared dictionary, the company builds institutional memory instead of re-debating the same SQL every quarter.

Extended BrightBrew scenario: cross-functional read

Imagine BrightBrew's Q3 review for questionnaire design. Finance asks whether improved onboarding justifies higher podcast CAC. Product asks whether compatibility tooling belongs in mobile web or email only. Operations asks whether shipment forecast accuracy supports green coffee buys. A weak survey design, sampling, and measurement answer addresses only one function. A strong answer shows how evidence flows: qualitative language from churn interviews becomes survey prevalence estimates, descriptive cohort curves localize the leak to Starter Kit promos, and a randomized onboarding test estimates causal churn reduction with confidence intervals translated into saved logos and monthly contribution.

Work the arithmetic on a conservative example. Suppose an onboarding test shows 30-day churn falling from 5.0% to 4.3% intent-to-treat among ten thousand assigns per arm. Absolute reduction 0.7 percentage points yields about seventy fewer churners per ten thousand signups. If BrightBrew adds forty thousand new subscribers in a month after rollout, a sustained effect near the point estimate implies roughly two hundred eighty retained logos in the first month cohort, with compounding value as retained subscribers continue billing. Multiply by monthly contribution near sixteen dollars to communicate magnitude to executives who do not live in p-values. Pair the point estimate with a confidence interval and a pre-written rule: roll out if the interval excludes zero harm on guardrails and includes materially positive churn impact.

Stakeholder conflict is normal. Sam Rivera may push to scale spend while the test is immature. Priya Nair may push to extend runtime for power. Elena Okonkwo must decide under calendar pressure from holiday inventory commits. Questionnaire Design gives you language to negotiate those tensions with evidence quality standards rather than charisma. If power is insufficient, the decision is extend or accept uncertainty, not pretend a noisy week-one lift is definitive. If qualitative sample is thin, the decision is fund ten more interviews, not quote three vivid anecdotes as national truth.

Translate lessons to your own context by replacing BrightBrew names while keeping structure. Pick one decision you face this quarter. Write the business question, three hypotheses, population rules, comparison group, primary metric, guardrails, and inconclusive outcome before collecting new data. If you cannot write those elements, you are not ready to field a survey or launch an experiment regardless of how easy the vendor dashboard makes clicks look.