Evaluating Trade-offs in Experiment Design and Learning Loops

Every experiment design trades speed against signal

Founders want fast, cheap, statistically airtight experiments. They get two of three. RelayOps cannot run a 40-site RCT (randomized controlled trial) with 95% confidence in one quarter on a $400,000 runway. They can run three staggered pilots with before-after baselines and honest confound logs. The managerial job is to choose trade-offs explicitly, not pretend perfect science.

This lesson compares speed vs rigor, small-N B2B vs consumer scale, qualitative vs quantitative evidence, and build vs concierge fulfillment. You will leave with a trade-off worksheet RelayOps uses before approving any experiment spend.

RelayOps is the anchor venture for ENT 402. It sells B2B (business-to-business, software sold to companies rather than consumers) field-service dispatch and scheduling software to mid-market commercial HVAC (heating, ventilation, and air conditioning) companies with 50 to 150 field technicians. Founders Maya Chen (CEO, former dispatch manager) and Jordan Okonkwo (CTO) completed 28 discovery interviews in ENT 401. Those interviews confirmed that dispatch managers lose roughly 14% of revenue to missed appointments, double-bookings, and slow emergency routing. The beachhead segment is commercial HVAC operators in Phoenix and Dallas. Stated willingness to pay in interviews ranged from $89 to $149 per technician per month for software that reliably solves dispatch chaos.

Bad trade-offs are invisible until a board member asks why you have "only three customers" and no control group. Good trade-offs are documented before the pilot starts.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Speed vs rigor

Speed measures calendar time to decision. Rigor measures confidence that the observed effect reflects the intervention, not noise. RelayOps emergency MVP targets gate decision by week 8. A rigorous multi-city RCT might take 18 months and $2M, incompatible with runway.

Teams increase rigor without infinite time by tightening metric definitions, logging confounds, and staggered rollouts. They sacrifice rigor consciously when ICE scores show a cheaper test resolves uncertainty enough to act.

Document residual risk after each fast experiment: "We accept 30% chance confounds explain 20% of improvement because heat wave coincided with week 2." Residual risk informs how aggressively to scale sales.

Investors accept speed-rigor trade-offs when founders articulate what would falsify the conclusion and what confirmatory test runs next.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Small-N B2B and statistical power

Statistical power is the probability of detecting a real effect if it exists. With three pilots and ~18 emergencies per week, RelayOps will never behave like a consumer app with millions of clicks. Founders use directional evidence plus triangulation: do time, usage, and revenue tell a consistent story?

Small-N techniques: use medians not means for skewed dispatch times; report confidence intervals honestly ("median 4.9 min, 95% bootstrap CI 4.2 to 6.1 on 54 jobs"); compare baseline to treatment within site before comparing across sites.

Pre-register minimum sample sizes. RelayOps requires ≥20 treatment jobs before primary metric read, derived from simulation that 20 jobs detect 5-minute vs 12-minute shift with acceptable error for gate decisions, not for academic publication.

Avoid p-hacking (running many analyses until one looks significant). One primary metric, one gate read date, amendments logged.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Evidence strength ladder for RelayOps:

Qualitative vs quantitative loops

Quantitative metrics show what changed. Qualitative evidence (interviews, dispatch shadowing, ticket themes) shows why. RelayOps runs paired loops: if usage stalls at 55%, numbers trigger three dispatcher interviews within 48 hours.

Qualitative sample bias warning: angry users talk; silent majority may be fine. Weight qualitative themes against usage distribution, not loudest voice.

Mixed-methods gate: quant fail triggers qual root cause before kill. Quant pass with qual confusion triggers UX iteration even if medians look good (batch entry pattern).

Record qual findings in assumption map as evidence reducing uncertainty on workflow hypotheses.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Build vs buy vs concierge trade-offs

Every experiment chooses fulfillment mode. Build software when repeated runs need automation and behavior must be tested in realistic UI. Concierge when learning is about outcomes before interface exists. Buy (use existing tools) when Zapier plus Airtable falsifies adoption in 48 hours.

RelayOps chose build for emergency console because ENT 401 showed trust requires visible audit trail, hard in spreadsheets. Jordan rejected pure buy because whiteboard IS the incumbent comparator.

Cost comparison: 5-week build ~$56,250 (5/4 × $45k burn) vs 2-week concierge test ~$22,500 plus $3k founder travel. ICE ranked shadowing plus concierge before full build in Unit 0.

Over-building before concierge when UI is not the risk wastes runway. Under-building when UI trust IS the risk produces false negatives.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Option A: three pilots parallel starting same week. Option B: stagger starts by 2 weeks to ship fixes. Engineering velocity: one critical fix per stagger window.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Part A: Trade-off table

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part B: Decision

Choose stagger (B) because treatment fidelity on adoption-sensitive MVP matters more than 4-week acceleration. Duplicate P1 bugs across three dispatch floors could poison beachhead reputation in Phoenix.

Residual risk: slower read delays Dallas expansion one month. Mitigation: begin Dallas LOI conversations during stagger week 6 using Phoenix interim metrics.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part C: Reconciliation

Extra 4 weeks × ($45,000/4) ≈ $45,000 ops burn ✓. Total pilot program ~12 weeks vs 8 ✓. Within $100,000 RAT cap when combined with $88k build from Unit 0 example ✓.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part D: Managerial read

Investor: "Parallel pilots show momentum." Response: "Momentum that breaks three dispatch floors simultaneously destroys referrals in a tight HVAC network. Stagger buys fix-forward learning."

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

StatRoute (fictional) delayed launch 5 months waiting for 200-job sample per site while competitors signed pilots. RelayOps's pre-registered 20-job minimum balances rigor with runway, documented in trade-off memo.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

RelayOps can run 2-week concierge-only test ($12k founder time) before any code, or proceed to build. Concierge predicts usage with 0.6 correlation to software from prior founder experience. Build costs $56k. Runway $177k after pilots. Which path and what residual risk remains?

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Solution

If runway tight and A3 uncertainty still 5, run concierge first ($12k) to drop uncertainty before $56k build. Total $68k vs $56k build-only if concierge fails early saves full build.

Residual risk: concierge lacks UI trust signals; software adoption may be 15-20 points lower than concierge. Plan confirmatory read week 4 software pilot.

Check: $12k + $56k = $68k < $100k RAT cap ✓; $177k runway supports sequence ✓

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Key takeaways

• Document speed vs rigor trade-offs and residual risk before experiments run.
• Small-N B2B uses directional evidence, medians, and triangulation not p-hacking.
• Pair qualitative and quantitative loops when metrics and stories conflict.
• Choose build, concierge, or buy based on which assumption is riskiest.
• Staggered pilots trade calendar time for treatment fidelity in ops software.

1. For a venture you know, name one experiment where speed was over-prioritized vs rigor.
2. What minimum sample would you pre-register for RelayOps dispatch time?
3. Continue to Lesson 4: Experiment Design and Learning Loops: Case Analysis and Recommendations.