Understanding Experiment Design and Learning Loops

Experiments are contracts with reality

A startup experiment is not a science fair project. It is a contract: if we expose belief X to condition Y, we will observe metric Z within timeframe T, and we will decide D based on preset thresholds. Without that contract, teams collect anecdotes and call them validation.

ENT 402 Unit 1 teaches experiment design for ventures that already completed discovery. RelayOps knows dispatch chaos costs revenue. The experiment asks whether RelayOps changes behavior and outcomes in live HVAC operations, not whether people like slides.

RelayOps is the anchor venture for ENT 402. It sells B2B (business-to-business, software sold to companies rather than consumers) field-service dispatch and scheduling software to mid-market commercial HVAC (heating, ventilation, and air conditioning) companies with 50 to 150 field technicians. Founders Maya Chen (CEO, former dispatch manager) and Jordan Okonkwo (CTO) completed 28 discovery interviews in ENT 401. Those interviews confirmed that dispatch managers lose roughly 14% of revenue to missed appointments, double-bookings, and slow emergency routing. The beachhead segment is commercial HVAC operators in Phoenix and Dallas. Stated willingness to pay in interviews ranged from $89 to $149 per technician per month for software that reliably solves dispatch chaos.

Jordan wants to A/B test button colors. Maya wants proof dispatch time falls from 12 minutes toward 5. This lesson gives vocabulary to design experiments that settle founder debates and survive investor diligence.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Hypothesis structure and falsifiability

A usable hypothesis states population, intervention, comparator, outcome, and timeframe. RelayOps example: "For dispatch managers at 50-to-150 tech commercial HVAC firms in Phoenix, using the RelayOps emergency console (intervention) versus whiteboard baseline (comparator) will reduce median emergency dispatch time from 12 minutes to ≤5 minutes within 30 days of rollout (timeframe)."

Falsifiability means the test can fail clearly. If success is defined as "customers feel happier," failure is ambiguous. If success is median time ≤5 minutes on ≥20 jobs, failure is measurable. Ambiguous hypotheses let teams persevere on vibes.

Hypotheses should link to one primary assumption from the map. Secondary metrics (dispatcher satisfaction surveys) inform iteration but do not override primary kill criteria without a written protocol change.

Document null hypothesis explicitly: RelayOps does not improve dispatch time vs baseline. Statistical rigor varies in B2B small-N settings, but conceptual nulls keep teams honest when they cherry-pick best days.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Treatment, control, and baseline periods

Treatment is the new experience: RelayOps console mandatory for emergency jobs. Control is the counterfactual: whiteboard and phone tree without RelayOps. Baseline period records pre-intervention metrics so improvement claims compare against actual history, not memory.

Perfect randomized control trials (RCTs, experiments that randomly assign subjects to treatment or control) are rare in early B2B pilots because withholding software from half a dispatch floor is politically impossible. RelayOps uses before-after design with baseline week: measure 12-minute median from call logs, deploy console, measure weeks 2 to 5.

Interrupted time series improves before-after by tracking weekly medians across baseline and treatment. A downward trend after deployment strengthens causal claims versus one snapshot.

Confounds must be logged: heat wave spikes emergency volume, new dispatcher hire, competitor technician poaching. Experiment logs note confounds so gate decisions discount or delay reads when external shocks dominate.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Design choices for RelayOps pilot:

Metrics: primary, guardrail, and diagnostic

Every experiment declares one primary metric tied to the RAT. RelayOps primary: median emergency dispatch time from job creation to technician en route. Guardrail metrics must not degrade catastrophically: customer complaint count, missed job rate, dispatcher overtime hours.

Diagnostic metrics explain mechanism: percent jobs entered live, average clicks to assign, SMS confirmation latency. Diagnostics guide iteration when primary metric misses but guardrails hold.

Vanity metrics (site visits, demo counts) are excluded from gate decisions. Leading indicators like daily active dispatchers predict lagging retention if usage habits form.

Define metric calculation in writing before launch. Dispatch time starts at job record creation or at customer call timestamp? RelayOps chooses call timestamp from phone system export to prevent gaming.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Learning loops and documentation discipline

A learning loop closes when the team updates the assumption map, publishes a decision memo, and selects the next experiment. Open loops (pilots running without weekly reads) waste runway and erode customer trust.

RelayOps weekly pilot read: usage percent, median time, confounds, support tickets. Week 4 gate uses cumulative data, not last-day spikes.

Experiment documentation includes protocol, raw data location, analysis script owner, and decision outcome. Future fundraising diligence asks for reproducibility, not hero stories.

Loops connect across units: MVP scope choices (Unit 0) set what experiments can measure. Activation metrics (Unit 2) explain why usage stalls even when dispatch time improves.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Desert Cool agrees to 4-week baseline plus 8-week treatment. Average 18 emergency jobs per week. Target read at 20+ treatment jobs.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Part A: Hypothesis and thresholds

H1: RelayOps reduces median dispatch time from baseline 12.0 minutes to ≤5.0 minutes by treatment week 8. Kill: if treatment week 4 median >9.0 minutes AND usage <60%, pause. Primary metric: median minutes call-to-en-route. Guardrail: missed jobs ≤ baseline 2.1/week.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part B: Sample and timeline math

Baseline 4 weeks × 18 jobs = 72 jobs (median stable). Treatment weeks 1-4: 72 jobs expected. Gate read at job 60+ treatment jobs ≈ week 4. Treatment weeks 5-8: additional 72 jobs for success read at 20+ jobs requirement exceeded.

If week 4 median is 6.2 min with 74% usage, guardrails hold: iterate onboarding, do not kill. If median 10.1 min at 55% usage: kill criteria met.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part C: Reconciliation

18 jobs/week × 4 = 72 ✓. 72 treatment jobs > 20 minimum ✓. Baseline 12 min from ENT 401 ✓. Kill 9 min and 60% usage consistent with Unit 0 gates ✓.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part D: Managerial read

Customer question: "Can we skip baseline to go live faster?" Answer: "Baseline costs 4 weeks but prevents arguing about memory. Without it we cannot prove ROI to your owner in dollars or minutes."

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

RouteBuddy (fictional) "experimented" by asking technicians if they liked a new app. 82% said yes. Dispatch times unchanged. The hypothesis had no comparator or time bound. RelayOps's written protocol would reject that design before build.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

RelayOps debates primary metric: (M1) median dispatch time or (M2) percent jobs entered within 2 minutes of call. Treatment week 3 shows M1 at 5.8 min (good) but M2 at 41% (bad). Guardrails fine. Which metric should have been primary for RAT on adoption? What experiment change follows?

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Solution

RAT on adoption is desirability: dispatchers must use console live. M2 is the better primary for that RAT; M1 is outcome that depends on usage quality. M1 good with M2 bad suggests batch entry after calls, which understates real-time value.

Follow-up experiment: shadow 3 dispatch sessions, add UI prompt requiring call timestamp sync, re-run weeks 4-6 with M2 primary target ≥70% within 2 minutes. Keep M1 as guardrail ≤7 min.

Check: 41% < 60% kill usage threshold from Unit 0, iterate not stop ✓

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Key takeaways

• Experiments are contracts with falsifiable hypotheses and preset decisions.
• Treatment, control, and baseline designs must fit B2B political realities.
• One primary metric, explicit guardrails, diagnostics for iteration.
• Learning loops close with documented decisions and updated assumption maps.
• RelayOps measures minutes and usage, not demo applause.

1. Write a full hypothesis sentence for one RelayOps assumption using population-intervention-comparator-outcome-timeframe.
2. Name two confounds that could distort a Phoenix HVAC summer pilot.
3. Continue to Lesson 2: How Experiment Design and Learning Loops Works in Practice.