Experiment Design and Learning Loops: Case Analysis and Recommendations

Integrated case: from messy pilot data to board recommendation

Unit 1 ends with a full case read. You inherit twelve weeks of RelayOps pilot data, qualitative notes, and conflicting stakeholder interpretations. Your job mirrors a founding team preparing a board update: synthesize evidence, state residual risks, and recommend persevere, pivot, or scale with explicit next experiments.

Case analysis is where experiment design meets capital allocation. Numbers without narrative fail fundraising. Narrative without numbers fails operational discipline.

RelayOps is the anchor venture for ENT 402. It sells B2B (business-to-business, software sold to companies rather than consumers) field-service dispatch and scheduling software to mid-market commercial HVAC (heating, ventilation, and air conditioning) companies with 50 to 150 field technicians. Founders Maya Chen (CEO, former dispatch manager) and Jordan Okonkwo (CTO) completed 28 discovery interviews in ENT 401. Those interviews confirmed that dispatch managers lose roughly 14% of revenue to missed appointments, double-bookings, and slow emergency routing. The beachhead segment is commercial HVAC operators in Phoenix and Dallas. Stated willingness to pay in interviews ranged from $89 to $149 per technician per month for software that reliably solves dispatch chaos.

This lesson uses consolidated results from Desert Cool, SunLine, and Valley Air plus Dallas LOI status. Recommendations must reconcile revenue, usage, dispatch time, and runway.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Evidence synthesis across sites

Multi-site cases require separation of pattern vs anecdote. RelayOps asks: do successful sites share traits (owner mandate, emergency volume, champion tenure)? Do failures share traits (low emergency volume, parallel spreadsheet, recent ERP project)?

Build an evidence matrix rows as metrics, columns as sites, plus row for beachhead pattern summary. Highlight metrics that move together vs outliers requiring qual explanation.

SunLine at 52% usage with 6.8-minute median is an outlier on usage but not time. Qual notes: SunLine emergencies only 9/week vs 18 at Desert Cool; dispatchers batch-enter because volume feels manageable. Pattern: wedge weak when emergency volume low.

Weight revenue at risk: losing SunLine (118 techs) hurts logos; persevering on wrong wedge hurts more firms later.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

RelayOps week-12 evidence matrix:

Recommendation frameworks

Board-ready recommendations use situation-complication-resolution structure. Situation: what we tested. Complication: mixed or clear results. Resolution: decision plus next experiment with budget and date.

Quantify economic consequence: renewing Desert Cool and Valley at $99/tech vs losing SunLine. Annual recurring revenue (ARR, yearly subscription revenue run rate) impact: keep 160 techs × $99 × 12 = $190,080 vs full 278 techs × $99 × 12 = $330,264. Gap $140,184 informs how much to invest saving SunLine-like accounts.

Recommend segment refinement when pattern clear: prioritize firms with ≥15 emergencies/week for next sales cycle. De-prioritize low-volume accounts or pitch audit-trail secondary wedge.

Every recommendation lists confirmatory experiment if scaling: Dallas 2-site stagger with segment filter, 8-week read, $60k budget cap.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Stakeholder-specific reads

Owners care about ROI in revenue and overtime. RelayOps translates 4.6-minute median into estimated recovered jobs using ENT 401 14% revenue loss framing: if dispatch errors drive 3 points of that 14%, improving speed recovers fraction of lost revenue. Use conservative assumptions and show math.

Dispatchers care about daily friction. Recommendation includes UX roadmap tied to qual themes, not vague "we will improve UI."

Investors care about repeatable acquisition in beachhead. Two renewals plus segment pattern supports limited scale; SunLine outlier requires pivot playbook not denial.

Engineering cares about stability before new modules. Recommendation sequences tech debt week after renewals signed.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Writing the next experiment charter

Case resolution ends with a charter for the next learning loop: hypothesis, sites, segment filter, metrics, budget, owner, date. RelayOps Dallas charter: 2 firms, ≥15 emergencies/week, 105 and 88 techs, before-after 4+8 weeks, primary usage ≥70%, secondary median ≤5 min, budget $55k, start when Phoenix ARR ≥$150k.

Charters prevent strategy drift into unrelated Dallas enterprise deals that violate beachhead focus.

Include stop rules for expansion: if 0 of 2 Dallas sites hit usage by week 6, pause geographic expansion and fix onboarding nationally.

Link charter ICE score to backlog so team understands priority vs feature requests.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Prepare recommendation after week 12. Cash $177k, burn $45k/mo, renewals pending signature.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Part A: Situation and complication

Situation: three Phoenix pilots completed emergency queue RAT with staggered starts. Complication: 2/3 exceed usage and time gates; SunLine weak on usage (52%) despite moderate time (6.8 min). Qual link: low emergency volume enables whiteboard persistence.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part B: Resolution and economics

Resolution: Persevere on emergency speed wedge for high-volume segment. Pivot offer for SunLine: 60-day audit-trail trial at $79/tech while pausing speed guarantees. Sign renewals Desert Cool + Valley: 160 techs × $99 × 12 = $190,080 ARR. Full cohort would be $330,264; gap $140,184 at risk if SunLine churns.

Next charter: Dallas stagger 2 high-volume sites, $55k, gate week 8. Do not hire sales rep until Dallas 1/2 usage pass.

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part C: Reconciliation

160 × 99 × 12 = 190,080 ✓. 278 × 99 × 12 = 330,264 ✓. Difference 140,184 ✓. $55k next experiment + 3 months $45k burn = $190k > $177k cash, requires renewal signatures before Dallas start ✓ (timing constraint valid).

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Part D: Managerial read

Board: "Cut SunLine and scale sales now." Response: "Scaling before segment pattern costs $140k ARR learning and repeats low-volume failure. Dallas confirmatory test is cheaper than nationwide rep hire."

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

FleetNow (fictional) board deck highlighted best pilot site only. Due diligence uncovered two failed sites hidden in appendix. RelayOps evidence matrix and ARR math prevent selective storytelling.

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Dallas pilot 1 (105 techs, 17 emerg/week) week 6: usage 73%, median 5.2 min. Dallas pilot 2 (88 techs, 11 emerg/week) week 6: usage 58%, median 6.1 min. Recommend persevere, pivot, or stop Dallas expansion with ARR math (use $99/tech/mo).

Rehearse reconciliation checks at the bottom of every worked example the way accountants foot a ledger. RelayOps examples use technician counts, price per seat, weekly emergency volume, and runway months that must multiply consistently. If 92 technicians at $99 per month times three months does not equal the pilot revenue line in the table, the lesson fails its MBA standard even when the narrative sounds plausible.

Customer discovery from ENT 401 is the anchor evidence layer beneath every term in this lesson. Problem validation justifies why RelayOps exists; MVP vocabulary explains how founders test behavior change without pretending interviews predict Monday-morning whiteboard habits. Keep both layers visible when writing gate memos so investors see a chain from 28 interviews to three paid pilots to renewal arithmetic, not a jump from slides to product-market fit slogans.

Solution

Pattern repeats Phoenix: high volume passes (73%, near time target), low volume struggles (58%). Persevere geographic expansion for firms ≥15 emerg/week; stop generic Dallas playbook.

ARR if both renew at $99: 193 × 99 × 12 = $229,284. If only pilot 1 renews: 105 × 99 × 12 = $124,740. Risk $104,544 if low-volume segment not filtered.

Charter update: Dallas sales filter ≥15 emerg/week; offer audit wedge to low-volume. Check: 105+88=193 ✓; 193×99×12=229,284 ✓

Operational vocabulary at RelayOps is measured against Phoenix pilot scorecards, not dictionary completeness. Maya ties each term from this lesson to a field on the weekly dashboard Desert Cool, SunLine, and Valley Air review together. When a dispatch manager says "production ready," the glossary entry lists uptime, silent job drops, and override visibility, not feature parity with ServiceTitan. Jordan links engineering milestones to those same words so pull requests either advance the published RAT or appear on a deferral list with assumption ranks.

Founders should rehearse definitions aloud before customer calls the way finance teams rehearse earnings scripts. If Maya cannot define "live entry" in one sentence with a numeric threshold, dispatchers will not comply consistently. ENT 401 established that mid-market HVAC firms lose roughly 14% of revenue to dispatch chaos; ENT 402 vocabulary explains how MVP tests prove whether RelayOps recovers a measurable slice of that loss without claiming full product-market fit prematurely.

Board members and pilot customers interpret the same English words through different incentives. Owners hear ROI (return on investment, profit or cost savings compared with spend). Dispatchers hear Tuesday-morning friction. Engineers hear technical debt. RelayOps publishes a single learning agenda so "success" always references emergency dispatch time, usage percentage, and renewal intent together rather than whichever metric flatters one stakeholder today.

Document vocabulary changes in the assumption map version history the same way you version pricing. When RelayOps redefines activation from "first login" to "first completed emergency loop," every cohort chart gets a footnote. Without version discipline, teams compare incompatible retention curves and draw wrong scale decisions heading into Dallas expansion or Unit 3 product-market fit measurement.

Key takeaways

• Synthesize multi-site evidence in matrices, not best-site anecdotes.
• Recommendations pair decisions with ARR impact and next experiment charters.
• Segment patterns from outliers refine ideal customer profile.
• Stakeholder reads translate metrics into ROI, UX, and scale narratives.
• Confirmatory expansion tests precede broad sales hiring.

1. Draft a three-paragraph board recommendation for RelayOps using situation-complication-resolution.
2. What segment filter would you add to RelayOps CRM from this case?
3. Continue to Unit 2 Lesson 1: The Strategic Logic of Activation, Engagement and Retention.