Forecast Accuracy and Model Validation

A model that explains history can still fail tomorrow

A grocery chain deployed a demand forecast built from 36 months of regression with R² = 0.91. Buyers trusted it until a supplier strike and a viral recipe trend broke category relationships overnight. Warehouses stocked out of olive oil while yogurt expired. The model was not "wrong" on past data. It was unvalidated for decisions that punish error asymmetrically.

In-sample fit (metrics computed on the same data used to train the model) flatters regressions. R², R²_adj, and low training error measure how well coefficients bend to history. They do not automatically measure how well next month's demand, churn, or revenue will be predicted. Model validation estimates performance on unseen data (observations withheld from fitting) that mimics the forward decision. Lesson 2 and Lesson 3 taught you to fit lines and read R²; this lesson teaches the discipline that keeps those numbers honest: train/test splits, error metrics managers can interpret, comparisons to naive baselines, and diagnostics for overfitting.

From Unit 4: Inference and Uncertainty, you already treat sample statistics as uncertain. Validation extends that mindset to forecasts: report MAE (mean absolute error, average absolute gap between forecast and actual*) and RMSE (root mean squared error, square root of average squared error*) on holdout periods, not only coefficients on training rows.

Why in-sample R² misleads deployment decisions

Three mechanisms turn shiny training R² into boardroom embarrassment:

Overfitting: With enough predictors or flexible functional forms, a model can memorize noise. Training R² climbs while future error explodes. Lesson 3's R²_adj penalizes extra k but still uses in-sample data only.

Regime change: Pricing policy, competitors, macro shocks, or product redesign shift the data-generating process. Coefficients fit to 2019–2023 need not govern 2026.

Leakage: Accidentally including future information in training features (e.g., using month-end inventory to predict mid-month sales) inflates in-sample fit and destroys real forecast value.

Managerial rule: No production forecast without reporting test error vs a naive baseline on data the model never saw during fitting.

Train/test validation: mechanics and spreadsheet steps

Holdout validation splits data into:

• Training set: used to estimate β coefficients
• Test set: used only to score predictions Ŷ from frozen coefficients

Random splits work for cross-sectional data (customers, stores at a point in time). Time series (observations ordered in time) require chronological splits: train on earlier periods, test on later periods, because random shuffling leaks future into past.

Spreadsheet workflow (time series example, 24 months):

1. Column A: month_index 1–24
2. Column B: actual Y
3. Column C: predictor X (or multiple X columns for Lesson 3 models)
4. Label rows 1–18 as TRAIN, rows 19–24 as TEST (helper column split)
5. Fit regression using only TRAIN rows (Toolpak Regression on filtered range, or SLOPE/INTERCEPT on train subset)
6. Write coefficients on a small params table; do not refit using test rows
7. In TEST rows, compute Ŷ = β₀ + β₁X (and other β if multiple)
8. Column E: error = Y − Ŷ; Column F: |error|
9. Test MAE = AVERAGE(F19:F24) for test rows only
10. Compare to naive forecast (next section)

Check: test MAE uses exactly six rows if holdout is six months ✓; training fit used exactly 18 rows ✓

Typical split sizes: With 24–60 periods, hold out last 20–25%. With thousands of cross-sectional units, 80/20 random split is common. Small n makes single splits noisy; use rolling validation (below).

Naive baselines: the forecast you must beat

A naive baseline is a simple rule a skeptical operator would try first. If your regression cannot beat it on test data, complexity is unjustified.

Common baselines:

Beating naive on test MAE is minimum bar for deployment. Beating naive on RMSE matters when large errors are especially costly (stockouts, staffing catastrophes).

Error metrics: MAE, RMSE, and MAPE

MAE = (1/m) Σ|Yᵢ − Ŷᵢ| over m test points. Same units as Y; easy to explain: "We miss by $5.5k on average."

RMSE = √[(1/m) Σ(Yᵢ − Ŷᵢ)²]. Penalizes large errors more than MAE. Use when under-forecasting a heat wave by 50 units hurts more than ten misses of 5 units.

MAPE (mean absolute percentage error, average |error / actual|) scales errors relatively. Warning: undefined or unstable when actuals near zero; use sMAPE or avoid MAPE for low-volume SKUs.

Report at least MAE and compare RMSE when tail risk matters. Always state m (test count).

Time-series cross-validation (rolling origin)

Single holdout with m = 6 months is noisy. Rolling-origin validation repeats:

1. Train months 1–12, test month 13 → error₁
2. Train 1–13, test 14 → error₂
3. Continue through series
4. Average test errors across folds

More stable estimate of forward error; costs compute time. Spreadsheet: automate with helper columns or scripts; conceptually identical to repeated train/test with expanding window.

Overfitting diagnostics: train vs test gap

Compare training error to test error on same model specification:

Train great, test terrible signals overfit. Deploy the simpler model unless test error clearly favors complexity.

R² limits on test data: You can compute R² on test set (1 − SS_res_test/SS_tot_test). It may be negative if model loses to the mean benchmark. Negative test R² is a flashing red light.

Connecting validation to managerial decisions

Validation outputs feed decisions, not slides:

• Inventory: RMSE on high-volume SKUs drives safety stock dollars
• Staffing: MAE on daily traffic sets shift buffers
• Capital: If test error swamps projected benefit, delay automation spend

Pair metrics with cost asymmetry: under-forecast vs over-forecast may not be equally expensive. A weighted loss function is advanced; even verbal asymmetry improves judgment.

Data leakage: how train/test splits fail silently

Leakage means training used information from the "future" or from the test target itself. In-sample R² looks excellent; deployment fails.

Managers reviewing analytics should ask: "Could this feature exist before we need the forecast?" If not, leakage is likely.

Train/test split sizes and cross-sectional data

For customer-level models (churn, credit limit), random 80/20 split is common when rows are independent. Use stratified splits if Y is rare (churn 2%): ensure test set has enough positives to score.

For store-week panels, prefer grouped holdout: hold out entire stores or entire recent weeks, not random rows, if within-store correlation is strong.

Document m (test count) and population represented. A test set of six weeks says little about annual seasonality unless those six weeks span diverse conditions.

Rolling-origin cross-validation: spreadsheet sketch

With 24 months in column A and actual Y in B:

1. Helper column fold_test_month = 13, 14, …, 24 for rolling tests.
2. For fold testing month 13: fit regression on rows 1–12; predict row 13; store error in column E.
3. Expand training window: fit 1–13, test 14; continue.
4. =AVERAGE(ABS(error_column)) across folds = rolling MAE.

This averages 12 one-step-ahead errors (months 13–24) instead of one six-month block. More work, more stable estimate. Automate in Python/R in production; spreadsheet sketch proves concept for MBA reviews.

MAPE, sMAPE, and when percent errors mislead

MAPE = (100/m) Σ|Y − Ŷ| / |Y|. Executives like percents. Problems:

• Actual Y near 0 blows up MAPE
• Asymmetric penalty treats under- and over-forecast differently in percent terms
• Low-volume SKUs dominate MAPE rankings noisily

sMAPE (symmetric MAPE) uses denominator (|Y| + |Ŷ|)/2 to stabilize somewhat. Still fragile at zeros.

Managerial default: lead with MAE in units (cases, dollars). Add MAPE only for high-volume lines with floors on Y. Compare MAE to naive in same units.

Integrating validation with regression lessons

A model card template: Training R²_adj, Test MAE, Naive MAE, Train/Test MAE ratio, Last revalidation date, Known regime risks.

When validation says "deploy" vs "pilot"

These thresholds are policy choices; document yours instead of debating R² alone in committee.

Forecast review meeting agenda (30 minutes)

1. Show last period actual vs forecast table (not only chart).
2. Report MAE, RMSE, naive comparison.
3. Flag largest errors; assign ops vs model cause.
4. Confirm no leakage in refreshed training cut.
5. Decide retrain date or hold coefficients frozen.

Repeating this agenda builds organizational memory that validation is ongoing, not a one-time homework exercise from Lesson 4.

Worked extension: computing test R² by hand (FreshMart)

On FreshMart test months 19–24, use actuals [120, 130, 115, 140, 135, 150] (mean Ȳ_test = 131.667).

SS_tot = Σ(Y − Ȳ_test)² = (120−131.667)² + … + (150−131.667)² ≈ 136 + 0.111 + 277.778 + 69.444 + 11.111 + 336.111 ≈ 830.555

Model errors given: 5, 5, −3, 8, 7, 5 → SS_res = 25+25+9+64+49+25 = 197

R²_test = 1 − 197/830.555 ≈ 0.763

Test R² can still look decent while MAE vs naive tells deployment story; report both. If SS_res > SS_tot, test R² goes negative (model worse than mean).

Check: SS arithmetic shown stepwise for audit ✓

Governance: revalidation cadence and ownership

Assign model owner (name, role) and revalidation frequency (monthly for demand, quarterly for churn scores). Owner delivers: test MAE vs naive, largest errors, known regime risks, recommendation freeze vs retrain. R² from training month is not on the agenda.

When test error degrades >20% vs prior quarter, trigger root cause: data pipeline break, competitor, or overfit. Validation is operations hygiene, not a statistics homework artifact.

Spreadsheet template columns for rolling validation log

Fill automatically where possible; human reviews last column monthly. Persistent "no" in beat_naive? retires the model regardless of training R².

Extended FreshMart naive vs model narrative

Buyers often ask why not always use last week (naive). In FreshMart test window, naive suffered at weeks 21 and 22: after a spike to 130, naive forecast 130 while actual fell to 115; after fall to 115, naive stayed high while actual jumped to 140. Regression with trend and promo terms smoothed those turns partially. Neither model is clairvoyant; model won on average MAE because it used X variables naive ignores. Document which X (promo, holiday flag) drove each Ŷ so buyers trust mechanism, not black box.

If training R² was 0.95 but test MAE lost to naive, you would reject deployment despite training glamour. FreshMart case is the happy path; governance rules exist for the unhappy path.

Practice mindset: validation as permissioning

Treat test MAE vs naive as a permission slip, not a grade. Procurement may permission auto-POs only when rolling MAE beats seasonal naive three consecutive months. Marketing may permission spend models only when test error stable after competitor entry. Writing the permission rule before seeing results prevents retroactive goalpost moves when training R² flatters.

Lesson 2 and Lesson 3 deliver coefficients; Lesson 4 decides whether those coefficients earn operational trust. Lesson 5 decides whether changing X is worth the bet economically and causally.

Summary table: which metric to report when

Validation answers reliability of predictions from Lessons 2–3; it does not replace causal design when the decision is to change a lever.

Cross-validation vs single holdout (when to use which)

Single holdout (FreshMart six weeks) is fast and easy to explain. Use when n is moderate and leadership needs one number this week. Rolling-origin CV use when n is short-to-medium and error estimate feels jumpy when you move holdout boundary one month. Full k-fold on cross-sectional data (PayFlow customers) when rows are exchangeable and you need stable MAE for ranking model versions.

Document which method produced the MAE on the model card. Comparing models evaluated with different validation schemes is apples-to-oranges.

Document the validation method in every forecast deck footnote so finance compares models on consistent holdout rules.

Treat test MAE as the headline number in weekly ops reviews; training R² belongs in the appendix for auditors, not the first slide.

FreshMart forecasts weekly unit demand for packaged salads. 24 weeks of data; train weeks 1–18, test weeks 19–24.

Part A: Test period scores

Model fitted on weeks 1–18 only (coefficients frozen before scoring rows 19–24).

Part B: MAE and RMSE (test only)

Errors (actual − model): 5, 5, −3, 8, 7, 5

|errors|: 5, 5, 3, 8, 7, 5 → sum = 33 → MAE = 33/6 = 5.5 units

Squared errors: 25, 25, 9, 64, 49, 25 → sum = 197 → RMSE = √(197/6) ≈ 5.73

Naive |errors|: |120−110|, |130−120|, |115−130|, |140−115|, |135−140|, |150−135| = 10, 10, 15, 25, 5, 15 → sum 80 → MAE_naive = 80/6 ≈ 13.33

Check: model MAE 5.5 < 13.33 naive ✓ → complexity justified vs last-value rule on this holdout

RMSE_naive: errors 10,10,−15,25,−5,15 → squares sum = 100+100+225+625+25+225 = 1300 → √(1300/6) ≈ 14.7

Model wins on RMSE too; week 22 large miss still visible (RMSE > MAE).

Part C: Spreadsheet steps recap

1. Fit regression Y ~ trend + promo_flag on rows 2–19 (weeks 1–18)
2. Copy β₀, β₁, β₂ to params cells
3. Rows 20–25 (weeks 19–24): Ŷ via formula, no refit
4. =AVERAGE(ABS(actual-forecast)) on test block → MAE
5. Parallel column for naive; compare in one summary table for buyers

Part D: Managerial read

Buyers can use the model for short horizon orders but should keep ~6-unit MAE buffer on salad category. R² on weeks 1–18 alone would omit week 22 shock visibility. Revalidate after promo calendar changes. Question for ops: "If RMSE doubles in summer, what expiration write-off does that imply?"

PayFlow scores 30-day churn risk for 2,000 SMB (small and medium business) customers using Lesson 3 regression. Random 80/20 split: 1,600 train, 400 test (not time-ordered; churn risk is cross-sectional snapshot).

Part A: Training vs test metrics

Naive test MAE (predict train mean churn for everyone): 0.071

Part B: Read

Model beats naive on test MAE (0.058 vs 0.071) ✓ but train/test gap signals mild overfit or population shift. Test R² = 0.31 much lower than train 0.44; deploy with recalibration schedule.

Part C: Spreadsheet steps (cross-sectional)

1. Column split: RAND() < 0.8 → train, else test (or stratified by churn flag).
2. Filter train rows; Toolpak Regression on train only.
3. Apply β to test rows; compute test MAE = AVERAGE(ABS(actual-predicted)) on test filter.
4. Naive column: test rows all get train mean Ȳ_train.

Check: 1600 + 400 = 2000 ✓

Part D: Managerial read

PayFlow can use scores for call routing if test MAE gain justifies ops cost. Do not claim R² = 0.44 to investors; quote test MAE vs naive. Revalidate quarterly (Lesson 5: scores predict; interventions need experiments).

LumenDrink built two models on 36 months of marketing mix data, same train/test split (30 train, 6 test).

Part A: Read

High train R² = 0.92 with catastrophic test R² = −0.15 and MAE 19.4 vs 8.2 linear. Polynomial memorized noise (overfit).

Part B: Spreadsheet check

Compute SS_tot on test months from test mean actual; SS_res from test errors. Negative test R² means model loses to predicting test mean always.

Part C: Managerial read

Deploy linear for budget scenarios; reject polynomial despite training glamour. Collect more months before adding flexible terms. Lesson 5 reminder: even validated forecasts are not causal effects of spend.

Check: test m = 6 stated consistently ✓

Three-month test holdout:

Actual: [50, 52, 48]

Forecast: [51, 50, 49]

Tasks:

1. Compute errors, MAE, and RMSE.
2. Naive last-value from prior actual 49 forecasts [49, 50, 52] for months 1–3 of test (given). Compute naive MAE.
3. Which model wins? If over-forecast cost is 2× under-forecast cost, which metric story might change?

Solution

Errors (actual − forecast): −1, +2, −1

|errors|: 1, 2, 1 → sum 4 → MAE = 4/3 ≈ 1.33

Squared errors: 1, 4, 1 → sum 6 → RMSE = √(6/3) = √2 ≈ 1.41

Check: MAE ≤ RMSE always ✓

Naive errors: 50−49=1, 52−50=2, 48−52=−4 → |1|,|2|,|4| sum 7 → MAE_naive = 7/3 ≈ 2.33

Model MAE 1.33 beats naive 2.33 ✓

Asymmetric cost: RMSE penalizes the naive month-3 miss (4 units) heavily; weighted business loss might favor model more than MAE alone if large under-forecast on month 2 (actual 52, forecast 50) is costly. Document cost weights when stakes asymmetric.

Your team reports R²_adj = 0.78 on n = 60 months for a revenue model with k = 8 predictors. Test MAE is $120k; naive seasonal naive MAE is $115k.

Tasks:

1. Should you deploy? Explain in a paragraph a CFO would accept.
2. Name two validation steps you would run next.
3. How can R²_adj be high while test MAE loses to naive?

Solution

1. Deploy? Not yet on accuracy grounds: holdout MAE exceeds seasonal naive, so the model does not beat a simpler benchmark forward despite strong in-sample R²_adj. CFO read: "The model fits history well but does not win where it counts (next quarters). Fix specification, reduce k, or enrich seasonality before trusting automated forecasts."

2. Next steps: (a) Rolling-origin CV across 60 months for stable error; (b) Error segmentation by product line to see where model fails; (c) try seasonal naive + regression adjustment hybrid.

3. High R²_adj, bad test MAE: In-sample fit includes months used to tune predictors; overfitting with k = 8; regime shift in test months; leakage removed after audit; naive seasonal rule genuinely strong for this business.

A demand model uses R²_adj = 0.81 on 36 months with k = 6 predictors. Procurement wants to rely on it for 90-day orders. Rolling CV average MAE beats seasonal naive by 8%. A competitor enters in month 34.

Tasks:

1. Should procurement rely without change? Write a three-sentence memo.
2. Which Lesson 5 concern is irrelevant here, and which Lesson 4 concern dominates?
3. Suggest one model governance rule.

Solution

1. "Model beat seasonal naive historically by 8%, supporting cautious use with safety stock buffers. Competitor entry in month 34 likely violates stationarity; revalidate on post-entry holdout before locking 90-day POs. Until revalidation, blend model with naive and widen buffers on affected SKUs."

2. Lesson 5 causation (price intervention) is secondary to Lesson 4 forecast drift; regime change dominates.

3. Rule: auto-revalidate when any competitor launch or >10% price policy change occurs; freeze coefficients until new test MAE reported.

Key takeaways

• Train/test validation freezes training coefficients and scores unseen periods; time series split chronologically.
• Report test MAE/RMSE and compare to naive baselines; training R² is not deployment accuracy.
• RMSE emphasizes large errors; MAPE needs care with zero/low actuals.
• Rolling-origin CV reduces luck in single holdouts; train/test error gaps reveal overfitting.
• Test R² can be negative; validation aligns models with decisions that pay for error.

1. Backtest your organization's main forecast at least six months out-of-sample; report test MAE vs naive.
2. Plot train vs test error for your current model. Is there an overfit gap?
3. Continue to Lesson 5: Causality, Confounding, and Experimental Design.