Data Quality, Missingness, and Bias

Garbage in, gospel out

Modern BI (business intelligence, reporting and dashboard tools) renders imperfect data with perfect typography. A regional health insurer nearly expanded a telehealth benefit because post-visit surveys showed 92% satisfaction among respondents. What the slide omitted: only 8% of members responded, and dissatisfied members were three times likelier to skip the survey. The true population satisfaction was unknowable from the export, but the missingness pattern screamed bias. The decision waited; a redesigned mixed-mode survey (app + mail) with incentives produced a credible 71% top-box estimate and a much smaller program.

From Lessons 1 and 2, you know how to ask decision-grade questions and classify fields by scale and structure. This lesson asks a prior question: should you trust the rows at all? Data quality is managerial risk control, not an IT hygiene task. Missingness is often informative. Bias describes systematic gaps between the data you have and the population the decision affects.

Before interpreting any analysis, ask: How was this data generated, and what could make it wrong? That question sounds simple. It is the difference between a lender trusting portfolio loss rates and discovering months later that charged-off loans were excluded from the export because of a filter bug.

Quality problems rarely announce themselves as missing cells. They appear as confident charts. A VP (vice president) of Product sees "feature adoption up 30%" when the instrumentation change logged any click as adoption. Lesson 2 warned that changing instruments creates false trends. This lesson connects instrument change to accuracy and definition drift.

Dimensions of data quality

Quality is multidimensional. A dataset can be timely yet biased, consistent yet incomplete. Before interpreting any analysis, walk these dimensions explicitly.

Accuracy asks whether values reflect reality. Wrong shipping addresses, mis-keyed refunds, and test accounts in production exports destroy accuracy. Completeness asks whether values are present or missing, and whether absence is random or patterned. Consistency asks whether the same concept uses the same definition across systems (finance "customer" vs support "user"). Timeliness asks whether the data is fresh enough for the decision clock. Uniqueness asks whether one real-world entity maps to one row (duplicate customers inflate counts). Validity asks whether values obey business rules (negative ages, end date before start date).

A dashboard can pass freshness checks and still mislead if the underlying collection process excludes the people the decision affects. Quality work begins with documenting how data was generated: sensor, user self-report, sales rep entry, algorithmic inference.

Document answers in the memo footnote. Credibility rises when limitations sit beside conclusions, not in an appendix nobody reads.

Missingness is information, not just blank cells

Missing data is not neutral noise. It carries signals about collection design, user behavior, and power. Statisticians classify missingness patterns to guide fixes (and to warn when fixes lie).

MCAR (missing completely at random): the probability of missingness does not depend on observed or unobserved values. Rare in business. A random sensor failure might approximate MCAR.

MAR (missing at random): missingness relates to observed variables but not the missing value itself after conditioning on observables. Example: high-income customers skip optional income, but you observe segment and zip; if segment is recorded, estimates may be recoverable with care.

MNAR (missing not at random): missingness depends on the unobserved value itself. Example: dissatisfied customers do not answer surveys; low performers skip self-assessments. MNAR is common and dangerous. The average of respondents is not an estimate of the population.

Managers should ask: "Who is missing, and why?" not only "What percent blank?" A 2% missing rate among enterprise accounts may matter more than a 20% missing rate among free-tier hobbyists if the decision is enterprise renewal.

Imputation (filling blanks with means, medians, or model predictions) can help exploratory work but injects assumptions. For high-stakes decisions, prefer transparency: report results with complete cases, with imputation, and with bounds; show whether the decision flips.

In spreadsheets, blank cells are visible. In warehouses, missingness hides as NULLs that AVG() silently skips. Always report n and percent missing alongside means. SQL pattern: SELECT AVG(score), COUNT(score) AS n_nonnull, COUNT(*) AS n_total FROM survey. If n_nonnull / n_total is 0.22, the mean is based on a selective fifth of the population.

Weighting adjustments are advanced but worth requesting when stakes are high. If you know population gender mix but respondents skew male, survey weights can reduce (not eliminate) bias. Document weights in the footnote.

Selection bias and coverage gaps

Your dataset is rarely the full population (from Lesson 1). Selection bias enters when inclusion rules correlate with the outcome you care about.

Survivorship bias studies only entities that lasted. Analyzing only customers who renewed tells you nothing about why others left. Self-selection means participants opt in. Beta testers who love gadgets will praise a hardware app. Convenience samples survey only active app users while silent churners vanish. Reporting bias means frontline teams log wins and skip losses (pipeline optimism).

Coverage asks what percent of the target population appears in the table. If only digital-channel buyers are measured, a brick-and-mortar expansion decision is blind. If only logged-in users generate NPS (Net Promoter Score, percent promoters minus percent detractors), you measure fans, not the market.

Mini-pattern: "Power users love the feature" from an in-app prompt. Non-users never saw the prompt. The data question from Lesson 1 fails before SQL runs because the population is wrong.

Measurement bias and definition drift

Even complete data can lie if the yardstick moves. Measurement bias arises from how you ask, how you instrument, or how humans comply.

Leading survey wording inflates agreement. Financial estimates round toward targets. Social desirability pushes employees to rate managers kindly in open calendars. Definition drift means the same word changed meaning: marketing "lead" in 2024 may not equal marketing "lead" in 2026 after a CRM redesign.

Instrument changes create false trends. A new mobile app may log sessions differently, spiking "engagement" without behavior change. A support team reclassifying P3 tickets as P2 raises severity rates without worse service.

Managers should insist on a one-sentence metric definition with owner, source system, and last change date. Debates without shared definitions are theatre.

When marketing changes the "qualified lead" rule mid-quarter, finance must decide whether to restate prior quarters or split the series at the change date. Restatement is painful but honest. Silent changes teach the organization that numbers are negotiable, which destroys the coordination value of data described in Lesson 1.

Data quality triage for managers

Before a major capital or policy decision, run a 15-minute audit. It is cheap compared with a wrong launch.

1. Source: Is this the system of record or a convenience export?
2. Definition: Do finance and operations agree on the metric sentence?
3. Coverage: What percent of the decision population is represented?
4. Missingness: What fields are blank; who is blank?
5. Uniqueness: Are join keys trustworthy; do duplicates exist?
6. Outliers: Are extremes errors (fat finger) or real whales (keep but label)?

Record findings in the decision memo. If the biggest risk is MNAR survey response, say so and fund better collection instead of debating decimal points.

The 15-minute audit scales to acquisitions and vendor selection. During diligence, ask for a sample export and run uniqueness and definition checks before trusting a seller's "active user" claim. Many deals discover that "active" meant "logged in once lifetime." That is a consistency failure worth millions in revised price.

Fixing, flagging, and sensitivity analysis

Not every flaw is fixable before the meeting. Options:

• Exclude with documented rules (complete-case analysis)
• Impute with documented method and risk of bias
• Weight respondents to match known population margins (survey science)
• Sensitivity analysis (does the decision change if assumptions swing?)
• Delay and collect better data if cost of error exceeds delay cost

The managerial mindset is robustness: decisions that flip on one imputation choice are weak decisions. Push for ranges and stress tests.

Spreadsheet path: duplicate the analysis tab three ways (complete cases, imputed median, pessimistic scenario). SQL path: run the same aggregate with WHERE email IS NOT NULL vs left join imputation table; compare. Check lines: row counts printed beside every headline rate.

Lineage, ownership, and the data quality memo

Every metric should trace to a lineage story: source system, extraction time, transformations, joins, and owner. Lineage is how you catch definition drift before it becomes a trend story on CNBC.

Assign a data owner per domain (sales pipeline, product usage, finance revenue). Owners approve definition changes and sign quality memos before major decisions. The memo is one page: business question (Lesson 1), population, known biases (this lesson), scale notes (Lesson 2), grain (preview Lesson 4), and sensitivity results.

When two departments debate a number, the first deliverable is not a deeper model. It is a side-by-side definition table. You will often discover they are not arguing about outcomes; they are arguing about labels.

Anomaly detection is not a substitute for governance. A spike in orders may be real (viral campaign) or a duplicate import. Flag anomalies, then investigate with quality dimensions rather than auto-deleting rows.

For surveys, track response rate by segment over time. A rising headline score with falling response among unhappy segments is MNAR alarm, not celebration.

Outliers: errors, whales, and policy choices

Outliers are extreme values. Some are fat-finger errors (an extra zero). Some are whales (one enterprise order worth millions). Deleting all outliers is as dangerous as keeping all blindly.

Workflow: plot or sort extremes; validate against source documents; if error, fix upstream; if whale, report results with and without whales labeled. Finance often requires whale reporting separately because means are not decision metrics for revenue recognition.

In SQL, use QUALIFY or subqueries to list top 20 rows by amount for manual review before aggregating. In spreadsheets, conditional formatting above three standard deviations is a triage flag, not an auto-delete rule.

Linking quality to decisions under time pressure

Executives often say "we decide Friday" while data is messy. The triage path is not perfection. It is bounded risk. Document the biggest known flaw, quantify direction of bias if possible (MNAR surveys likely understate dissatisfaction), and show whether the decision flips under plausible correction.

Example framing for the board: "Renewal recommended if downtime improvement exceeds 10% even if survey bias makes satisfaction look 5 points too high." That sentence connects Lesson 1 decisions to Lesson 3 limits.

When quality fails the decision threshold, the right output is sometimes "delay and collect" with a cost estimate of delay vs error. That is leadership, not analyst obstruction.

Building a data quality culture

Quality improves when teams measure it. Track simple quarterly stats: duplicate rate in CRM, percent of tickets missing product area, survey response rate by segment, count of definition changes without version notes. Publish trends to executives, not only analysts.

Celebrate catches, not only launches. When an analyst blocks a bad metric from a board deck, treat it as risk avoided. Lesson 1 decisions depend on this culture; otherwise translation work lands on biased exports no matter how clever the SQL.

Quality metrics belong in operating reviews the same way revenue and churn appear. A rising duplicate rate is an early warning of integration debt before a major acquisition closes.

Duplicate detection mechanics (spreadsheet and SQL)

Duplicates hide in merges. Detection patterns:

Spreadsheet: sort by email, conditional highlight where email equals cell above; count flagged rows. Compare COUNTA(email) vs UNIQUE(email).

SQL: SELECT email, COUNT(*) AS n FROM leads GROUP BY email HAVING COUNT(*) > 1. Resolve by keeping earliest created_at unless business rules say otherwise; log how many rows removed.

After dedupe, rerun headline metrics. If conversion changes more than one point, duplicates were material. NovaCredit's 18% to 11.2% story is typical when duplicates inflated numerators.

ApexFleet operates 1,200 leased vans. Leadership considers extending a third-party maintenance contract costing $1.4M annually. Vendor cites "94% satisfied fleet managers" from email surveys.

Part A: Setup and audit

Business question: Should we renew the maintenance contract?

Data available: Email survey n=148 responses; CRM lists 380 fleet manager contacts; service tickets and downtime logs for all vans.

Quality audit:

Missingness concern: Non-respondents disproportionately manage vans with high downtime in prior quarter (observable from tickets). MNAR likely: unhappy managers avoid survey.

Part B: Bias-adjusted read

Compare downtime for responders vs non-responders (observable):

Check: 145 + 235 = 380 CRM contacts ✓.

Survey satisfaction among responders = 94%. Population downtime suggests non-responders experience worse service. A naive satisfaction headline is not decision-grade.

Part C: Decision-grade data question (Lesson 1 link)

Translate to: "Among all 1,200 vans, did vendor-linked downtime hours per van change after contract start vs prior year, controlling for mileage?" Use ticket logs (ratio data, panel structure from Lesson 2). Result: vendor-linked downtime down 8% company-wide, but down 22% for responder fleets and up 3% for non-responder fleets.

Part D: Managerial read

Do not renew solely on survey applause. Renegotiate SLAs (service level agreements, contracted performance standards) with penalties tied to downtime per mileage, weighted across all vans not only vocal managers. If vendor refuses, run a competitive bid with pilot on high-downtime depots. Document MNAR survey bias in the board memo footnote.

Part E: Sensitivity on contract decision

If vendor-linked downtime is 10% worse than modeled for non-responder fleets, NPV (net present value, value of cash flows discounted to today) of renewal turns negative. Decision is robust to moderate miss but not to full neglect of non-responders. Collect ticket-based KPIs monthly regardless of survey waves.

NovaCredit, a fintech lender, evaluates a marketing funnel dashboard showing 18% lead-to-application conversion in May 2026.

Part A: Quality failures discovered

• Definition drift: Marketing "lead" includes web form fills; sales "qualified lead" requires income field complete. Dashboard mixes both.
• Duplicates: Two CRM imports created 12% duplicate emails; conversions counted twice.
• Selection: Only digital leads appear; partner branch referrals missing from export.
• Validity: 4% of rows show income = 0 placeholder treated as real.

Part B: Spreadsheet remediation (described)

Step 1: dedupe on email keeping earliest created_at. Step 2: split metrics: marketing_lead flag vs sales_qualified flag. Step 3: exclude income=0 placeholders from qualified funnel. Step 4: add partner feed join on referral_code.

Recomputed qualified conversion: 11.2%, not 18%.

Checks: deduped rows 88,400 vs raw 100,500 ✓; qualified numerator + disqualified sum to filtered base ✓.

SQL plain English: WITH dedup AS (...), qualified AS (SELECT * FROM dedup WHERE income_valid_flag=1) SELECT COUNT(application_id)/COUNT(lead_id) FROM qualified WHERE month='2026-05'.

Part C: Managerial read

Pause ad spend scaling. Fix data pipeline before budgeting Q3. Assign named owners: marketing owns top-of-funnel counts; sales ops owns qualified definition; data engineering owns dedupe keys. The cost of bad quality here is paid media aimed at the wrong conversion rate.

Part D: Weekly quality standup (15 minutes)

NovaCredit instituted a weekly standup: any metric moving more than 5% week over week requires a lineage note before it enters the executive email. Within three weeks, two "wins" were downgraded to data fixes. The ritual cost almost nothing. The saved spend on mis-targeted ads exceeded the analyst time.

GreenPeak Hotels emailed post-stay surveys in June 2026. Dashboard shows 4.6 average rating (1–5 ordinal) from 2,100 responses. Total June stays: 9,800. Complaints to front desk rose 15% same month.

Tasks:

1. Audit completeness and hypothesize missingness mechanism (MCAR, MAR, MNAR).
2. Compute response rate and state coverage risk for a brand advertising claim ("guests love us").
3. Propose two analyses that do not rely solely on respondents.
4. Write a sensitivity plan: what decision changes if true satisfaction is 0.5 stars lower?

Solution

1. Audit: Completeness = 2,100/9,800 = 21.4% responded. Timeliness OK. Uniqueness check required on stay_id. MNAR plausible: unhappy guests skip email surveys while angry guests may complain in person (observable alternate channel). Not MCAR because complaint channel correlates with satisfaction.

2. Response rate 21.4%. Advertising "guests love us" based on respondents overstates population; coverage risk is high. At minimum, disclose "among email respondents."

3. Non-survey analyses: (a) Complaint rate per 1,000 stays from front desk logs (ratio, all stays). (b) Repeat booking rate within 90 days by property (behavioral outcome, panel-like). Both cover non-respondents.

4. Sensitivity: If true mean were 4.1 instead of 4.6, loyalty program investment might shrink; if franchise renewal depends on >4.3 threshold, two properties could flip from approve to hold. Run decision with 4.6, 4.1, and 3.8 scenarios; if action changes below 4.3, collect better data before national campaign.

Check: 2,100 responses + 7,700 non-responses = 9,800 stays ✓.

A SaaS firm compares churn among "activated" users who completed onboarding checklist vs others. Activation requires four sessions in week one. Only 35% of signups activate.

Tasks:

1. Name the selection bias type and explain who is missing from the activated group.
2. Why does comparing churn activated vs non-activated not prove onboarding causes retention?
3. Suggest one data collection fix and one analytic fix.

Solution

1. Self-selection and survivorship elements: users with low initial need or poor fit never activate; busy users may churn before week one ends. Missing population: early churners and low-intent signups.

2. Activation correlates with intent and fit; churn differences may reflect who activates, not the checklist itself (confounding). Without a randomized push toward activation, causation is unsupported.

**3. Collection fix: randomized onboarding prompts for a holdout (ethical and product policy permitting). Analytic fix: model time-to-activation and early usage separately; compare churn among matched pairs with similar first-day behavior using propensity scoring (preview for later units).

Explain why in prose: Activation is a useful descriptive split but a weak causal proof without design. Governance and ethics (Lesson 5) may constrain experiments; analytic honesty still requires stating limits.

A marketplace shows seller ratings based only on buyers who complete a post-purchase survey within 48 hours. Average rating is 4.7. Dispute rate is rising.

Tasks:

1. Name two bias mechanisms.
2. Propose one metric using all orders, not only respondents.
3. Draft a footnote sentence for the public seller page.

Solution

1. MNAR (dissatisfied buyers skip surveys); selection (only fast respondents counted).

2. Dispute rate per 100 orders or on-time delivery rate across all shipments (ratio, full population).

3. Footnote: "Star rating reflects surveys completed within 48 hours of delivery (typically 18% of orders). See dispute and on-time rates for full performance."

Check: if dispute data covers all orders, dispute rate denominator equals order count ✓.

Key takeaways

• Evaluate accuracy, completeness, consistency, timeliness, uniqueness, and validity before trusting analysis.
• Classify missingness; MNAR is common in surveys and self-report, and breaks naive averages.
• Selection and measurement bias often invalidate "customers love it" claims; check coverage and definitions.
• Document limitations beside conclusions; run sensitivity analyses on high-stakes calls.
• Fix pipelines and definitions before scaling decisions driven by bad exports.

1. Pick one KPI you trust and write a one-paragraph quality memo using all six dimensions.
2. What population is systematically absent from your main dashboard?
3. Continue to Lesson 4: Structuring Data for Analysis.