7.5 Statistical Versus Practical Significance

ASQ Certified Six Sigma Black Belt CSSBB

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & CSSBB Exam Strategy

1.1 CSSBB Exam Format, Scoring, and Testing Windows 1.2 Eligibility, Project Affidavits, Application, and Recertification 1.3 Open-Book Strategy, Reference Materials, and a 16-Week Study Plan

2 Chapter 2: Organization-Wide Planning and Deployment

2.1 Fundamentals of Six Sigma and Lean Methodologies 2.2 Selecting the Right Methodology: DMAIC, DFSS, and Lean 2.3 Business Systems, Processes, and Stakeholder Relationships 2.4 Strategic Planning and Deployment of Six Sigma Initiatives 2.5 Six Sigma Roles and Responsibilities 2.6 Organizational Barriers and Common Causes of Six Sigma Failure 2.7 Change Management: Stakeholder Analysis, Readiness, and Communication

3 Chapter 3: Organizational Process Management and Measures

3.1 Impact of Six Sigma Projects on Customers, Suppliers, and Stakeholders 3.2 Benchmarking Types and Performance Goal Setting 3.3 Performance Measures: Balanced Scorecard, KPIs, KBIs, OKRs, and Line of Sight 3.4 Leading and Lagging Indicators and Performance Dashboards 3.5 Financial Measures: Revenue Growth, Margin, NPV, ROI, and Cost-Benefit Analysis 3.6 Cost of Poor Quality and Hard versus Soft Savings

4 Chapter 4: Team Management

4.1 Team Types and Constraints 4.2 Team Roles, Responsibilities, and Member Selection 4.3 Team Success Factors and Ground Rules 4.4 Motivational Techniques and Overcoming Demotivators 4.5 Stages of Team Development and Conflict Resolution 4.6 Team Communication Plans 4.7 Team Leadership Models: Direct, Coach, Support, and Delegate 4.8 Group Behaviors, Conflict Resolution, and Meeting Management 4.9 Facilitation Techniques and Team Decision-Making Methods 4.10 Team Training: Needs Assessment, Delivery, and Evaluation

5 Chapter 5: Define Phase

5.1 Customer Identification and Segmentation 5.2 Customer Data Collection: Methods, Validity, and Reliability 5.3 Customer Requirements: CTX, CTQ Trees, and Operational Definitions 5.4 The Kano Model and Quality Function Deployment 5.5 SIPOC and Process Mapping 5.6 Business Case Justification 5.7 The Project Charter and the Problem Statement 5.8 Project Scope and Boundaries 5.9 SMART Goals and Project Performance Measurements 5.10 Project Charter Reviews and Toll-Gate Governance 5.11 Project Management Tools: Gantt Charts, Toll-Gate Reviews, WBS, and RACI 5.12 Analytical Tools Used Throughout the DMAIC Cycle

6 Chapter 6: Measure Phase

6.1 Process Flow Metrics and the Hidden Factory 6.2 Process Analysis Tools and Value Stream Mapping 6.3 Types of Data and Measurement Scales 6.4 Sampling Concepts, Methods, and Sample Size for Estimation 6.5 Data Collection Plans and Common Pitfalls 6.6 Gage R&R for Variable Measurement Systems 6.7 Attribute Agreement Analysis and Audit MSA 6.8 Measurement Systems Across the Organization, and Metrology 6.9 Population Parameters, Sample Statistics, and Valid Statistical Conclusions 6.10 Descriptive Statistics: Central Tendency, Dispersion, and Shape 6.11 Graphical Methods for Exploratory Data Analysis 6.12 Probability Concepts, Common Distributions, and the Central Limit Theorem 6.13 Additional Distributions: Hypergeometric, Exponential, Lognormal, Weibull, and Bivariate 6.14 Process Capability and Performance Indices 6.15 Capability Studies, Attributes Capability, Yield Metrics, and Long-Term Capability

7 Chapter 7: Analyze Phase

7.1 The Correlation Coefficient, Its Confidence Interval, and Causation 7.2 Linear Regression and Residual Analysis 7.3 Multivariate Tools: Factor Analysis, Discriminant Analysis, and MANOVA 7.4 Hypothesis Testing Terminology, Errors, and Single-Sample Tests 7.5 Statistical Versus Practical Significance 7.6 Sample Size for Hypothesis Tests 7.7 Point Estimates, Confidence, Prediction, and Tolerance Intervals 7.8 Tests for Means, Variances, and Proportions 7.9 Analysis of Variance 7.10 Chi-Square Goodness-of-Fit and Tests of Independence 7.11 Types of Risk and Failure Mode and Effects Analysis 7.12 Gap Analysis and Root Cause Analysis 7.13 Waste Analysis: The Seven Classic Wastes and Resource Under-Utilization

8 Chapter 8: Improve Phase

8.1 Design of Experiments: Objectives, Terminology, and Effects 8.2 DOE Design Principles: Randomization, Blocking, Replication, and Resolution 8.3 Planning and Evaluating Designed Experiments 8.4 One-Factor Experiments: Completely Randomized, Randomized Block, and Latin Square Designs 8.5 Full Factorial Experiments and Design Selection 8.6 Two-Level Fractional Factorial Experiments and Confounding 8.7 Response Surface Methodology and Optimization 8.8 Waste Elimination: Pull Systems, Kanban, 5S, and Poka-Yoke 8.9 Cycle-Time Reduction: Continuous Flow, SMED, and Heijunka 8.10 Kaizen, Kaizen Blitz, Theory of Constraints, and OEE 8.11 Implementation Planning, Piloting, and Solution Selection

9 Chapter 9: Control Phase

9.1 SPC Objectives, Variable Selection, and Rational Subgrouping 9.2 Control Charts for Variables Data 9.3 Control Charts for Attributes Data 9.4 Short-Run SPC and Moving Average Charts 9.5 Control Chart Analysis: Common Cause, Special Cause, and Run Rules 9.6 Total Productive Maintenance and Visual Controls 9.7 Measurement System Reanalysis 9.8 Control Plan Development and Process Owner Handoff 9.9 Lessons Learned, Documentation, and Training for Sustainment 9.10 Ongoing Evaluation of the Improved Process

10 Chapter 10: Design for Six Sigma (DFSS)

10.1 DFSS Frameworks: DMADV, DMADOV, and Design Transfer 10.2 Design for X (DFX) 10.3 Robust Design, Parameter Design, and Verification 10.4 Tolerance Design and Statistical Tolerancing

7.5 Statistical Versus Practical Significance

Key Takeaways Explain This Topic AI

Statistical significance means the observed effect is unlikely under the null hypothesis; practical significance means the effect is large enough to justify action.

A p-value is a function of both effect size and sample size, so significance can always be manufactured by collecting more data.

Effect size measures such as Cohen's d and the difference in means with a confidence interval express magnitude independently of sample size.

A non-significant result from an underpowered study is not evidence that no effect exists.

The four combinations of statistical and practical significance require four different responses, and only one of them justifies implementation.

Last updated: August 2026

Two different questions

Statistical significance Practical significance

Question Is the observed effect likely to be real, or could it be sampling variation? Is the effect large enough to matter?

Answered by p-value against α \alpha α Effect size compared with a threshold set by the business
Depends on Effect size and sample size and variability Effect size and the economics of the process
Who decides The statistical procedure The process owner, the customer, or finance
The critical asymmetry: statistical significance is computed, practical significance is judged , and

the judgment must be made by someone who knows what the process is worth.

Why sample size drives the p-value

For a two-sample comparison of means the test statistic is roughly t = x ˉ 1 − x ˉ 2 s p 1 n 1 + 1 n 2 t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}} t = s p n 1 1 + n 2 1 x ˉ 1 − x ˉ 2

The numerator is the effect. The denominator shrinks as n n n grows. Hold the effect constant and
increase n n n , and t t t grows without bound -- so any non-zero difference eventually becomes

statistically significant.

Worked illustration. Two filling lines differ in mean fill weight by 0.05 g, with s = 1.2 s = 1.2 s = 1.2 g on a

product with a tolerance of ± 8 \pm 8 ± 8 g.

n n n per group Approximate t t t p-value Statistically significant at 0.05?

30 0.16 0.87 No
500 0.66 0.51 No
5,000 2.08 0.037 Yes
50,000 6.59 < 0.001 Yes
At 50,000 units per group the difference is overwhelmingly significant and completely irrelevant: 0.05 g

against an 8 g tolerance is 0.6% of the specification width. Nobody should spend a dollar closing it.

Measuring effect size

Report magnitude, not just the p-value.

Difference in means with a confidence interval. The most direct and the most useful. "Mean cycle
time fell by 1.8 days, 95% CI 1.1 to 2.5 days" tells the reader the size, the direction, and the

precision in the units of the problem.

Cohen's d , the standardized difference: d = x ˉ 1 − x ˉ 2 s p o o l e d d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}} d = s p oo l e d x ˉ 1 − x ˉ 2

d d d Conventional label
0.2 Small
0.5 Medium
0.8 Large
Proportion of variance explained. r 2 r^2 r 2 in regression, or eta-squared in ANOVA: the share of

output variation attributable to the factor.

Practical thresholds derived from the process. The most defensible option where they exist:

A shift smaller than the measurement system's uncertainty cannot be acted on reliably.

A shift smaller than a set fraction of the tolerance is not worth chasing.

A saving below the organization's project threshold does not justify the implementation cost.

Set the practical threshold before the analysis, in the charter. Setting it afterwards invites

choosing the threshold that makes the result look good.

The four combinations

Practically significant Not practically significant

Statistically significant Act. Real and worth doing. Do not act. Real but trivial; usually a symptom of a very large sample.

Not statistically significant Investigate further. Potentially valuable but not yet demonstrated -- check power and collect more data. Stop. Neither real nor important.

The bottom-left cell is the one teams handle worst. A promising effect that fails to reach significance

in an underpowered study is frequently reported as "no difference", and the opportunity is discarded.

The correct response is to compute the power the study actually had, and if it was low, treat the result

as inconclusive rather than negative.

What a p-value is not

Misinterpretation Correction

"p = 0.03 means there is a 3% chance the null is true" The p-value is the probability of data this extreme given the null is true, not the probability the null is true

"p = 0.001 means the effect is large" It means the evidence against the null is strong; magnitude is a separate question

"p = 0.06 proves there is no effect" Failure to reject is not acceptance; the effect may be real but undetected

"p < 0.05 means the result will replicate" Replication probability depends on power, not on the p-value
Reporting practice for a Six Sigma project

A defensible Analyze-phase conclusion states all four of:

The effect size in process units -- "mean cycle time reduced by 1.8 days".

The confidence interval -- "95% CI 1.1 to 2.5 days".

The p-value and the test used -- "two-sample t-test, p = 0.002".

The practical threshold and the verdict -- "the charter set a 1.0 day threshold; the entire

confidence interval exceeds it, so the effect is both real and material".

Point 4 is what distinguishes a Black Belt's analysis from a statistics exercise. The confidence
interval lying entirely above the practical threshold is the strongest possible statement, because it

resolves both questions at once.