Study guide
7.3 Linear Regression, Correlation Analysis, and Design of Experiments (ANOVA)
FE Industrial and Systems Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, CBT Strategy & Handbook Navigation
1.1 FE Industrial Exam Structure, CBT Logistics, and Pacing 1.2 Using the NCEES FE Reference Handbook and Approved Calculators
2 Chapter 2: Mathematics for Industrial Engineers
2.1 Analytic Geometry, Coordinate Systems, and Trigonometry 2.2 Calculus: Differential, Integral, and Series Approximations 2.3 Linear Algebra: Matrix Operations, Systems of Equations, and Vectors
3 Chapter 3: Engineering Sciences Core
3.1 Thermodynamics Fundamentals and Fluid Flow Principles 3.2 Statics, Dynamics, and Mechanics of Materials Core 3.3 Electrical Circuits, DC/AC Analysis, and Power in Industrial Systems
4 Chapter 4: Ethics and Professional Practice
4.1 NCEES Model Rules of Professional Conduct and Technical Society Ethics 4.2 Engineering Licensure, Public Welfare, and Professional Liability 4.3 Engineering Contracts, Intellectual Property, and Conflicts of Interest
5 Chapter 5: Engineering Economics
5.1 Time Value of Money, Discount Factors, and Equivalence 5.2 Evaluating Alternatives: Present Worth, Equivalent Annual Cost, and IRR 5.3 Incremental Rate of Return, Replacement Analysis, and Break-Even 5.4 Depreciation Methods, MACRS, Income Taxes, and Inflation
6 Chapter 6: Probability Foundations & Distributions
6.1 Basic Probability Concepts, Conditional Probability, and Bayes' Theorem 6.2 Discrete Probability Distributions 6.3 Continuous Probability Distributions
7 Chapter 7: Statistical Inference, Regression & Design of Experiments
7.1 Sampling Distributions, Central Limit Theorem, and Confidence Intervals 7.2 Hypothesis Testing: Means, Proportions, and Variances 7.3 Linear Regression, Correlation Analysis, and Design of Experiments (ANOVA)
8 Chapter 8: Data, Logic, Modeling & Linear Programming
8.1 Optimization Modeling: Decision Variables, Objectives, and Constraints 8.2 Linear Programming: Graphical Solutions and Simplex Fundamentals 8.3 Duality, Sensitivity Analysis, and Integer Linear Programming 8.4 Data, Logic Development, and Analytics
9 Chapter 9: Stochastic Models & Simulation
9.1 Queuing Theory: Little's Law, M/M/1, and Multi-Server Systems 9.2 Discrete-Time Markov Chains: Transition Matrices and Steady-State Probabilities 9.3 Discrete-Event Simulation Principles and Monte Carlo Modeling
10 Chapter 10: Engineering Management, Project Planning & Performance Measurement
10.1 Organizational Structure, Leadership, and Worker Motivation Theories 10.2 Project Management: WBS, Critical Path Method (CPM), PERT, and Crashing 10.3 Earned Value Management (EVM), Decision Trees, and Risk Preference 10.4 Performance Measurement: Productivity, KPIs, Wage Systems, and the Balanced Scorecard
11 Chapter 11: Manufacturing & Production Systems
11.1 Manufacturing Processes, Material Processing, and Tolerancing 11.2 Assembly Line Balancing, Takt Time, Cycle Time, and Throughput 11.3 Sustainable Manufacturing, Waste Minimization, and Life-Cycle Assessment
12 Chapter 12: Production Planning, Inventory Control & Sequencing
12.1 Demand Forecasting Methods and Forecast Error Tracking 12.2 Aggregate Planning, Master Production Scheduling (MPS), and MRP 12.3 Deterministic Inventory Models: EOQ, EPQ, and Safety Stock 12.4 Job Sequencing, Dispatching Rules, and Flow Shop Scheduling
13 Chapter 13: Lean Systems & Theory of Constraints
13.1 Lean Foundations: Value Stream Mapping, Seven Wastes, and 5S 13.2 Pull Systems, Kanban, Just-In-Time (JIT), and SMED Setup Reduction 13.3 Theory of Constraints: Bottleneck Identification and Drum-Buffer-Rope
14 Chapter 14: Facilities Planning & Material Handling
14.1 Facility Location Analysis: Qualitative Factor Rating and Centroid Method 14.2 Facility Layout Design and Flow Analysis 14.3 Material Handling Equipment, Warehousing, and AS/RS
15 Chapter 15: Supply Chain Design & Logistics
15.1 Supply Chain Network Architecture and Transportation Modes 15.2 Distribution Strategies, Cross-Docking, and Vehicle Routing 15.3 The Bullwhip Effect, Risk Pooling, and Supply Chain Coordination
16 Chapter 16: Human Factors & Cognitive Ergonomics
16.1 Cognitive Engineering, Human-Machine Interface, Displays, and Controls 16.2 Anthropometric Principles, Percentile Design, and Biomechanical Posture 16.3 Environmental Ergonomics: Industrial Noise, Lighting, and Heat Stress
17 Chapter 17: Physical Ergonomics & Workplace Safety
17.1 Physical Ergonomics: The NIOSH Manual Lifting Equation 17.2 Cumulative Trauma Disorders and Postural Assessment 17.3 Workplace Safety: OSHA Standards, Hazard Analysis, and Control Hierarchy
18 Chapter 18: Work Design & Measurement
18.1 Methods Engineering, Process Charting, and Principles of Motion Economy 18.2 Direct Stopwatch Time Study: Observed Time, Performance Rating, and Allowances 18.3 Predetermined Time Systems, Work Sampling, and Learning Curves
19 Chapter 19: Quality Management, Six Sigma & Analytical Tools
19.1 Quality Philosophies, Total Quality Management, and Cost of Quality 19.2 Six Sigma DMAIC Methodology and Quality Function Deployment 19.3 The Seven Basic Quality Tools, Root Cause Analysis, and Taguchi Loss Function
20 Chapter 20: Statistical Process Control
20.1 Statistical Process Control Principles, Variation, and Run Rules 20.2 Variable Control Charts: X-bar & R, X-bar & S, and Individual-Moving Range 20.3 Attribute Control Charts: Defectives (p, np) and Defects (c, u)
21 Chapter 21: Process Capability & Acceptance Sampling
21.1 Process Capability Indices and Defect Levels 21.2 Acceptance Sampling Concepts and Operating Characteristic Curves 21.3 Sampling Risks, AQL, LTPD, and Standard Sampling Plans
22 Chapter 22: Systems Engineering & Life-Cycle Management
22.1 The Systems Engineering V-Model, Stakeholder Requirements, and Functional Architecture 22.2 Configuration Management, Interface Control, and Verification & Validation 22.3 Life-Cycle Costing, Maintainability, and System Availability
23 Chapter 23: Reliability Engineering & Risk Analysis
23.1 Reliability Mathematics, Hazard Rates, and the Bathtub Curve 23.2 System Reliability Configurations: Series, Parallel, and k-out-of-n Systems 23.3 Failure Modes and Effects Analysis and Fault Tree Analysis
7.3 Linear Regression, Correlation Analysis, and Design of Experiments (ANOVA)
Key Takeaways Explain This Topic AI
Ordinary Least Squares (OLS) regression minimizes the sum of squared residuals (SSE), yielding slope β̂₁ = S_xy / S_xx and intercept β̂₀ = ȳ - β̂₁x̄, guaranteeing the fitted line passes through the centroid (x̄, ȳ).
The Pearson correlation coefficient r (-1 ≤ r ≤ 1) quantifies linear association, sharing the sign of β̂₁; its square, the coefficient of determination R² = SSR / SST, measures the proportion of total response variance explained by the model.
Model adequacy requires validating that residuals are independent, normally distributed with zero mean, and have constant variance (homoscedasticity); funnel patterns in residual vs. fitted plots signify heteroscedasticity.
One-way ANOVA partitions total variation into between-treatment (SS_treatments) and within-treatment error (SS_error); the omnibus test statistic F₀ = MS_treatments / MS_error tests the equality of k group means.
In 2^k full factorial experimental designs, k factors at 2 levels yield 2^k runs; main effects measure the average response change across factor levels, while interaction effects identify synergistic or antagonistic factor coupling.
Last updated: September 2026
Industrial and systems engineers frequently model empirical relationships between process variables, optimize multivariable systems, and isolate root causes of excessive manufacturing variability. Simple and multiple linear regression quantify how controllable process inputs drive performance outputs. Analysis of Variance (ANOVA) and Design of Experiments (DOE) provide structured, mathematically rigorous frameworks for evaluating multi-factor systems while minimizing required experimental trials. The NCEES FE Reference Handbook details these techniques under the Linear Regression and Analysis of Variance subsections. Mastering these methods is critical for both the FE Industrial exam and real-world process optimization.
- 1. Simple Linear Regression Model and Ordinary Least Squares (OLS)
- Model Formulation
The simple linear regression model assumes that a continuous response variable y y y is linearly related to a single independent regressor variable x x x plus an unobservable random error term ε \varepsilon ε : y i = β 0 + β 1 x i + ε i , i = 1 , 2 , … , n y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad i = 1, 2, \dots, n y i = β 0 + β 1 x i + ε i , i = 1 , 2 , … , n
where:
- β 0 \beta_0 β 0 : Population intercept (value of y y y when x = 0 x = 0 x = 0 )
- β 1 \beta_1 β 1 : Population slope (change in mean response per unit increase in x x x )
ε i \varepsilon_i ε i : Independent and identically distributed random error, ε i ∼ i.i.d. N ( 0 , σ 2 ) \varepsilon_i \sim \text{i.i.d. } N(0, \sigma^2) ε i ∼ i.i.d. N ( 0 , σ 2 )
The Ordinary Least Squares (OLS) Criterion
OLS determines the parameter estimates β ^ 0 \hat{\beta}_0 β ^ 0 and β ^ 1 \hat{\beta}_1 β ^ 1 that minimize the Sum of Squared Errors (SSE) between observed responses y i y_i y i and fitted responses y ^ i = β ^ 0 + β ^ 1 x i \hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i y ^ i = β ^ 0 + β ^ 1 x i : SSE = ∑ i = 1 n e i 2 = ∑ i = 1 n ( y i − y ^ i ) 2 = ∑ i = 1 n [ y i − ( β ^ 0 + β ^ 1 x i ) ] 2 \text{SSE} = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \sum_{i=1}^n [y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i)]^2 SSE = i = 1 ∑ n e i 2 = i = 1 ∑ n ( y i − y ^ i ) 2 = i = 1 ∑ n [ y i − ( β ^ 0 + β ^ 1 x i ) ] 2
Closed-Form Parameter Formulas
Taking partial derivatives with respect to β ^ 0 \hat{\beta}_0 β ^ 0 and β ^ 1 \hat{\beta}_1 β ^ 1 and setting them to zero yields the normal equations. Solving produces the standard NCEES formulas: S x x = ∑ i = 1 n ( x i − x ˉ ) 2 = ∑ i = 1 n x i 2 − ( ∑ x i ) 2 n S_{xx} = \sum_{i=1}^n (x_i - \bar{x})^2 = \sum_{i=1}^n x_i^2 - \frac{(\sum x_i)^2}{n} S xx = i = 1 ∑ n ( x i − x ˉ ) 2 = i = 1 ∑ n x i 2 − n ( ∑ x i ) 2 S y y = ∑ i = 1 n ( y i − y ˉ ) 2 = ∑ i = 1 n y i 2 − ( ∑ y i ) 2 n = SST S_{yy} = \sum_{i=1}^n (y_i - \bar{y})^2 = \sum_{i=1}^n y_i^2 - \frac{(\sum y_i)^2}{n} = \text{SST} S yy = i = 1 ∑ n ( y i − y ˉ ) 2 = i = 1 ∑ n y i 2 − n ( ∑ y i ) 2 = SST S x y = ∑ i = 1 n ( x i − x ˉ ) ( y i − y ˉ ) = ∑ i = 1 n x i y i − ( ∑ x i ) ( ∑ y i ) n S_{xy} = \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \sum_{i=1}^n x_i y_i - \frac{(\sum x_i)(\sum y_i)}{n} S x y = i = 1 ∑ n ( x i − x ˉ ) ( y i − y ˉ ) = i = 1 ∑ n x i y i − n ( ∑ x i ) ( ∑ y i ) β ^ 1 = S x y S x x \hat{\beta}_1 = \frac{S_{xy}}{S_{xx}} β ^ 1 = S xx S x y β ^ 0 = y ˉ − β ^ 1 x ˉ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} β ^ 0 = y ˉ − β ^ 1 x ˉ
Fundamental Invariants :
The fitted regression line always passes through the bivariate centroid ( x ˉ , y ˉ ) (\bar{x}, \bar{y}) ( x ˉ , y ˉ ) .
The sum of the raw residuals is identically zero: ∑ i = 1 n e i = ∑ i = 1 n ( y i − y ^ i ) = 0 \sum_{i=1}^n e_i = \sum_{i=1}^n (y_i - \hat{y}_i) = 0 ∑ i = 1 n e i = ∑ i = 1 n ( y i − y ^ i ) = 0 .
- 2. Pearson Correlation and Coefficient of Determination (R²)
- Pearson Product-Moment Correlation Coefficient ( r r r )
The sample correlation coefficient r r r measures the strength and direction of a linear relationship between two variables: r = S x y S x x S y y , − 1 ≤ r ≤ + 1 r = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}, \quad -1 \le r \le +1 r = S xx S yy S x y , − 1 ≤ r ≤ + 1
r = + 1 r = +1 r = + 1 : Perfect positive linear association.
r = − 1 r = -1 r = − 1 : Perfect negative linear association.
r = 0 r = 0 r = 0 : No linear relationship (variables may still share a strong non-linear relationship).
Link to Regression Slope : β ^ 1 = r ( s y s x ) \hat{\beta}_1 = r \left(\frac{s_y}{s_x}\right) β ^ 1 = r ( s x s y ) . Consequently, the sign of β ^ 1 \hat{\beta}_1 β ^ 1 is always identical to the sign of r r r .
Partitioning Total Variation: SST = SSR + SSE
Total variability in the response variable ( y y y ) is partitioned into two mutually exclusive sums of squares: SST = SSR + SSE \text{SST} = \text{SSR} + \text{SSE} SST = SSR + SSE
Component Formula Degrees of Freedom Meaning
Total Sum of Squares (SST) ∑ ( y i − y ˉ ) 2 = S y y \sum (y_i - \bar{y})^2 = S_{yy} ∑ ( y i − y ˉ ) 2 = S yy n − 1 n - 1 n − 1 Total variation of y y y about y ˉ \bar{y} y ˉ
Regression Sum of Squares (SSR) ∑ ( y ^ i − y ˉ ) 2 = β ^ 1 S x y \sum (\hat{y}_i - \bar{y})^2 = \hat{\beta}_1 S_{xy} ∑ ( y ^ i − y ˉ ) 2 = β ^ 1 S x y 1 1 1 Variation explained by the linear relationship
Error Sum of Squares (SSE) ∑ ( y i − y ^ i ) 2 = SST − SSR \sum (y_i - \hat{y}_i)^2 = \text{SST} - \text{SSR} ∑ ( y i − y ^ i ) 2 = SST − SSR n − 2 n - 2 n − 2 Unexplained residual noise
Coefficient of Determination ( R 2 R^2 R 2 ) R 2 = SSR SST = 1 − SSE SST = r 2 , 0 ≤ R 2 ≤ 1 R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}} = r^2, \quad 0 \le R^2 \le 1 R 2 = SST SSR = 1 − SST SSE = r 2 , 0 ≤ R 2 ≤ 1
Interpretation : R 2 R^2 R 2 represents the proportion of total variation in the response variable y y y that is explained by the fitted linear model with regressor x x x .
Caution : High R 2 R^2 R 2 does not imply causation, nor does it guarantee model validity if residuals exhibit non-random structure.
3. Residual Analysis and Model Adequacy Checking
Statistical inferences (CIs and hypothesis tests on β 1 \beta_1 β 1 ) rely on the Gauss-Markov assumptions: errors ε i \varepsilon_i ε i must be independent, normally distributed, have zero mean, and possess constant variance σ 2 \sigma^2 σ 2 ( homoscedasticity ). Residuals e i = y i − y ^ i e_i = y_i - \hat{y}_i e i = y i − y ^ i serve as the empirical proxies for ε i \varepsilon_i ε i . Residuals vs Fitted (Ideal) Residuals vs Fitted (Heteroscedastic) Residuals vs Fitted (Non-linear)
- ┌────────────────┐ ┌────────────────┐ ┌────────────────┐
- │ • • • • │ │ • • │ │ • • • │
- e │• • • • │ e │ • • • │ e │ • • │
- 0 ───┼────────────────┼─── 0 ───┼────────────────┼─── 0 ───┼────────────────┼───
- │ • • • • │ │ • • • │ │• │
- │ • • • │ │ • • │ │ • • • • │
- └────────────────┘ └────────────────┘ └────────────────┘
- y_hat y_hat y_hat
Residuals vs. Fitted Values ( y ^ i \hat{y}_i y ^ i ) :
Random scatter band around zero : Assumptions satisfied.
Funnel / megaphone shape : Non-constant variance (heteroscedasticity). Requires a variance-stabilizing transform such as ln ( y ) \ln(y) ln ( y ) or y \sqrt{y} y .
Curved / parabolic pattern : Non-linear relationship; indicates a higher-order term (e.g., x 2 x^2 x 2 ) has been omitted.
Normal Probability Plot (Q-Q Plot) of Residuals : Plots sorted residuals against theoretical normal quantiles. Points should align tightly along a 45 ∘ 45^\circ 4 5 ∘ straight line; S-shaped deviations indicate heavy tails, while convex/concave curves indicate skewness.
Residuals vs. Time / Run Order : Detects machine drift, ambient temperature shifts, or operator fatigue. Patterns indicate autocorrelation, violating the independence assumption.
4. Multiple Linear Regression (MLR) Basics
When k k k predictor variables drive the response: y = β 0 + β 1 x 1 + β 2 x 2 + ⋯ + β k x k + ε y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_k x_k + \varepsilon y = β 0 + β 1 x 1 + β 2 x 2 + ⋯ + β k x k + ε
Adjusted R 2 R^2 R 2 ( R adj 2 R^2_{\text{adj}} R adj 2 )
Adding additional predictor variables to an MLR model always increases the raw R 2 R^2 R 2 (or leaves it unchanged), even if the added variables are pure random noise. To prevent over-fitting, the adjusted R 2 R^2 R 2 penalizes model complexity: R adj 2 = 1 − SSE / ( n − k − 1 ) SST / ( n − 1 ) = 1 − ( n − 1 n − k − 1 ) ( 1 − R 2 ) R^2_{\text{adj}} = 1 - \frac{\text{SSE} / (n - k - 1)}{\text{SST} / (n - 1)} = 1 - \left( \frac{n - 1}{n - k - 1} \right)(1 - R^2) R adj 2 = 1 − SST / ( n − 1 ) SSE / ( n − k − 1 ) = 1 − ( n − k − 1 n − 1 ) ( 1 − R 2 )
If an added variable does not reduce SSE \text{SSE} SSE by an amount sufficient to offset the loss of a degree of freedom, R adj 2 R^2_{\text{adj}} R adj 2 will decrease.
Multicollinearity
Occurs when two or more predictor variables are highly correlated with each other. Multicollinearity does not reduce the overall predictive power of the model, but it inflates the standard errors of individual coefficients β ^ j \hat{\beta}_j β ^ j , making them unstable and leading to high p p p -values for variables that are actually important.
5. One-Way Analysis of Variance (ANOVA)
One-way ANOVA tests the null hypothesis that k k k different treatment populations share the same mean: H 0 : μ 1 = μ 2 = ⋯ = μ k vs. H 1 : At least two treatment means differ H_0: \mu_1 = \mu_2 = \dots = \mu_k \quad \text{vs.} \quad H_1: \text{At least two treatment means differ} H 0 : μ 1 = μ 2 = ⋯ = μ k vs. H 1 : At least two treatment means differ
Partitioning the Sum of Squares
Let k k k be the number of treatments, n i n_i n i the sample size of treatment i i i , and N = ∑ i = 1 k n i N = \sum_{i=1}^k n_i N = ∑ i = 1 k n i the total observations (for a balanced design with n n n units per treatment, N = k n N = kn N = kn ): SS total = SS treatments + SS error \text{SS}_{\text{total}} = \text{SS}_{\text{treatments}} + \text{SS}_{\text{error}} SS total = SS treatments + SS error SS total = ∑ i = 1 k ∑ j = 1 n i ( y i j − y ˉ ⋅ ⋅ ) 2 \text{SS}_{\text{total}} = \sum_{i=1}^k \sum_{j=1}^{n_i} (y_{ij} - \bar{y}_{\cdot\cdot})^2 SS total = i = 1 ∑ k j = 1 ∑ n i ( y ij − y ˉ ⋅⋅ ) 2 SS treatments = ∑ i = 1 k n i ( y ˉ i ⋅ − y ˉ ⋅ ⋅ ) 2 \text{SS}_{\text{treatments}} = \sum_{i=1}^k n_i (\bar{y}_{i\cdot} - \bar{y}_{\cdot\cdot})^2 SS treatments = i = 1 ∑ k n i ( y ˉ i ⋅ − y ˉ ⋅⋅ ) 2 SS error = ∑ i = 1 k ∑ j = 1 n i ( y i j − y ˉ i ⋅ ) 2 = SS total − SS treatments \text{SS}_{\text{error}} = \sum_{i=1}^k \sum_{j=1}^{n_i} (y_{ij} - \bar{y}_{i\cdot})^2 = \text{SS}_{\text{total}} - \text{SS}_{\text{treatments}} SS error = i = 1 ∑ k j = 1 ∑ n i ( y ij − y ˉ i ⋅ ) 2 = SS total − SS treatments
The Standard One-Way ANOVA Table
Source of Variation Sum of Squares (SS) Degrees of Freedom (df) Mean Square (MS) F-Statistic ( F 0 F_0 F 0 )
Treatments (Between) SS treatments \text{SS}_{\text{treatments}} SS treatments k − 1 k - 1 k − 1 MS treatments = SS treatments k − 1 \text{MS}_{\text{treatments}} = \frac{\text{SS}_{\text{treatments}}}{k - 1} MS treatments = k − 1 SS treatments F 0 = MS treatments MS error F_0 = \frac{\text{MS}_{\text{treatments}}}{\text{MS}_{\text{error}}} F 0 = MS error MS treatments
Error (Within) SS error \text{SS}_{\text{error}} SS error N − k N - k N − k MS error = SS error N − k \text{MS}_{\text{error}} = \frac{\text{SS}_{\text{error}}}{N - k} MS error = N − k SS error —
- Total SS total \text{SS}_{\text{total}} SS total N − 1 N - 1 N − 1 — —
- The Decision Rule
Under H 0 H_0 H 0 , the test statistic follows an F F F -distribution with ν 1 = k − 1 \nu_1 = k - 1 ν 1 = k − 1 numerator degrees of freedom and ν 2 = N − k \nu_2 = N - k ν 2 = N − k denominator degrees of freedom: If F 0 > F α , k − 1 , N − k , Reject H 0 \text{If } F_0 > F_{\alpha, \, k-1, \, N-k}, \quad \text{Reject } H_0 If F 0 > F α , k − 1 , N − k , Reject H 0
6. Design of Experiments (DOE) Foundations
Design of Experiments is an active, planned testing approach where input factors are deliberately varied to observe and quantify corresponding changes in the output response.
Core Terminology
Factor : An independent controllable variable (e.g., cutting speed, furnace temperature, feed rate).
Level : A specific value assigned to a factor (e.g., Temperature at 200 ∘ C 200^\circ\text{C} 20 0 ∘ C and 250 ∘ C 250^\circ\text{C} 25 0 ∘ C ).
Treatment : A specific combination of factor levels applied to an experimental run.
Experimental Unit : The smallest physical entity to which a treatment is applied (e.g., a single silicon wafer, a stamped metal bracket).
The Three Fundamental Principles of DOE
Replication : Independent repetition of the basic experiment under identical treatment conditions. Replication provides an estimate of experimental error ( MS error \text{MS}_{\text{error}} MS error ) and increases the precision of effect estimates.
Randomization : Allocating both the experimental units and the sequential run order randomly. Randomization balances out the effects of unobserved, uncontrolled nuisance variables (e.g., tool wear, ambient humidity).
Blocking : Used to isolate and systematically eliminate variability caused by known, controllable nuisance factors that are not of primary interest (e.g., operator shift, raw material heat/lot). In a Randomized Complete Block Design (RCBD) , each block contains a complete set of treatments.
7. 2 k 2^k 2 k Full Factorial Designs
In a 2 k 2^k 2 k factorial design, there are k k k factors, each evaluated at exactly 2 levels : low (denoted by − - − or − 1 -1 − 1 ) and high (denoted by + + + or + 1 +1 + 1 ). The total number of required treatment runs is N = 2 k N = 2^k N = 2 k (or N = n ⋅ 2 k N = n \cdot 2^k N = n ⋅ 2 k with n n n replicates).
Main Effects and Contrasts
The main effect of factor A A A is the average change in response observed as factor A A A transitions from its low level to its high level across all levels of all other factors: Effect ( A ) = y ˉ A + − y ˉ A − = Contrast A n ⋅ 2 k − 1 \text{Effect}(A) = \bar{y}_{A^+} - \bar{y}_{A^-} = \frac{\text{Contrast}_A}{n \cdot 2^{k-1}} Effect ( A ) = y ˉ A + − y ˉ A − = n ⋅ 2 k − 1 Contrast A
where Contrast A = ∑ ( ± y i ) \text{Contrast}_A = \sum (\pm y_i) Contrast A = ∑ ( ± y i ) , using the signs from the design matrix column for factor A A A .
Interaction Effects
An interaction occurs when the effect of factor A A A depends upon the level of factor B B B . The interaction column A B AB A B is computed by taking the algebraic product of the signs in columns A A A and B B B ( + × + = + + \times + = + + × + = + , + × − = − + \times - = - + × − = − ). No Interaction (Parallel) Strong Interaction (Non-Parallel)
- Response (y) Response (y)
- │ │
- B_high ────┼───────────────────── B_high ─────┼───────────────
- │ │ ╲ ╱
- B_low ────┼───────────────────── B_low ─────┼───────────╳───
- │ │ ╱ ╲
- └──────────┬────────── └──────────┬───
- Factor A Factor A
Interaction Plot Rule : If the profile lines for different factor levels are parallel , no interaction exists. If the lines are non-parallel or cross , a strong interaction is present, and main effects cannot be interpreted in isolation.
8. Response Surface Methodology (RSM) Intuition
Response Surface Methodology (RSM) combines mathematical and statistical techniques to model and optimize systems where continuous input variables influence a continuous response (e.g., maximizing chemical yield or minimizing surface roughness).
Phase 1 (Screening & First-Order Model) : Fit a linear plane y = β 0 + ∑ β i x i + ε y = \beta_0 + \sum \beta_i x_i + \varepsilon y = β 0 + ∑ β i x i + ε using a 2 k 2^k 2 k factorial design. Determine the path of steepest ascent (gradient vector) to move the operating window toward the optimum.
Phase 2 (Optimization & Second-Order Model) : When curvature is detected near the peak, augment the design with center points and axial (star) points to form a Central Composite Design (CCD) or a Box-Behnken design . Fit a full second-order polynomial model: y = β 0 + ∑ i = 1 k β i x i + ∑ i = 1 k β i i x i 2 + ∑ ∑ i < j β i j x i x j + ε y = \beta_0 + \sum_{i=1}^k \beta_i x_i + \sum_{i=1}^k \beta_{ii} x_i^2 + \sum \sum_{i < j} \beta_{ij} x_i x_j + \varepsilon y = β 0 + i = 1 ∑ k β i x i + i = 1 ∑ k β ii x i 2 + ∑ i < j ∑ β ij x i x j + ε Solve for the stationary point (maximum, minimum, or saddle point) via canonical analysis.
- 9. Step-by-Step Worked Engineering Examples
- Worked Example 7.3.1: Cutting Tool Feed Rate vs. Surface Roughness (OLS)
Problem : A manufacturing engineer evaluates the effect of CNC lathe feed rate ( x x x , in mm/rev \text{mm/rev} mm/rev ) on machined surface roughness ( y y y , in μ m \mu\text{m} μ m ). Data collected from n = 5 n = 5 n = 5 test coupons is summarized below: ∑ x i = 1.00 , ∑ y i = 15.0 , ∑ x i 2 = 0.220 , ∑ y i 2 = 47.50 , ∑ x i y i = 3.20 \sum x_i = 1.00, \quad \sum y_i = 15.0, \quad \sum x_i^2 = 0.220, \quad \sum y_i^2 = 47.50, \quad \sum x_i y_i = 3.20 ∑ x i = 1.00 , ∑ y i = 15.0 , ∑ x i 2 = 0.220 , ∑ y i 2 = 47.50 , ∑ x i y i = 3.20
Determine the least-squares regression line y ^ = β ^ 0 + β ^ 1 x \hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x y ^ = β ^ 0 + β ^ 1 x , the correlation coefficient r r r , and the coefficient of determination R 2 R^2 R 2 .
Solution :
Compute Sample Means : x ˉ = 1.00 5 = 0.20 mm/rev , y ˉ = 15.0 5 = 3.00 μ m \bar{x} = \frac{1.00}{5} = 0.20\text{ mm/rev}, \quad \bar{y} = \frac{15.0}{5} = 3.00\;\mu\text{m} x ˉ = 5 1.00 = 0.20 mm/rev , y ˉ = 5 15.0 = 3.00 μ m
Compute Sums of Squares : S x x = ∑ x i 2 − ( ∑ x i ) 2 n = 0.220 − ( 1.00 ) 2 5 = 0.220 − 0.200 = 0.020 S_{xx} = \sum x_i^2 - \frac{(\sum x_i)^2}{n} = 0.220 - \frac{(1.00)^2}{5} = 0.220 - 0.200 = 0.020 S xx = ∑ x i 2 − n ( ∑ x i ) 2 = 0.220 − 5 ( 1.00 ) 2 = 0.220 − 0.200 = 0.020 S y y = ∑ y i 2 − ( ∑ y i ) 2 n = 47.50 − ( 15.0 ) 2 5 = 47.50 − 45.00 = 2.50 = SST S_{yy} = \sum y_i^2 - \frac{(\sum y_i)^2}{n} = 47.50 - \frac{(15.0)^2}{5} = 47.50 - 45.00 = 2.50 = \text{SST} S yy = ∑ y i 2 − n ( ∑ y i ) 2 = 47.50 − 5 ( 15.0 ) 2 = 47.50 − 45.00 = 2.50 = SST S x y = ∑ x i y i − ( ∑ x i ) ( ∑ y i ) n = 3.20 − ( 1.00 ) ( 15.0 ) 5 = 3.20 − 3.00 = 0.20 S_{xy} = \sum x_i y_i - \frac{(\sum x_i)(\sum y_i)}{n} = 3.20 - \frac{(1.00)(15.0)}{5} = 3.20 - 3.00 = 0.20 S x y = ∑ x i y i − n ( ∑ x i ) ( ∑ y i ) = 3.20 − 5 ( 1.00 ) ( 15.0 ) = 3.20 − 3.00 = 0.20
Compute Slope β ^ 1 \hat{\beta}_1 β ^ 1 and Intercept β ^ 0 \hat{\beta}_0 β ^ 0 : β ^ 1 = S x y S x x = 0.20 0.020 = 10.0 μ m / ( mm/rev ) \hat{\beta}_1 = \frac{S_{xy}}{S_{xx}} = \frac{0.20}{0.020} = 10.0\;\mu\text{m}/(\text{mm/rev}) β ^ 1 = S xx S x y = 0.020 0.20 = 10.0 μ m / ( mm/rev ) β ^ 0 = y ˉ − β ^ 1 x ˉ = 3.00 − ( 10.0 ) ( 0.20 ) = 3.00 − 2.00 = 1.00 μ m \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} = 3.00 - (10.0)(0.20) = 3.00 - 2.00 = 1.00\;\mu\text{m} β ^ 0 = y ˉ − β ^ 1 x ˉ = 3.00 − ( 10.0 ) ( 0.20 ) = 3.00 − 2.00 = 1.00 μ m Fitted Model: y ^ = 1.00 + 10.0 x \text{Fitted Model:} \quad \hat{y} = 1.00 + 10.0 x Fitted Model: y ^ = 1.00 + 10.0 x
Compute Correlation r r r and R 2 R^2 R 2 : r = S x y S x x S y y = 0.20 0.020 × 2.50 = 0.20 0.050 = 0.20 0.22361 ≈ + 0.8944 r = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}} = \frac{0.20}{\sqrt{0.020 \times 2.50}} = \frac{0.20}{\sqrt{0.050}} = \frac{0.20}{0.22361} \approx +0.8944 r = S xx S yy S x y = 0.020 × 2.50 0.20 = 0.050 0.20 = 0.22361 0.20 ≈ + 0.8944 R 2 = r 2 = ( 0.8944 ) 2 = 0.80 ( 80.0 % ) R^2 = r^2 = (0.8944)^2 = 0.80 \quad (80.0\%) R 2 = r 2 = ( 0.8944 ) 2 = 0.80 ( 80.0% ) Interpretation : 80.0 % 80.0\% 80.0% of the total variation in surface roughness is explained by feed rate. Sanity check : because ∣ r ∣ ≤ 1 |r| \le 1 ∣ r ∣ ≤ 1 always, S x y 2 S_{xy}^2 S x y 2 can never exceed S x x S y y S_{xx} S_{yy} S xx S yy . Here 0.20 2 = 0.040 < 0.020 × 2.50 = 0.050 0.20^2 = 0.040 < 0.020 \times 2.50 = 0.050 0.2 0 2 = 0.040 < 0.020 × 2.50 = 0.050 , so the summary statistics are internally consistent. If a computed ∣ r ∣ |r| ∣ r ∣ exceeds 1, the input sums contain an error.
Worked Example 7.3.2: One-Way ANOVA on Packaging Line Takt Times
Problem : An industrial engineering department audits k = 3 k = 3 k = 3 automated cartoning machines. A balanced sample of n = 5 n = 5 n = 5 cycle time trials is recorded for each machine ( N = 15 N = 15 N = 15 total runs). The data yields: SS treatments = 48.0 , SS total = 84.0 \text{SS}_{\text{treatments}} = 48.0, \quad \text{SS}_{\text{total}} = 84.0 SS treatments = 48.0 , SS total = 84.0
Construct the complete ANOVA table and determine whether machine cycle time means differ at α = 0.05 \alpha = 0.05 α = 0.05 ( F 0.05 , 2 , 12 = 3.89 F_{0.05, \, 2, \, 12} = 3.89 F 0.05 , 2 , 12 = 3.89 ).
Solution :
Calculate Error Sum of Squares : SS error = SS total − SS treatments = 84.0 − 48.0 = 36.0 \text{SS}_{\text{error}} = \text{SS}_{\text{total}} - \text{SS}_{\text{treatments}} = 84.0 - 48.0 = 36.0 SS error = SS total − SS treatments = 84.0 − 48.0 = 36.0
Determine Degrees of Freedom : df treatments = k − 1 = 3 − 1 = 2 \text{df}_{\text{treatments}} = k - 1 = 3 - 1 = 2 df treatments = k − 1 = 3 − 1 = 2 df error = N − k = 15 − 3 = 12 \text{df}_{\text{error}} = N - k = 15 - 3 = 12 df error = N − k = 15 − 3 = 12 df total = N − 1 = 15 − 1 = 14 \text{df}_{\text{total}} = N - 1 = 15 - 1 = 14 df total = N − 1 = 15 − 1 = 14
Compute Mean Squares : MS treatments = SS treatments df treatments = 48.0 2 = 24.0 \text{MS}_{\text{treatments}} = \frac{\text{SS}_{\text{treatments}}}{\text{df}_{\text{treatments}}} = \frac{48.0}{2} = 24.0 MS treatments = df treatments SS treatments = 2 48.0 = 24.0 MS error = SS error df error = 36.0 12 = 3.0 \text{MS}_{\text{error}} = \frac{\text{SS}_{\text{error}}}{\text{df}_{\text{error}}} = \frac{36.0}{12} = 3.0 MS error = df error SS error = 12 36.0 = 3.0
Compute Test Statistic F 0 F_0 F 0 : F 0 = MS treatments MS error = 24.0 3.0 = 8.00 F_0 = \frac{\text{MS}_{\text{treatments}}}{\text{MS}_{\text{error}}} = \frac{24.0}{3.0} = 8.00 F 0 = MS error MS treatments = 3.0 24.0 = 8.00
Decision and Conclusion :
Critical value: F 0.05 , 2 , 12 = 3.89 F_{0.05, \, 2, \, 12} = 3.89 F 0.05 , 2 , 12 = 3.89 .
Because F 0 = 8.00 > 3.89 F_0 = 8.00 > 3.89 F 0 = 8.00 > 3.89 , we reject H 0 H_0 H 0 .
Conclusion: There is statistically significant evidence at α = 0.05 \alpha = 0.05 α = 0.05 that the mean cycle times of the cartoning machines differ.
10. NCEES Reference Handbook Tips & Realistic Exam Traps
ANOVA Degrees of Freedom Balance : On incomplete ANOVA table problems, always verify the additive identity of degrees of freedom: df total = df treatments + df error \text{df}_{\text{total}} = \text{df}_{\text{treatments}} + \text{df}_{\text{error}} df total = df treatments + df error . For k k k treatments and N N N total observations, this is always ( N − 1 ) = ( k − 1 ) + ( N − k ) (N - 1) = (k - 1) + (N - k) ( N − 1 ) = ( k − 1 ) + ( N − k ) .
The Extrapolation Trap : A regression model is strictly valid only across the experimental range of the regressor variable [ x min , x max ] [x_{\min}, x_{\max}] [ x m i n , x m a x ] . Predicting response values beyond this interval is an invalid extrapolation that assumes linearity persists without empirical evidence.
R 2 R^2 R 2 vs. r r r Sign Trap : If given R 2 = 0.81 R^2 = 0.81 R 2 = 0.81 , r r r is either + 0.90 +0.90 + 0.90 or − 0.90 -0.90 − 0.90 . You must look at the slope β ^ 1 \hat{\beta}_1 β ^ 1 or the sign of S x y S_{xy} S x y to determine whether r r r is positive or negative.
Interaction Overrides Main Effects : If an interaction effect ( A B AB A B ) is statistically significant, you cannot interpret the main effects ( A A A or B B B ) in isolation. The optimal setting of factor A A A depends directly on the level of factor B B B .