Study guide

6.3 Measurement System Analysis (MSA) Foundations & Calibration

CSSC Certified Six Sigma Green Belt Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Six Sigma Overview & Exam Fundamentals

1.1 CSSC Certification Overview, Exam Structure & Body of Knowledge 1.2 History & Evolution of Six Sigma 1.3 Six Sigma Organizational Roles & Belts Hierarchy 1.4 The DMAIC Methodology Architecture vs. DMADV / DFSS 1.5 Other Process Improvement & Quality Methods, and When to Choose Six Sigma

2 Chapter 2: Quality Fundamentals & Defect Metrics

2.1 Critical to Quality (CTQ) Characteristics & VOC Translation 2.2 Defect Metrics: DPU, DPO, DPMO, & the 1.5-Sigma Shift 2.3 First Time Yield (FTY) & Rolled Throughput Yield (RTY) 2.4 Cost of Quality (COQ) & Cost of Poor Quality (COPQ)

3 Chapter 3: Lean Principles & Waste Elimination

3.1 Foundations of Lean & The Toyota Production System (TPS) 3.2 The Eight Wastes: CSSC's Seven Muda, TIMWOODS, and Muri/Mura 3.3 The 5S Visual Workplace Methodology 3.4 Value Stream Mapping (VSM), Cycle Time & Takt Time

4 Chapter 4: Project Selection & Team Management

4.1 Six Sigma Project Selection & Business Case Development 4.2 Project Charter Architecture 4.3 Stakeholder Management & Communication Planning 4.4 Team Dynamics & Tuckman's Team Development Stages 4.5 The 15-Point Project Viability Model & Process-Level Selection

5 Chapter 5: Define Phase & Process Mapping

5.1 High-Level Process Mapping with SIPOC 5.2 Flowcharts, Swimlane Diagrams & Process Value Analysis 5.3 Quality Function Deployment (QFD) & The House of Quality 5.4 Critical to Quality (CTQ) Flowdown & Operational Definitions 5.5 The CSSC Define Toolset: Ground Rules, In/Out of the Box, Is–Is Not & Tollgate Checklists

6 Chapter 6: Measure Phase & Measurement Systems

6.1 Data Types: Continuous (Variable) vs. Discrete (Attribute) Data 6.2 Data Collection Planning & Sampling Strategies 6.3 Measurement System Analysis (MSA) Foundations & Calibration 6.4 Variable & Attribute Gage R&R Studies

7 Chapter 7: Analyze Phase & Root Cause Analysis

7.1 Qualitative Root Cause Tools: 5 Whys & Cause-and-Effect (Fishbone) Diagrams 7.2 Failure Mode and Effects Analysis (FMEA) & RPN Calculation 7.3 Pareto Analysis & The 80/20 Rule 7.4 Graphical Data Exploration: Histograms, Box Plots, & Multi-Vari Charts 7.5 The Root Cause Verification Matrix

8 Chapter 8: Descriptive Statistics & Distributions

8.1 Measures of Central Tendency & Dispersion 8.2 The Normal Distribution & The Empirical Rule (68-95-99.7) 8.3 Standard Normal Distribution, Z-Scores & Probability Calculations 8.4 Common Non-Normal Distributions: Binomial, Poisson & Chi-Square

9 Chapter 9: Inferential Statistics & Hypothesis Testing

9.1 Fundamentals of Hypothesis Testing: Null vs. Alternative, Alpha & Beta Risks 9.2 One-Sample and Two-Sample t-Tests for Means 9.3 Paired t-Tests & Non-Parametric Alternatives 9.4 Analysis of Variance (One-Way & Two-Way ANOVA) & Chi-Square Contingency Tests 9.5 Sample Size Determination for Hypothesis Tests

10 Chapter 10: Correlation, Regression, & Experimentation

10.1 Linear Correlation & Pearson's Correlation Coefficient (r) 10.2 Simple Linear Regression & Coefficient of Determination (R^2) 10.3 Introduction to Design of Experiments (DOE) & Factorial Designs 10.4 Analyzing Experimental Results: Main Effects & Interaction Plots

11 Chapter 11: Improve Phase & Solution Implementation

11.1 Brainstorming, Creative Thinking & Solution Selection Matrices 11.2 Piloting Solutions & Conducting Risk Assessments 11.3 Implementation Planning & Change Management (ADKAR) 11.4 Mistake-Proofing (Poka-Yoke) & Standard Work

12 Chapter 12: Control Phase, SPC, & Process Capability

12.1 Statistical Process Control (SPC) & Shewhart Control Rules 12.2 Variable Control Charts: I-MR, X-bar & R, X-bar & S 12.3 Attribute Control Charts: p, np, c, and u Charts 12.4 Process Capability & Performance Indices: Cp, Cpk, Pp, Ppk 12.5 Communicating Statistical Results to Business Audiences

6.3 Measurement System Analysis (MSA) Foundations & Calibration

Key Takeaways Explain This Topic AI

Measurement System Analysis (MSA) validates data integrity before analysis; total observed variation is the sum of true process variation and measurement system variation (σ²_TV = σ²_PV + σ²_MS).

Variances add linearly (σ²_TV = σ²_PV + σ²_MS), but standard deviations do NOT; attempting to subtract standard deviations directly (σ_PV = σ_TV - σ_MS) is a severe mathematical error.

Excessive measurement system error inflates Producer's Risk (α / Type I error: rejecting conforming product) and Consumer's Risk (β / Type II error: releasing non-conforming product).

Total measurement error is partitioned into Precision (spread: Repeatability and Reproducibility) and Accuracy (location: Bias, Linearity, and Stability).

The Metrological 10-to-1 Rule (Rule of Ten) mandates that a gage's discrimination/resolution must be at least 10 times finer than the specification tolerance band or process variation.

Last updated: September 2026

6.3 Measurement System Analysis (MSA) Foundations & Calibration

Quick Summary: Before analyzing process capability or diagnosing root causes in DMAIC, continuous improvement practitioners must answer one fundamental question: "Can we trust our data?" Measurement System Analysis (MSA) is the mathematical and experimental methodology used to isolate, quantify, and evaluate the variation contributed by the measurement process itself. Total observed variance equals true process variance plus measurement system variance ( σ TV 2 = σ PV 2 + σ MS 2 \sigma^2_{\text{TV}} = \sigma^2_{\text{PV}} + \sigma^2_{\text{MS}} σ TV 2 ​ = σ PV 2 ​ + σ MS 2 ​ ). Measurement error divides into Accuracy (Bias, Linearity, Stability) and Precision (Repeatability and Reproducibility). Validating metrological traceability to national standards (NIST) and ensuring gages satisfy the 10-to-1 resolution rule are mandatory prerequisites for credible Six Sigma data.

The Critical Premise: "Can We Trust Our Data?"

Every dataset collected in an operational environment contains a combination of two distinct phenomena:

The true variation of the process output (actual part-to-part or transaction-to-transaction variation).

The measurement system error introduced by gages, fixtures, procedures, software algorithms, environmental fluctuations, and human appraisers.

Mathematically, every recorded observation ( Y observed Y_{\text{observed}} Y observed ​ ) is contaminated by measurement error ( ϵ measurement \epsilon_{\text{measurement}} ϵ measurement ​ ): Y observed = Y true + ϵ measurement Y_{\text{observed}} = Y_{\text{true}} + \epsilon_{\text{measurement}} Y observed ​ = Y true ​ + ϵ measurement ​

If the measurement system exhibits excessive variation or offset, the data cannot be trusted. Attempting to calculate process capability ( C p / C p k C_p / C_{pk} C p ​ / C p k ​ ), construct control charts, or execute hypothesis tests on unverified data produces dangerous illusions—leading teams to optimize non-existent problems, overlook critical defects, or misjudge root causes.

The Operational Consequences: Producer's vs. Consumer's Risk

When a measurement system introduces noise, observed values near specification limits are misclassified, generating two major operational risks: The Impact of Measurement System Error

  • Lower Spec Upper Spec
  • Limit Limit
  • (LSL) (USL)
  • │ │
  • Conforming Product │ Conforming Product │ Conforming Product
  • ────────────────────────┼────────────────────────┼────────────────────────
  • │ │
  • Producer's Risk │ │ Producer's Risk
  • (Type I / α) │ │ (Type I / α)
  • Good part called │ │ Good part called
  • bad and scrapped│ │ bad and scrapped
  • ◀──────┼──────▶ ◀──────┼──────▶
  • │ Consumer's Risk │ Consumer's Risk
  • │ (Type II / β) │ (Type II / β)
  • │ Bad part called │ Bad part called
  • │ good and shipped │ good and shipped

Producer's Risk (Type I Error / α \alpha α / False Alarm): A conforming unit that is physically within specification limits is measured as out-of-spec due to measurement error pushing the reading over the limit. The producer suffers financial losses from unnecessary scrap, rework, reinspection, and production halts.

Consumer's Risk (Type II Error / β \beta β / Miss): A non-conforming unit that is physically outside specification limits is measured as acceptable because measurement system noise pulls the reading inside the boundary. The defective unit escapes to the customer, leading to field failures, warranty claims, customer dissatisfaction, and potential safety liabilities.

The Total Observed Variation Model

The fundamental mathematical relationship governing all continuous measurement systems is the additivity of variance : σ Total 2 = σ Process 2 + σ Measurement System 2 \sigma^2_{\text{Total}} = \sigma^2_{\text{Process}} + \sigma^2_{\text{Measurement System}} σ Total 2 ​ = σ Process 2 ​ + σ Measurement System 2 ​ σ TV 2 = σ PV 2 + σ MS 2 \sigma^2_{\text{TV}} = \sigma^2_{\text{PV}} + \sigma^2_{\text{MS}} σ TV 2 ​ = σ PV 2 ​ + σ MS 2 ​

Where:

  • σ TV 2 \sigma^2_{\text{TV}} σ TV 2 ​ = Total Observed Variance (Total Variation)
  • σ PV 2 \sigma^2_{\text{PV}} σ PV 2 ​ = Actual Part-to-Part Process Variance

σ MS 2 \sigma^2_{\text{MS}} σ MS 2 ​ = Measurement System Variance (also designated as σ Gage 2 \sigma^2_{\text{Gage}} σ Gage 2 ​ or σ GRR 2 \sigma^2_{\text{GRR}} σ GRR 2 ​ ) Total Observed Process Variation Model

  • Total Observed Variation
  • (σ²_Total / σ²_TV)
  • │
  • ┌─────────────────────┴─────────────────────┐
  • ▼ ▼
  • Actual Process Variation Measurement System Variation
  • (σ²_Part / σ²_PV) (σ²_MS / σ²_GRR)
  • │ │
  • Part-to-Part Drift ┌──────────────┴──────────────┐
  • Raw Material Lots ▼ ▼
  • Tool Wear Repeatability Reproducibility
  • Environmental Shift (Equipment / EV) (Appraiser / AV)
  • The Additivity Rule of Variance

Important

The Variance Additivity Axiom: Variances ( σ 2 \sigma^2 σ 2 ) add linearly, but standard deviations ( σ \sigma σ ) do NOT. To compute total standard deviation, you must sum the component variances and take the square root:

σ TV = σ PV 2 + σ MS 2 \sigma_{\text{TV}} = \sqrt{\sigma^2_{\text{PV}} + \sigma^2_{\text{MS}}} σ TV ​ = σ PV 2 ​ + σ MS 2 ​ ​

Conversely, to isolate the true process standard deviation from total observed variation, you must subtract variances, not standard deviations:

σ PV = σ TV 2 − σ MS 2 \sigma_{\text{PV}} = \sqrt{\sigma^2_{\text{TV}} - \sigma^2_{\text{MS}}} σ PV ​ = σ TV 2 ​ − σ MS 2 ​ ​

Attempting to calculate true process variation by direct subtraction of standard deviations ( σ PV = σ TV − σ MS \sigma_{\text{PV}} = \sigma_{\text{TV}} - \sigma_{\text{MS}} σ PV ​ = σ TV ​ − σ MS ​ ) is a catastrophic mathematical error that severely distorts process capability.

Dissecting Measurement Error: Precision vs. Accuracy

Measurement system error is divided into two distinct categories: Accuracy (location/centering) and Precision (spread/dispersion). Precision vs. Accuracy

  • High Precision / Low Accuracy High Accuracy / High Precision
  • (Clustered off-target) (Clustered on-target)
  • ┌────────────────┐ ┌────────────────┐
  • │ • │ │ │
  • │ ••• │ │ •• │
  • │ • ◎ │ │ •••• │
  • │ │ │ •• │
  • └────────────────┘ └────────────────┘
  • Small Repeatability / Zero Bias, Zero Drift
  • High Systematic Bias Small Spread
  • Low Precision / High Accuracy Low Precision / Low Accuracy
  • (Centered but scattered) (Scattered and off-target)
  • ┌────────────────┐ ┌────────────────┐
  • │ • │ │ • │
  • │ • │ │ • │
  • │ • ◎ • │ │ • │
  • │ • • │ │ • ◎ │
  • └────────────────┘ └────────────────┘
  • Large Repeatability / Large Repeatability /
  • Zero Systematic Bias Large Systematic Bias

Accuracy (Location / Centering): The difference between the average of observed measurements and the true reference master value. Accuracy encompasses three components: Bias , Linearity , and Stability .

Precision (Spread / Dispersion): The closeness of agreement among repeated measurements of the same item under specified conditions. Precision encompasses two components: Repeatability (equipment variation) and Reproducibility (appraiser variation). Precision is evaluated using Gage R&R studies (examined in Section 6.4).

  • The Three Components of Measurement Accuracy
  • 1. Bias

Bias is the systematic distance between the observed average of repeated measurements and the true reference value of a certified master standard: Bias = X ˉ observed − X reference \text{Bias} = \bar{X}_{\text{observed}} - X_{\text{reference}} Bias = X ˉ observed ​ − X reference ​

Bias is frequently expressed as a percentage of process tolerance or total process variation: % Bias = ( ∣ Bias ∣ Tolerance ) × 100 % = ( ∣ X ˉ observed − X reference ∣ USL − LSL ) × 100 % \%\text{Bias} = \left( \frac{|\text{Bias}|}{\text{Tolerance}} \right) \times 100\% = \left( \frac{|\bar{X}_{\text{observed}} - X_{\text{reference}}|}{\text{USL} - \text{LSL}} \right) \times 100\% % Bias = ( Tolerance ∣ Bias ∣ ​ ) × 100% = ( USL − LSL ∣ X ˉ observed ​ − X reference ​ ∣ ​ ) × 100%

Statistical Significance of Bias

To verify whether an observed bias is statistically significant or merely the result of random sampling noise, the Green Belt conducts a one-sample t t t -test against the certified reference value ( H 0 : μ = X reference H_0: \mu = X_{\text{reference}} H 0 ​ : μ = X reference ​ vs. H 1 : μ ≠ X reference H_1: \mu \ne X_{\text{reference}} H 1 ​ : μ  = X reference ​ ): t = X ˉ observed − X reference s n = Bias s n t = \frac{\bar{X}_{\text{observed}} - X_{\text{reference}}}{\frac{s}{\sqrt{n}}} = \frac{\text{Bias}}{\frac{s}{\sqrt{n}}} t = n ​ s ​ X ˉ observed ​ − X reference ​ ​ = n ​ s ​ Bias ​

Where s s s is the sample standard deviation of n n n repeated trials on the master standard. If ∣ t ∣ > t crit |t| > t_{\text{crit}} ∣ t ∣ > t crit ​ , the bias is statistically significant and requires calibration adjustment.

Common Root Causes of Bias: Worn contact tips or measuring anvils, incorrect zero-setting, improper tare weight, thermal expansion from operator body heat, incorrect reference standards, or consistent operator parallax error.

2. Linearity

Linearity evaluates whether a gage maintains constant accuracy across its entire intended operating range. It represents the change in bias across the scale envelope. Linearity Evaluation Plot

  • Bias
  • ▲ +
  • │ +
  • │ +
  • + │ + (Positive bias on large parts)
  • │ +
  • 0 ─┼───────────────────────────────+──────────────────────────▶ Reference Size
  • │ +
  • │ +
  • │ +
  • │ + (Negative bias on small parts)
  • └─────────────────────────────────────────────────────────
  • Assessing Linearity

To assess linearity, a metrology technician selects 5 certified reference standards spanning the operating range (from smallest to largest parts). The technician measures each standard 10 to 12 times in randomized order, computes the average bias for each standard, and fits a simple linear regression equation: Bias = a × ( X reference ) + b \text{Bias} = a \times (X_{\text{reference}}) + b Bias = a × ( X reference ​ ) + b

Where a a a represents the slope of the linearity line and b b b represents the intercept. Linearity is evaluated as: Linearity = ∣ a ∣ × Process Variation = ∣ a ∣ × ( 6 σ process ) \text{Linearity} = |a| \times \text{Process Variation} = |a| \times (6\sigma_{\text{process}}) Linearity = ∣ a ∣ × Process Variation = ∣ a ∣ × ( 6 σ process ​ ) % Linearity = ( Linearity Process Variation ) × 100 % = ∣ a ∣ × 100 % \%\text{Linearity} = \left( \frac{\text{Linearity}}{\text{Process Variation}} \right) \times 100\% = |a| \times 100\% % Linearity = ( Process Variation Linearity ​ ) × 100% = ∣ a ∣ × 100%

If the slope ( a a a ) is statistically significantly different from zero ( p < 0.05 p < 0.05 p < 0.05 ), the gage exhibits significant linearity error. For example, a micrometer may be perfectly accurate when measuring a 10 mm standard (zero bias), but exhibits a + 0.04 mm +0.04\text{ mm} + 0.04 mm bias when measuring a 100 mm standard.

Common Root Causes of Linearity Error: Uneven mechanical lead screw wear, non-linear optical sensor response, mechanical deflection of large measuring frames, or improper electronic calibration curves.

3. Stability (Drift)

Stability (often referred to as drift ) is the total variation in measurements obtained when measuring the same master standard repeatedly over extended operational time.

Monitoring Stability with Control Charts

Stability is evaluated by establishing a standardized calibration monitoring schedule. At regular intervals (e.g., at the beginning of each operating shift), an inspector measures a certified master standard 3 to 5 times and plots the results on an X ˉ \bar{X} X ˉ - R R R chart or I I I - M R MR MR (Individuals and Moving Range) control chart .

If the control chart displays statistical stability (all points fall inside control limits with no runs, trends, or shifts), the measurement system is stable .

If points exceed control limits or exhibit non-random trends (such as 6 consecutive points steadily rising), the measurement system exhibits drift or instability.

Common Root Causes of Instability: Ambient temperature fluctuations expanding machine frames, inadequate warm-up times for electronic sensors, accumulation of dust, dirt, or cutting fluid on optical lenses, electronic sensor aging, or mechanical wear in guideways.

Summary of Accuracy Error Components

Accuracy Component Operational Question Answered Mathematical Metric Standard Evaluation Tool

Bias "Is the gage measuring high or low on average compared to the true standard?" Bias = X ˉ − X ref \text{Bias} = \bar{X} - X_{\text{ref}} Bias = X ˉ − X ref ​ One-sample t t t -test against certified master standard

Linearity "Does the gage measure with equal accuracy across small, medium, and large parts?" Slope ( a a a ) of Bias = a ( X ref ) + b \text{Bias} = a(X_{\text{ref}}) + b Bias = a ( X ref ​ ) + b Regression analysis across 5 master standards spanning range

Stability "Does the gage's calibration stay consistent over shifts, days, and months?" Out-of-control signals on SPC chart I I I - M R MR MR or X ˉ \bar{X} X ˉ - R R R control charts plotted over time

Metrology, Calibration Systems & Traceability

Metrology is the scientific study of measurement. To ensure that measurement data captured in one facility is comparable to measurements taken anywhere in the global supply chain, organizations must maintain formalized calibration systems. Metrological Traceability Pyramid

  • ┌─────────────────┐
  • │ BIPM / NIST │ National / International
  • │Primary Standards│ Unbroken Chain of Traceability
  • └────────┬────────┘
  • ▼
  • ┌─────────────────┐
  • │ Accredited Lab │ Secondary / Transfer
  • │ Master Standards│ Calibration Standards
  • └────────┬────────┘
  • ▼
  • ┌─────────────────┐
  • │ Shop Floor Tool │ Working Gages & Production
  • │ Working Gages │ Inspection Instruments
  • └─────────────────┘
  • Metrological Traceability

Traceability is the property of a measurement result whereby the result can be related to a stated reference—typically a national or international physical standard—through an unbroken chain of documented calibrations , each contributing to measurement uncertainty.

In the United States, standards are governed by the National Institute of Standards and Technology (NIST) .

Internationally, metrological harmony is maintained by the International Bureau of Weights and Measures (BIPM) under the International System of Units (SI).

Calibration Standards Hierarchy

Primary Standards: The highest metrological quality standards maintained by national institutes (NIST). These embody the fundamental SI physical constants (e.g., defining the meter via the speed of light in a vacuum).

Secondary / Transfer Standards: High-precision laboratory standards maintained by accredited calibration laboratories (ISO/IEC 17025 accredited). These are calibrated directly against primary standards.

Working Standards: Shop-floor reference items (e.g., precision gage block sets, master setting rings) used within an industrial plant to calibrate production gages.

Calibration Management Requirements

Calibration Intervals: Gages must have established calibration schedules based on manufacturer guidelines, usage frequency, and historical stability (e.g., monthly, quarterly, or annually).

Calibration Labeling: Every measuring instrument must display a tamper-evident calibration label detailing: Gage ID, Date Calibrated, Expiration Date, Technician ID, and Calibration Standard utilized.

Out-of-Calibration Containment Protocol: If an instrument is found to be out of calibration during scheduled recertification, quality engineering must issue a formal containment alert. All production lots inspected with that instrument since the last valid calibration date must be identified, contained, and re-evaluated for potential customer defect escape.

Gage Resolution & The 10-to-1 Rule (Rule of Ten)

Before evaluating precision or accuracy, a Green Belt must verify that the gage possesses adequate resolution (also termed discrimination or readability ).

Resolution is the smallest readable increment that a measuring instrument can detect and reliably display.

The 10-to-1 Rule (The Rule of Ten / Gage Maker's Tolerance)

Important

The 10-to-1 Resolution Standard: The measuring instrument must have a resolution that is at least 10 times finer than the specification tolerance band (or 10 times finer than the expected 6 σ 6\sigma 6 σ process variation).

Instrument Resolution ≤ 1 10 × Tolerance Band = USL − LSL 10 \text{Instrument Resolution} \le \frac{1}{10} \times \text{Tolerance Band} = \frac{\text{USL} - \text{LSL}}{10} Instrument Resolution ≤ 10 1 ​ × Tolerance Band = 10 USL − LSL ​

Instrument Resolution ≤ 1 10 × ( 6 σ process ) \text{Instrument Resolution} \le \frac{1}{10} \times (6\sigma_{\text{process}}) Instrument Resolution ≤ 10 1 ​ × ( 6 σ process ​ )

Symptoms and Consequences of Inadequate Resolution

If a gage violates the 10-to-1 rule (e.g., having a resolution of only 1/2 or 1/3 of the tolerance), the data suffers severe quantization distortion:

Comb-Like Histograms: Process histograms appear chopped into a few wide, blocky bars with empty gaps, completely destroying normal distribution curves.

Step Patterns on Control Charts: Run charts and control charts display unnatural horizontal steps where multiple consecutive points take identical values, generating false out-of-control signals.

Low Distinct Categories: In Gage R&R software, an inadequate resolution causes the Number of Distinct Categories ( n d c ndc n d c ) to collapse to 1, rendering capability indices ( C p / C p k C_p / C_{pk} C p ​ / C p k ​ ) mathematically invalid.

  • Worked Calculations: Total Variation Deconstruction & Bias Evaluation
  • Problem 1: Isolating True Process Standard Deviation

A precision machining line manufactures stainless steel spool valves. A quality engineer measures 100 consecutive valves and records an observed process standard deviation of σ TV = 0.080 mm \sigma_{\text{TV}} = 0.080\text{ mm} σ TV ​ = 0.080 mm . A subsequent MSA isolates the measurement system's standard deviation at σ MS = 0.048 mm \sigma_{\text{MS}} = 0.048\text{ mm} σ MS ​ = 0.048 mm .

What is the true part-to-part process standard deviation ( σ PV \sigma_{\text{PV}} σ PV ​ )?

Step 1: Convert Standard Deviations to Variances σ TV 2 = ( 0.080 ) 2 = 0.006400 mm 2 \sigma^2_{\text{TV}} = (0.080)^2 = 0.006400\text{ mm}^2 σ TV 2 ​ = ( 0.080 ) 2 = 0.006400 mm 2 σ MS 2 = ( 0.048 ) 2 = 0.002304 mm 2 \sigma^2_{\text{MS}} = (0.048)^2 = 0.002304\text{ mm}^2 σ MS 2 ​ = ( 0.048 ) 2 = 0.002304 mm 2

Step 2: Apply the Additivity of Variance to Solve for σ PV 2 \sigma^2_{\text{PV}} σ PV 2 ​ σ TV 2 = σ PV 2 + σ MS 2 \sigma^2_{\text{TV}} = \sigma^2_{\text{PV}} + \sigma^2_{\text{MS}} σ TV 2 ​ = σ PV 2 ​ + σ MS 2 ​ σ PV 2 = σ TV 2 − σ MS 2 = 0.006400 − 0.002304 = 0.004096 mm 2 \sigma^2_{\text{PV}} = \sigma^2_{\text{TV}} - \sigma^2_{\text{MS}} = 0.006400 - 0.002304 = 0.004096\text{ mm}^2 σ PV 2 ​ = σ TV 2 ​ − σ MS 2 ​ = 0.006400 − 0.002304 = 0.004096 mm 2

Step 3: Compute Process Standard Deviation σ PV = 0.004096 = 0.064 mm \sigma_{\text{PV}} = \sqrt{0.004096} = 0.064\text{ mm} σ PV ​ = 0.004096 ​ = 0.064 mm

Critical Exam Lesson: If the engineer had committed the common error of subtracting standard deviations directly ( 0.080 − 0.048 = 0.032 mm 0.080 - 0.048 = 0.032\text{ mm} 0.080 − 0.048 = 0.032 mm ), the estimated process variation would have been reported as 0.032 mm 0.032\text{ mm} 0.032 mm instead of the true value of 0.064 mm 0.064\text{ mm} 0.064 mm —an erroneous underestimate of 50% that would completely corrupt baseline capability!

Problem 2: Evaluating Measurement System Bias with Hypothesis Testing

A calibration lab evaluates a digital outside micrometer used on a critical shaft diameter. The certified master gage block has an accredited NIST-traceable dimension of X ref = 25.000 mm X_{\text{ref}} = 25.000\text{ mm} X ref ​ = 25.000 mm . The part tolerance is 25.000 ± 0.100 mm 25.000 \pm 0.100\text{ mm} 25.000 ± 0.100 mm (Total Tolerance = 0.200 mm = 0.200\text{ mm} = 0.200 mm ).

A technician records n = 20 n = 20 n = 20 repeated measurements on the master standard, obtaining:

  • Sample Mean: X ˉ = 25.012 mm \bar{X} = 25.012\text{ mm} X ˉ = 25.012 mm
  • Sample Standard Deviation: s = 0.015 mm s = 0.015\text{ mm} s = 0.015 mm

Step 1: Compute Observed Bias and Percent Bias Bias = X ˉ − X ref = 25.012 − 25.000 = + 0.012 mm \text{Bias} = \bar{X} - X_{\text{ref}} = 25.012 - 25.000 = +0.012\text{ mm} Bias = X ˉ − X ref ​ = 25.012 − 25.000 = + 0.012 mm % Bias = ( ∣ Bias ∣ Tolerance ) × 100 % = ( 0.012 mm 0.200 mm ) × 100 % = 6.0 % \%\text{Bias} = \left( \frac{|\text{Bias}|}{\text{Tolerance}} \right) \times 100\% = \left( \frac{0.012\text{ mm}}{0.200\text{ mm}} \right) \times 100\% = 6.0\% % Bias = ( Tolerance ∣ Bias ∣ ​ ) × 100% = ( 0.200 mm 0.012 mm ​ ) × 100% = 6.0%

  • Step 2: Conduct a One-Sample t t t -Test for Statistical Significance
  • H 0 : μ = 25.000 H_0: \mu = 25.000 H 0 ​ : μ = 25.000 (Zero Bias)
  • H 1 : μ ≠ 25.000 H_1: \mu \ne 25.000 H 1 ​ : μ  = 25.000 (Statistically Significant Bias)

Significance level: α = 0.05 \alpha = 0.05 α = 0.05 , Degrees of Freedom: d f = n − 1 = 19 df = n - 1 = 19 df = n − 1 = 19 .

Critical value from two-tailed t t t -table: t crit = 2.093 t_{\text{crit}} = 2.093 t crit ​ = 2.093 .

Calculate the test statistic: t = X ˉ − X ref s n = 0.012 0.015 20 = 0.012 0.015 4.4721 = 0.012 0.003354 = 3.578 t = \frac{\bar{X} - X_{\text{ref}}}{\frac{s}{\sqrt{n}}} = \frac{0.012}{\frac{0.015}{\sqrt{20}}} = \frac{0.012}{\frac{0.015}{4.4721}} = \frac{0.012}{0.003354} = 3.578 t = n ​ s ​ X ˉ − X ref ​ ​ = 20 ​ 0.015 ​ 0.012 ​ = 4.4721 0.015 ​ 0.012 ​ = 0.003354 0.012 ​ = 3.578

Step 3: Draw the Statistical and Practical Conclusion

Because ∣ t ∣ = 3.578 > 2.093 |t| = 3.578 > 2.093 ∣ t ∣ = 3.578 > 2.093 ( p < 0.01 p < 0.01 p < 0.01 ), the null hypothesis is rejected. The + 0.012 mm +0.012\text{ mm} + 0.012 mm bias is statistically significant and cannot be attributed to random measurement noise. The micrometer consumes 6.0% of the customer's tolerance band purely in systematic offset. The tool must be recalibrated and zero-adjusted before returning to the production line.

Critical Exam Traps to Avoid

Trap 1: Directly Adding or Subtracting Standard Deviations — Calculating total variation as σ TV = σ PV + σ MS \sigma_{\text{TV}} = \sigma_{\text{PV}} + \sigma_{\text{MS}} σ TV ​ = σ PV ​ + σ MS ​ . Standard deviations are never directly additive; variances are: σ TV 2 = σ PV 2 + σ MS 2 \sigma^2_{\text{TV}} = \sigma^2_{\text{PV}} + \sigma^2_{\text{MS}} σ TV 2 ​ = σ PV 2 ​ + σ MS 2 ​ .

Trap 2: Confusing Accuracy with Precision — Accuracy reflects centering and distance from a reference standard (Bias, Linearity, Stability). Precision reflects spread and repeatability among multiple trials (EV and AV). A gage can be highly precise while being completely inaccurate.

Trap 3: Overlooking Gage Resolution Violations (The 10-to-1 Rule) — Using a digital caliper with 0.01 mm 0.01\text{ mm} 0.01 mm resolution to inspect a part with a tolerance of ± 0.02 mm \pm 0.02\text{ mm} ± 0.02 mm (total tolerance 0.04 mm 0.04\text{ mm} 0.04 mm ). The resolution is 1/4 of tolerance, violating the 10-to-1 rule and generating discrete data chunking.

Trap 4: Neglecting Out-of-Calibration Containment — Assuming that discovering an out-of-calibration gage only requires fixing the tool. Quality systems strictly require tracing and re-evaluating all products shipped since the last valid calibration check.

Loading diagram... Measurement System Metrology, Error Components, and Resolution Flowdown