Study guide
6.7 Attribute Agreement Analysis and Audit MSA
ASQ Certified Six Sigma Black Belt CSSBB
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Introduction & CSSBB Exam Strategy
1.1 CSSBB Exam Format, Scoring, and Testing Windows 1.2 Eligibility, Project Affidavits, Application, and Recertification 1.3 Open-Book Strategy, Reference Materials, and a 16-Week Study Plan
2 Chapter 2: Organization-Wide Planning and Deployment
2.1 Fundamentals of Six Sigma and Lean Methodologies 2.2 Selecting the Right Methodology: DMAIC, DFSS, and Lean 2.3 Business Systems, Processes, and Stakeholder Relationships 2.4 Strategic Planning and Deployment of Six Sigma Initiatives 2.5 Six Sigma Roles and Responsibilities 2.6 Organizational Barriers and Common Causes of Six Sigma Failure 2.7 Change Management: Stakeholder Analysis, Readiness, and Communication
3 Chapter 3: Organizational Process Management and Measures
3.1 Impact of Six Sigma Projects on Customers, Suppliers, and Stakeholders 3.2 Benchmarking Types and Performance Goal Setting 3.3 Performance Measures: Balanced Scorecard, KPIs, KBIs, OKRs, and Line of Sight 3.4 Leading and Lagging Indicators and Performance Dashboards 3.5 Financial Measures: Revenue Growth, Margin, NPV, ROI, and Cost-Benefit Analysis 3.6 Cost of Poor Quality and Hard versus Soft Savings
4 Chapter 4: Team Management
4.1 Team Types and Constraints 4.2 Team Roles, Responsibilities, and Member Selection 4.3 Team Success Factors and Ground Rules 4.4 Motivational Techniques and Overcoming Demotivators 4.5 Stages of Team Development and Conflict Resolution 4.6 Team Communication Plans 4.7 Team Leadership Models: Direct, Coach, Support, and Delegate 4.8 Group Behaviors, Conflict Resolution, and Meeting Management 4.9 Facilitation Techniques and Team Decision-Making Methods 4.10 Team Training: Needs Assessment, Delivery, and Evaluation
5 Chapter 5: Define Phase
5.1 Customer Identification and Segmentation 5.2 Customer Data Collection: Methods, Validity, and Reliability 5.3 Customer Requirements: CTX, CTQ Trees, and Operational Definitions 5.4 The Kano Model and Quality Function Deployment 5.5 SIPOC and Process Mapping 5.6 Business Case Justification 5.7 The Project Charter and the Problem Statement 5.8 Project Scope and Boundaries 5.9 SMART Goals and Project Performance Measurements 5.10 Project Charter Reviews and Toll-Gate Governance 5.11 Project Management Tools: Gantt Charts, Toll-Gate Reviews, WBS, and RACI 5.12 Analytical Tools Used Throughout the DMAIC Cycle
6 Chapter 6: Measure Phase
6.1 Process Flow Metrics and the Hidden Factory 6.2 Process Analysis Tools and Value Stream Mapping 6.3 Types of Data and Measurement Scales 6.4 Sampling Concepts, Methods, and Sample Size for Estimation 6.5 Data Collection Plans and Common Pitfalls 6.6 Gage R&R for Variable Measurement Systems 6.7 Attribute Agreement Analysis and Audit MSA 6.8 Measurement Systems Across the Organization, and Metrology 6.9 Population Parameters, Sample Statistics, and Valid Statistical Conclusions 6.10 Descriptive Statistics: Central Tendency, Dispersion, and Shape 6.11 Graphical Methods for Exploratory Data Analysis 6.12 Probability Concepts, Common Distributions, and the Central Limit Theorem 6.13 Additional Distributions: Hypergeometric, Exponential, Lognormal, Weibull, and Bivariate 6.14 Process Capability and Performance Indices 6.15 Capability Studies, Attributes Capability, Yield Metrics, and Long-Term Capability
7 Chapter 7: Analyze Phase
7.1 The Correlation Coefficient, Its Confidence Interval, and Causation 7.2 Linear Regression and Residual Analysis 7.3 Multivariate Tools: Factor Analysis, Discriminant Analysis, and MANOVA 7.4 Hypothesis Testing Terminology, Errors, and Single-Sample Tests 7.5 Statistical Versus Practical Significance 7.6 Sample Size for Hypothesis Tests 7.7 Point Estimates, Confidence, Prediction, and Tolerance Intervals 7.8 Tests for Means, Variances, and Proportions 7.9 Analysis of Variance 7.10 Chi-Square Goodness-of-Fit and Tests of Independence 7.11 Types of Risk and Failure Mode and Effects Analysis 7.12 Gap Analysis and Root Cause Analysis 7.13 Waste Analysis: The Seven Classic Wastes and Resource Under-Utilization
8 Chapter 8: Improve Phase
8.1 Design of Experiments: Objectives, Terminology, and Effects 8.2 DOE Design Principles: Randomization, Blocking, Replication, and Resolution 8.3 Planning and Evaluating Designed Experiments 8.4 One-Factor Experiments: Completely Randomized, Randomized Block, and Latin Square Designs 8.5 Full Factorial Experiments and Design Selection 8.6 Two-Level Fractional Factorial Experiments and Confounding 8.7 Response Surface Methodology and Optimization 8.8 Waste Elimination: Pull Systems, Kanban, 5S, and Poka-Yoke 8.9 Cycle-Time Reduction: Continuous Flow, SMED, and Heijunka 8.10 Kaizen, Kaizen Blitz, Theory of Constraints, and OEE 8.11 Implementation Planning, Piloting, and Solution Selection
9 Chapter 9: Control Phase
9.1 SPC Objectives, Variable Selection, and Rational Subgrouping 9.2 Control Charts for Variables Data 9.3 Control Charts for Attributes Data 9.4 Short-Run SPC and Moving Average Charts 9.5 Control Chart Analysis: Common Cause, Special Cause, and Run Rules 9.6 Total Productive Maintenance and Visual Controls 9.7 Measurement System Reanalysis 9.8 Control Plan Development and Process Owner Handoff 9.9 Lessons Learned, Documentation, and Training for Sustainment 9.10 Ongoing Evaluation of the Improved Process
10 Chapter 10: Design for Six Sigma (DFSS)
10.1 DFSS Frameworks: DMADV, DMADOV, and Design Transfer 10.2 Design for X (DFX) 10.3 Robust Design, Parameter Design, and Verification 10.4 Tolerance Design and Statistical Tolerancing
6.7 Attribute Agreement Analysis and Audit MSA
Key Takeaways Explain This Topic AI
Attribute Agreement Analysis validates discrete inspection systems (pass/fail, go/no-go, visual defect categories).
Within-Appraiser Agreement assesses repeatability (inspector consistency across repeated trials), targeting >= 90%.
Fleiss' Kappa (kappa) measures agreement beyond chance, where kappa > 0.75 indicates good to excellent agreement, 0.40 <= kappa <= 0.75 indicates moderate agreement, and kappa < 0.40 indicates poor agreement.
Kendall's Coefficient of Concordance (W) evaluates ordinal attribute agreement among inspectors ranking defect severity categories.
Last updated: August 2026
While continuous MSA evaluates physical gauges using numeric measurements, many quality decisions in production and administrative processes rely on discrete attribute inspection (e.g., visual surface defect classification, go/no-go plug gauges, invoice audit pass/fail decisions, call center compliance scoring). Subjective human judgment introduces significant inspection error. Attribute Measurement System Analysis (also termed Attribute Agreement Analysis ) provides a structured statistical methodology to evaluate inspector consistency, repeatability, reproducibility, and accuracy against a verified master standard.
Experimental Design of an Attribute Agreement Study
To yield valid statistical conclusions, an Attribute Agreement study requires rigorous experimental design:
Sample Selection : Select 30 to 50 physical parts or transaction records. Crucially, the samples must cover the entire operational quality spectrum: approximately 50 % 50\% 50% conforming (clear pass), 25 % 25\% 25% non-conforming (clear fail), and 25 % 25\% 25% marginal / boundary samples (near specification limits).
Appraisers : Include 2 or 3 inspectors who routinely perform the inspection in daily operations.
Trials : Each inspector evaluates all samples across 2 or 3 independent trials in fully randomized order, blinding inspectors to sample identity.
Master Standard : A verified expert or precise destructive laboratory test establishes the true reference standard classification (Pass or Fail) for every sample.
Core Agreement Metrics & Benchmarks
Attribute MSA evaluates agreement across four distinct statistical dimensions:
Agreement Metric Operational Definition Operational Target Benchmark
Within-Appraiser Agreement (Repeatability) Percentage of samples for which an individual inspector provides identical decisions across all repeated trials. ≥ 90 % \ge 90\% ≥ 90% (Ideally ≥ 95 % \ge 95\% ≥ 95% )
Between-Appraiser Agreement (Reproducibility) Percentage of samples for which all inspectors agree with each other across all trials. ≥ 80 % \ge 80\% ≥ 80% (Ideally ≥ 90 % \ge 90\% ≥ 90% )
Appraiser-to-Standard Agreement (Accuracy) Percentage of samples for which an inspector's decision agrees with the true Master Standard. ≥ 90 % \ge 90\% ≥ 90% (Ideally ≥ 95 % \ge 95\% ≥ 95% )
Overall System Agreement Percentage of samples where ALL inspectors agree with each other AND agree with the Master Standard across all trials. ≥ 80 % \ge 80\% ≥ 80% (Ideally ≥ 90 % \ge 90\% ≥ 90% )
Statistical Indicators of Agreement: Cohen's & Fleiss' Kappa
Simple percentage agreement can be deceptive because appraisers can agree purely by random chance (e.g., guessing on a binary pass/fail test yields 50 % 50\% 50% agreement by chance). Six Sigma requires statistical metrics that correct for chance agreement:
Cohen's Kappa ( k k k ) & Fleiss' Kappa Formulation
Kappa ( κ \kappa κ ) measures the degree of agreement between categorical ratings beyond what would be expected by random chance alone: κ = P o − P e 1 − P e \kappa = \frac{P_o - P_e}{1 - P_e} κ = 1 − P e P o − P e
Where:
P o P_o P o = Observed proportion of concordant agreements across samples.
P e P_e P e = Expected proportion of agreements occurring purely by random chance.
Calculation of P e P_e P e for a 2x2 Attribute Table
Given two appraisers inspecting N N N items into Category 1 (Pass) and Category 2 (Fail): P e = P both pass + P both fail = ( p A 1 × p B 1 ) + ( p A 2 × p B 2 ) P_e = P_{\text{both pass}} + P_{\text{both fail}} = (p_{A1} \times p_{B1}) + (p_{A2} \times p_{B2}) P e = P both pass + P both fail = ( p A 1 × p B 1 ) + ( p A 2 × p B 2 )
Interpretation of Kappa ( κ \kappa κ )
κ = 1.00 \kappa = 1.00 κ = 1.00 : Perfect agreement.
κ > 0.75 \kappa > 0.75 κ > 0.75 : Good to Excellent Agreement (Measurement system is capable).
0.40 ≤ κ ≤ 0.75 0.40 \le \kappa \le 0.75 0.40 ≤ κ ≤ 0.75 : Fair to Moderate Agreement (Marginal; inspection criteria need standardization).
κ < 0.40 \kappa < 0.40 κ < 0.40 : Poor Agreement (Unacceptable; inspectors are guessing).
Kendall's Coefficient of Concordance ( W W W )
When attribute inspection involves ordinal rankings (such as ranking defect severity on a scale from 1 = Minor Cosmetic to 5 = Critical Safety Failure), Kendall's W W W is used. Kendall's W W W ranges from 0.00 0.00 0.00 (no agreement) to 1.00 1.00 1.00 (complete concordance among appraisers).
Inspector Misclassification Risk Analysis
When inspectors disagree with the Master Standard, classification errors are divided into two operational risks: Miss Rate (Type I Error / Consumer’s Risk) = Number of Bad Parts Classified as Good Total Actual Bad Parts × 100 % \text{Miss Rate (Type I Error / Consumer's Risk)} = \frac{\text{Number of Bad Parts Classified as Good}}{\text{Total Actual Bad Parts}} \times 100\% Miss Rate (Type I Error / Consumer’s Risk) = Total Actual Bad Parts Number of Bad Parts Classified as Good × 100% False Alarm Rate (Type II Error / Producer’s Risk) = Number of Good Parts Classified as Bad Total Actual Good Parts × 100 % \text{False Alarm Rate (Type II Error / Producer's Risk)} = \frac{\text{Number of Good Parts Classified as Bad}}{\text{Total Actual Good Parts}} \times 100\% False Alarm Rate (Type II Error / Producer’s Risk) = Total Actual Good Parts Number of Good Parts Classified as Bad × 100%
Miss Rate Risk : Non-conforming parts are shipped to customers, threatening product safety and customer satisfaction (Target ≤ 2 % \le 2\% ≤ 2% ).
False Alarm Rate Risk : Conforming parts are scrapped or reworked, creating unnecessary scrap costs and capacity losses (Target ≤ 5 % \le 5\% ≤ 5% ).
Corrective Action Strategies for Attribute MSA Failures
Physical Limit Samples (Boundary Boards) : Mount physical reference specimens representing the maximum allowable defect size at every inspection station.
Lighting & Ergonomics : Standardize ambient lighting intensity (lux level) and viewing angle in inspection booths.
Inspection SOPs : Eliminate vague visual criteria (e.g., "paint must look smooth") with quantitative visual definitions.
Worked Attribute Agreement Analysis Numerical Example
An audit manager evaluates a = 2 a = 2 a = 2 auditors (Appraiser A and Appraiser B) conducting compliance audits on N = 30 N = 30 N = 30 loan files across t = 2 t = 2 t = 2 independent trials. The files are benchmarked against a Master Standard ( 20 20 20 compliant files, 10 10 10 non-compliant files).
1. Cross-Tabulation Agreement Matrix (Appraiser A vs. Appraiser B)
- Appraiser A \ Appraiser B Pass (Compliant) Fail (Non-Compliant) Total
- Pass (Compliant) 18 2 20
- Fail (Non-Compliant) 1 9 10
- Total 19 11 30
- 2. Calculation of Observed Agreement ( P o P_o P o ) and Chance Agreement ( P e P_e P e )
Observed Proportion ( P o P_o P o ) : P o = 18 + 9 30 = 27 30 = 0.900 ( 90.0 % ) P_o = \frac{18 + 9}{30} = \frac{27}{30} = 0.900 \quad (90.0\%) P o = 30 18 + 9 = 30 27 = 0.900 ( 90.0% )
Expected Proportion by Chance ( P e P_e P e ) : P both pass = ( 20 30 ) × ( 19 30 ) = 380 900 = 0.4222 P_{\text{both pass}} = \left( \frac{20}{30} \right) \times \left( \frac{19}{30} \right) = \frac{380}{900} = 0.4222 P both pass = ( 30 20 ) × ( 30 19 ) = 900 380 = 0.4222 P both fail = ( 10 30 ) × ( 11 30 ) = 110 900 = 0.1222 P_{\text{both fail}} = \left( \frac{10}{30} \right) \times \left( \frac{11}{30} \right) = \frac{110}{900} = 0.1222 P both fail = ( 30 10 ) × ( 30 11 ) = 900 110 = 0.1222 P e = 0.4222 + 0.1222 = 0.5444 P_e = 0.4222 + 0.1222 = 0.5444 P e = 0.4222 + 0.1222 = 0.5444
3. Compute Cohen's Kappa ( κ \kappa κ ) κ = P o − P e 1 − P e = 0.900 − 0.5444 1 − 0.5444 = 0.3556 0.4556 = 0.7805 \kappa = \frac{P_o - P_e}{1 - P_e} = \frac{0.900 - 0.5444}{1 - 0.5444} = \frac{0.3556}{0.4556} = 0.7805 κ = 1 − P e P o − P e = 1 − 0.5444 0.900 − 0.5444 = 0.4556 0.3556 = 0.7805
Conclusion : Since κ = 0.7805 > 0.75 \kappa = 0.7805 > 0.75 κ = 0.7805 > 0.75 , the measurement system exhibits good to excellent inter-appraiser agreement .
Attribute MSA Troubleshooting & Best Practices
Boundary / Limit Samples : Establish visual physical limit boards representing maximum allowable defect thresholds at all inspection stations.
Blinding Operators : Ensure operators do not know which samples are reference standards or boundary samples during testing.
Periodic Re-certification : Conduct quarterly Attribute MSA studies to prevent inspector bias drift.
- Audit MSA for Attribute Measurement Systems
- The Body of Knowledge asks specifically that Black Belts use audit MSA for attribute measurement
- systems . An attribute agreement study answers whether appraisers agree; an audit MSA answers
- whether the whole inspection system -- appraiser, standard, procedure, environment, and decision rule --
produces the correct classification in normal operating conditions.
How it differs from a standard attribute agreement study
- Attribute agreement study Audit MSA
- Setting A designed trial with a curated part set The live inspection process
- Sample Selected to include borderline items, often 50/50 good and bad Drawn from actual production flow
What is evaluated Appraiser repeatability, reproducibility, and accuracy The complete inspection system, including procedure, standards, lighting, pacing, and workload
Known standard Established in advance by an expert panel Established by re-inspection, a reference method, or downstream escape data
Blinding Appraisers know they are in a study Ideally the audit is invisible to the appraiser
Detects Judgment inconsistency Judgment inconsistency plus systemic problems: ambiguous limit samples, inadequate lighting, takt-time pressure, missing standards
- The difference matters because an inspection system can pass a controlled study and still fail in
- production. Appraisers who agree perfectly when handed 30 parts in a quiet room may miss defects when
inspecting at takt on a moving line, or may drift after two hours.
- Running an audit MSA
- Define the audit sample. Draw units from the live flow immediately after the inspection station,
- including both accepted and rejected units. Rejected units matter: false rejects are as costly as
escapes and are invisible if only accepted units are audited.
Establish the truth. Re-inspect with a reference method: a more capable gauge, an expert panel, or
a downstream verification station.
Build the confusion matrix.
- Reference: defective Reference: good
- Inspector: defective Correct reject False reject (alpha, producer's risk)
- Inspector: good Escape (beta, consumer's risk) Correct accept
Compute the two error rates. Miss rate = Escapes Total truly defective , False alarm rate = False rejects Total truly good \text{Miss rate} = \frac{\text{Escapes}}{\text{Total truly defective}}, \qquad \text{False alarm rate} = \frac{\text{False rejects}}{\text{Total truly good}} Miss rate = Total truly defective Escapes , False alarm rate = Total truly good False rejects
- Audit the conditions, not just the outcome. Are limit samples current, unambiguous, and physically
- present? Is the standard written with decision rules for the borderline cases? Is the lighting to
- specification? Is the inspection time available at takt? Is the inspector rotating tasks to avoid
vigilance decay?
Set a recurring interval and put it in the control plan, so the audit happens by schedule.
- Typical findings
- Human visual inspection is conventionally around 80% effective at detecting the defects present, and
- audit MSA is how an organization discovers where on that scale it sits. The most common systemic
- findings are ambiguous or missing limit samples, a written standard that does not resolve the borderline
- cases, and insufficient inspection time at the current takt -- none of which appear in a controlled
attribute agreement study.
- The corrective actions are correspondingly systemic: rewrite the decision rules, refresh the limit
- samples, fix the lighting, re-balance the line to release inspection time, or move detection upstream to
error-proofing so that judgment is not required at all.