Study guide

11.3 Training Program Evaluation & Effectiveness Tracking

ASQ Certified Manager of Quality/Organizational Excellence CMQ/OE

Track your progress as you read

Finished sections get a check mark.

1 Chapter 0: Introduction & Exam Overview

0.1 Current CMQ/OE Exam Facts 0.2 Study Strategy by BoK Weight

2 Chapter 1: Leadership Principles & Organizational Structures

1.1 Leadership Theories & Styles 1.2 Organizational Structures & Matrix Management 1.3 Change Management Models & Strategies 1.4 Organizational Culture & Quality Governance

3 Chapter 2: Teams, Facilitation & Ethics

2.1 Team Formation & Development Stages 2.2 Facilitation Techniques & Team Empowerment 2.3 Conflict Resolution & Negotiation Strategies 2.4 ASQ Code of Ethics & Legal Compliance

4 Chapter 3: Strategic Planning & Environmental Analysis

3.1 Strategic Planning Models & Environmental Scans 3.2 Strategic Risk Management & Scenario Planning 3.3 Portfolio Management & Strategic Project Alignment

5 Chapter 4: Policy Deployment & Performance Alignment

4.1 Hoshin Kanri & Catchball Process 4.2 Benchmarking & Competitive Analysis 4.3 Knowledge Management & Best Practice Transfer

6 Chapter 5: Quality Management Systems & Cost of Quality

5.1 Quality Management System (QMS) Frameworks 5.2 Cost of Quality (COQ) Models 5.3 Process Management & Business Process Reengineering 5.4 Management Theories, Styles & Emotional Intelligence 5.5 Communication Skills for Quality Managers 5.6 Project Management for Quality Leaders

7 Chapter 6: Quality Improvement Frameworks & Auditing

6.1 Continuous Improvement Frameworks 6.2 Quality Auditing & Compliance Management 6.3 Measurement Systems & Key Performance Indicators

8 Chapter 7: Seven Basic & Management Quality Tools

7.1 The Seven Basic Quality Tools 7.2 The Seven Management & Planning Tools

9 Chapter 8: Advanced Problem Solving & Risk Analysis

8.1 Failure Mode and Effects Analysis (FMEA) 8.2 Quality Function Deployment (QFD) & House of Quality 8.3 Statistical Process Control & Process Capability 8.4 Lean Tools & Theory of Constraints 8.5 Sampling, MSA & Reliability Terminology

10 Chapter 9: Customer-Focused Organizations & VOC

9.1 Voice of the Customer & Kano Model 9.2 Customer Satisfaction, Loyalty & NPS Metrics 9.3 Customer Relationship Management & Complaint Resolution 9.4 Internal Customers, Segmentation & Multi-Customer Management

11 Chapter 10: Supply Chain Quality Management

10.1 Supplier Qualification, Selection & Evaluation 10.2 Supply Chain Risk Management & Logistics Alignment 10.3 Supplier Audits & Scorecard Systems 10.4 Supplier Risk Management & Communications

12 Chapter 11: Training and Workforce Development

11.1 Training Needs Analysis & Competency Modeling 11.2 Adult Learning Principles & Instructional Design 11.3 Training Program Evaluation & Effectiveness Tracking

11.3 Training Program Evaluation & Effectiveness Tracking

Key Takeaways Explain This Topic AI

Kirkpatrick's 4-Level Evaluation Model measures training impact at Level 1 (Reaction), Level 2 (Learning), Level 3 (Behavior/Transfer), and Level 4 (Results).

Phillips Level 5 ROI extends Kirkpatrick by calculating net financial benefits divided by total training costs: ROI (%) = (Net Benefits / Total Costs) x 100.

Level 3 (Behavior/Transfer) is the critical link in quality training, evaluating whether employees apply new skills on the shop floor 30, 60, or 90 days post-training.

Total Training Costs must capture both direct expenses (instructors, materials, LMS) and indirect expenses (participant wages, lost production time, facility overhead).

ISO 9001:2015 Clause 7.2 requires evaluating the effectiveness of training interventions, shifting focus from tracking training hours to validating performance capability.

Last updated: July 2026

11.3 Training Program Evaluation & Effectiveness Tracking

Quick Summary: Evaluating training effectiveness is essential to verify that workforce development investments produce measurable behavioral change and operational improvements. Using Kirkpatrick's Four-Level Evaluation Framework and Phillips' Level 5 ROI Formula , quality managers assess learner reaction, knowledge acquisition, workplace behavior transfer, business results, and financial return on investment.

Organizations spend substantial financial and human capital on workforce training. However, historically, quality managers have relied on superficial metrics—such as total training hours completed or immediate participant satisfaction scores ("smile sheets")—to justify training budgets. In modern quality management, training is treated as a strategic investment that must demonstrate verified competence, reduced operational risk, and positive financial return.

The Kirkpatrick Four-Level Evaluation Framework

Originally developed by Dr. Donald Kirkpatrick, this framework is the global standard for evaluating educational and training programs across four sequential levels: Level 4: RESULTS ───────────> Business Impact (Scrap reduction, ROI, Yield)

  • Level 3: BEHAVIOR ──────────> Job Application (Transfer of learning on Gemba)
  • Level 2: LEARNING ──────────> Knowledge/Skill Gain (Pre/post-tests, simulations)
  • Level 1: REACTION ──────────> Learner Satisfaction (Surveys, utility ratings)
  • Level 1: Reaction

Level 1 measures how participants respond to the training experience. It evaluates learner satisfaction, engagement, instructor effectiveness, and perceived utility of the content.

Instruments: Post-course surveys, numeric rating scales, open-ended feedback.

Focus: Affective reaction (Did they like it?) and Utility reaction (Do they believe it will help their work?).

Limitation: High satisfaction does not guarantee that learning occurred or that behavior will change.

Level 2: Learning

Level 2 measures the degree to which participants acquired the intended knowledge, skills, attitudes, confidence, and commitment during the training.

Instruments: Written pre/post-tests, hands-on skill demonstrations, rubric-scored performance assessments.

Focus: Quantifying the delta between entry knowledge and exit proficiency.

Limitation: Demonstrating skill in a classroom does not guarantee the employee will apply it consistently on the job.

Level 3: Behavior (Learning Transfer)

Level 3 measures the degree to which participants apply what they learned during training when they return to their actual work environment (the Gemba).

Instruments: 30/60/90-day post-training audits, direct supervisory observation checklists, 360-degree feedback, SPC chart audits.

Key Challenge: Learning transfer requires a supportive work environment. Barriers to transfer include lack of supervisor reinforcement, time pressure, and peer resistance.

Level 4: Results

Level 4 measures the targeted business outcomes resulting from the training program.

Instruments: Operational KPIs, Cost of Poor Quality (COPQ) tracking, scrap and rework logs, customer warranty return rates, safety incident counts.

Focus: Demonstrating direct connection between training execution and macro business performance.

Kirkpatrick Level Comparison

Level Focus Question Timing Common Tools Difficulty & Value

1. Reaction Did participants find the training engaging and relevant? Immediate (End of class) Smile sheets, Likert-scale surveys Low effort / Low value

2. Learning Did participants acquire the targeted KSAs? End of course Pre/post-tests, skill demonstrations Moderate effort / Moderate value

3. Behavior Are participants applying new skills on the job? 30 to 90 days post-training Gemba observations, process audits High effort / High value

4. Results Did the training produce desired business metrics? 3 to 12 months post-training Scrap logs, COPQ reports, yield metrics Highest effort / Highest value

Phillips Level 5: Return on Investment (ROI)

Developed by Dr. Jack Phillips, Level 5 ROI extends Kirkpatrick's framework by converting Level 4 operational results into monetary value and comparing them to total program costs.

1. Accounting for Total Training Costs

To calculate an accurate ROI, organizations must capture all direct and indirect costs:

Direct Costs: Instructor fees, courseware materials, facility rentals, e-learning software licenses, travel expenses.

Indirect Costs: Participant wages during training hours, lost production opportunity costs, administrative setup time. Total Training Costs = Direct Costs + Indirect Costs \text{Total Training Costs} = \text{Direct Costs} + \text{Indirect Costs} Total Training Costs = Direct Costs + Indirect Costs

2. ROI Formulas Net Financial Benefits = Gross Monetary Benefits − Total Training Costs \text{Net Financial Benefits} = \text{Gross Monetary Benefits} - \text{Total Training Costs} Net Financial Benefits = Gross Monetary Benefits − Total Training Costs ROI (%) = ( Net Financial Benefits Total Training Costs ) × 100 \text{ROI (\%)} = \left( \frac{\text{Net Financial Benefits}}{\text{Total Training Costs}} \right) \times 100 ROI (%) = ( Total Training Costs Net Financial Benefits ​ ) × 100 Benefit-Cost Ratio (BCR) = Gross Monetary Benefits Total Training Costs \text{Benefit-Cost Ratio (BCR)} = \frac{\text{Gross Monetary Benefits}}{\text{Total Training Costs}} Benefit-Cost Ratio (BCR) = Total Training Costs Gross Monetary Benefits ​

3. Isolating the Effects of Training

To ensure ROI validity, quality managers must isolate training impact from confounding operational variables (e.g., new raw material suppliers, new equipment, seasonal shifts). Common isolation methods include:

Control Group Analysis: Comparing trained operator cohorts against an untrained control group.

Trend Line Analysis: Projecting pre-training performance trends and measuring post-training divergence.

Expert & Customer Estimation: Gathering conservative impact estimates from process engineers and supervisors.

ISO 9001:2015 Clause 7.2 Effectiveness Tracking

ISO 9001:2015 Clause 7.2 specifically requires organizations to "evaluate the effectiveness of the actions taken" to address competence gaps. Internal quality auditors will look for evidence that:

Training needs were identified based on role requirements.

Training was delivered as planned.

Effectiveness was evaluated using objective criteria (Level 3 or 4), rather than merely checking attendance logs.

Retraining or corrective actions were initiated if effectiveness criteria were not met.

CMQ/OE Exam Tip: The exam often asks how to evaluate whether a training program was successful. Be ready to identify that Level 3 (Behavior) and Level 4 (Results) provide true effectiveness evidence under ISO 9001. A Level 1 survey showing 95% satisfaction is insufficient evidence of competence under ISO audits!

Practical Example: Six Sigma Green Belt ROI Calculation

A manufacturing plant enrolled 10 manufacturing engineers in a Six Sigma Green Belt certification course to address a recurring machining defect.

Financial Data:

Direct Training Costs: Tuition ($15,000) + Course Materials ($3,000) + Software Licenses ($2,000) = $20,000

Indirect Training Costs: Engineer Wages during 80 hours of class time ($25,000) + Admin Overhead ($5,000) = $30,000

Total Training Costs: $20,000 + $30,000 = $50,000

Operational Results (12 Months Post-Training):

Control group analysis confirmed scrap reduction directly attributable to Green Belt project = $200,000 annual savings

Calculations:

  • Net Financial Benefits: $200,000 - $50,000 = $150,000
  • Benefit-Cost Ratio (BCR): $200,000 / $50,000 = 4.0 to 1
  • Return on Investment (ROI): ($150,000 / $50,000) x 100 = 300%

Conclusion: For every $1.00 invested in Green Belt training, the organization recovered $4.00 in gross benefits, yielding a net ROI of 300% .

Six Sigma Green Belt Program: Training Costs vs. Scrap Savings ($)