Study guide
18.2 Item Analysis and Assessment Data
BCSP CIT Certified Instructional Trainer Study Guide 2026
Track your progress as you read
Finished sections get a check mark.
1 Credential and Training Standards
1.1 BCSP CIT Credential Scope, Exam Logistics & Recertification 1.2 ANSI/ASSP Z490 Criteria for Accepted EHS Training Practices
2 Adult Learning and Learning Access
2.1 Foundations of Andragogy: Knowles' Adult Learning Assumptions in Safety 2.2 Learner Literacy, ESL Accommodations & Plain Language Compliance 2.3 Opening a Course: Administrative Items and Learning Expectations
3 Communication and Facilitation
3.1 Verbal & Nonverbal Facilitation Techniques in Safety Training 3.2 Active Listening, Socratic Questioning & Teach-Back Verification 3.3 Feedback Frameworks, Corrective Coaching & Psychological Safety
4 Conflict and Instructional Technology
4.1 Managing Challenging Classroom Dynamics, Resistance & Conflict 4.2 Instructional Media, Presentation Tools & Technology Contingencies
5 Performance Needs and Existing Training
5.1 Performance Gaps and Training Necessity 5.2 Evaluating Existing Training and Interventions
6 Organizational, Task and Hazard Analysis
6.1 Organizational and Job Task Analysis 6.2 Hazard Analysis, KSAs and Task Evidence
7 Audience, Baseline and Prerequisites
7.1 Defining the Audience and Its Learning Needs 7.2 Baseline Knowledge and Prior-Learning Credit 7.3 Verifying Prerequisites and Readiness for Training
8 Applicable Training Requirements
8.1 OSHA Training Frameworks, General Duty Clause & Consensus Standards 8.2 Mandated Safety Programs: Lockout/Tagout, Confined Space & Fall Protection 8.3 Chemical & Biological Hazards: HazCom, Bloodborne & Respiratory Protection
9 Systematic and Iterative Design
9.1 ADDIE and the Systematic Approach to Training 9.2 SAM, Agile Development and Prototypes
10 Objectives and Learning Outcomes
10.1 Writing Observable ABCD Objectives 10.2 SMART, Objective Hierarchies and Stakeholder Consensus
11 Learning Domains and Practice
11.1 Cognitive Learning and Revised Bloom Taxonomy 11.2 Psychomotor, Affective and Constructive Alignment 11.3 Experiential Learning Cycles (Kolb), Adult Discovery & Instructional Scaffolding
12 Delivery Choices and Program Economics
12.1 Modality Matrix: Classroom ILT, E-Learning, Microlearning, Hands-On & OJT 12.2 Cost Classification and Worked Budgets 12.3 Resource Feasibility and Cost-Benefit Decisions
13 Developing and Validating Course Components
13.1 Lesson Plan Architecture: Components of Comprehensive Trainer & Participant Guides 13.2 Job Aids and Performance Support 13.3 SME Selection, Credible Sources and Validation
14 Development Planning and External Providers
14.1 Implementing the Instructional Development Plan 14.2 Deciding When and How to Outsource Development
15 Pilots and Recoverable Records
15.1 Representative Pilots and Course Revision 15.2 Record Retention, Privacy and Recovery
16 Implementation Environments and Logistics
16.1 Physical Classroom Design, Seating Arrangements, Ergonomics & Environmental Controls 16.2 Field, Shop & Hands-On Training: Safety Briefings, PPE Verification & Live Demonstration Safety 16.3 Virtual Instructor-Led Training (VILT), Technical Contingency Planning & Remote Engagement 16.4 Scheduling, Logistics and Course Time Management
17 Assessment and Remediation
17.1 Formative and Summative Assessment Tools 17.2 Assessment Criteria, Alignment and Remediation
18 Items, Performance and Standards
18.1 Writing Clear Items and Plausible Distractors 18.2 Item Analysis and Assessment Data 18.3 Performance Checklists and Evaluator Calibration 18.4 Standard Setting and the Angoff Calculation
19 Evaluation Models, Surveys and Learning Data
19.1 Evaluation Models and Feedback Surveys 19.2 Learning Outcomes and Evaluation Data
20 Transfer and Behavior Evidence
20.1 Transfer of Learning and Work Conditions 20.2 Behavior Observations and Change Reporting
21 Results, ROI and Improvement Reports
21.1 Organizational Results and Causal Limits 21.2 ROI, Course Improvement and Training Reports
18.2 Item Analysis and Assessment Data
Key Takeaways Explain This Topic AI
Item difficulty is the proportion correct for the stated population.
Upper-minus-lower discrimination can flag problems but does not diagnose their cause.
The worked counts reconcile to 100 learners, 60 correct and discrimination about minus 0.22.
Last updated: October 2026
Psychometric Evaluation: Item Analysis and Statistical Metrics
Following assessment administration, instructional trainers evaluate exam performance through classical test theory metrics to verify item health. Item Difficulty (p-value): p = R / N
Where: R = Number of examinees answering correctly, N = Total number of examinees
- Item Discrimination Index (D): D = P_U - P_L
- Where: P_U = Proportion correct in upper 27% group, P_L = Proportion correct in lower 27% group
- Item difficulty
The proportion correct is p = R / N p = R/N p = R / N . An item answered correctly by 60 of 100 candidates has p = 0.60 p = 0.60 p = 0.60 . A larger value means an easier item for that group. There is no universal acceptable range; an essential safety item may reasonably be easy after effective instruction. Consider the objective and sample before deciding whether to revise it.
Item discrimination
For a selected upper and lower group, D = P U − P L D = P_U - P_L D = P U − P L . The group selection method must be stated. A negative result prompts investigation of the key, wording, instruction and scoring; it does not prove a particular cause. In a small sample, results may be unstable. Resolve scoring concerns fairly before using the result for consequential decisions.
Distractor analysis
Check which options learners select and whether the pattern matches a plausible misconception. An unused distractor may be weak, but a fixed percentage is not a universal retirement rule. Review content, language and opportunity to learn before revising the item.
Calculating and interpreting a worked item analysis
Suppose 100 learners take a knowledge test. For one item, 60 answer correctly. The difficulty index is 60 / 100 = 0.60 60/100 = 0.60 60/100 = 0.60 . Despite its name, a larger value means the item was easier for this population. Do not interpret this as an intrinsic property independent of teaching or learner preparation.
For an illustrative discrimination calculation, select the highest-scoring 27 and lowest-scoring 27 learners based on overall test performance. Six upper-group learners and twelve lower-group learners answer this item correctly. The remaining 46 learners include 42 correct responses, so the total remains 60. The upper proportion is 6 / 27 6/27 6/27 , the lower is 12 / 27 12/27 12/27 , and D = 6 / 27 − 12 / 27 = − 6 / 27 D = 6/27 - 12/27 = -6/27 D = 6/27 − 12/27 = − 6/27 , approximately − 0.22 -0.22 − 0.22 . These internally consistent counts make a useful warning case.
The negative discrimination means this item was answered correctly less often by the selected upper group. It does not identify the cause by itself. Check the answer key, ambiguity, multiple defensible options, technical premise, scoring transfer and what was actually taught. A miskey is possible, but so are other problems. Investigate before using the questionable result in a consequential decision.
Finding Investigation
High proportion correct Essential mastery, easy cue or weak distractors?
Low proportion correct Difficult objective, instruction gap or ambiguous item?
Negative discrimination Key, wording, group selection and scoring correct?
Distractor rarely chosen Plausible misconception or obviously irrelevant option?
Linking analysis with content review
Statistics are supporting evidence. A high discrimination value does not prove technical accuracy; an item can separate groups while testing irrelevant vocabulary. An essential safety item that everyone answers correctly may still belong in a qualification assessment. Review the objective and content alongside the data.
For distractors, record counts by option and examine learner explanations where feasible. If many learners choose a plausible but incorrect action, determine whether it reflects a misconception that needs clearer instruction. If an option is absurd, replace it with a defensible novice error. Avoid making a revised distractor so similar to the key that two answers become valid.
Small samples require caution. A few responses can change an index substantially, and tied scores can complicate upper/lower grouping. Record the sample and grouping method. Do not claim psychometric equivalence merely because two forms have the same number of questions or similar averages. Form comparability requires appropriate content and measurement evidence.
Fair correction and reporting
If review confirms a key or scoring error, follow the applicable correction policy and communicate the effect on decisions. Preserve the original result, corrected result and rationale where the record policy requires it. Avoid concealing a defect by deleting a difficult item without investigating which learners or decisions were affected.
For a safety qualification, ensure unresolved measurement issues do not produce unsupported authorization. Other valid performance evidence may clarify competence, but a revised written score cannot stand in for a missing practical check. Plan remediation for genuine learning gaps separately from correction of the instrument.
Use a report that identifies the item, objective, sample, observed indices, review findings and action owner. After revision, gather new evidence rather than assuming the problem disappeared. This joins statistical analysis with technical review and instructional improvement.
Checking arithmetic before interpretation
Upper, lower and remaining groups must reconcile with the total sample. In this example, 27 plus 27 plus 46 equals 100, and six plus twelve plus 42 equals 60 correct responses. If the counts do not reconcile, correct the data before discussing item quality.
Keep rounding until the final reported value. The exact discrimination is minus six twenty-sevenths, approximately minus 0.22. A rounded index should not be mixed with incompatible percentage claims. Report the underlying counts where they help readers audit the result.
Key takeaways
Item difficulty is the proportion correct for the stated population.
Upper-minus-lower discrimination can flag problems but does not diagnose their cause.
The worked counts reconcile to 100 learners, 60 correct and discrimination about minus 0.22.