Study guide

9.3 Systematically Reevaluating Assessments, Protocols, and Interventions

NSCA Certified Performance and Sport Scientist CPSS

Track your progress as you read

Finished sections get a check mark.

1 Introduction: CPSS Exam Overview, Eligibility, and Case-Study Strategy

1.1 CPSS Exam Format, Eligibility Pathways, and the 2026 Blueprint 1.2 Mastering Athlete Case Studies and Research Review Scenarios

2 Chapter 1: Scientific Process, Adaptation Theory, and Periodization (Domain 1)

2.1 Training Principles, Adaptation Models, and Injury-Risk Mitigation 2.2 Evaluating Performance Programs Using the Scientific Process 2.3 Periodization Paradigms, Fitness-Fatigue Models, and Tapering

3 Chapter 2: Scientific Disciplines in Athlete Preparation (Domain 1)

3.1 Exercise Physiology, Bioenergetics, and Systemic Adaptations 3.2 Applied Biomechanics, Force-Time Kinetics, and Force-Velocity Profiling 3.3 Motor Learning, Skill Acquisition, and Constraints-Led Practice Design 3.4 Performance Nutrition, Recovery Modalities, and Sport Psychology

4 Chapter 3: Technology Review, Research Appraisal, and Knowledge Translation (Domain 1)

4.1 Critically Reviewing Sport Performance Products, Technologies, and Trends 4.2 Evaluating Research Studies: Design, Validity, and Practical Relevance 4.3 Communicating Research Findings to Support Evidence-Based Practice

5 Chapter 4: Sport Demands Analysis, KPIs, and Resource Strategy (Domain 2)

5.1 Researching Biomechanical, Tactical, and Technical Sport Demands 5.2 Physiological Demands, Injury Epidemiology, and Environmental Constraints 5.3 Qualitative and Mixed-Methods Research for Sport Demands Analysis 5.4 Collaborating with Stakeholders to Establish Key Performance Indicators 5.5 Identifying Resources and Strategies for Short- and Long-Term Goals

6 Chapter 5: Profiling Benchmarks and Needs Analysis Communication (Domain 2)

6.1 Developing Normative Benchmarks: Z-Scores, T-Scores, and Percentiles 6.2 Multidimensional Athlete Profiling and Return-to-Performance Milestones 6.3 Communicating Needs Analysis Results to Multidisciplinary Stakeholders

7 Chapter 6: Assessment Ethics, Safety Standards, and Tool Psychometrics (Domain 3)

7.1 Ethical, Legal, and Privacy Considerations in Athlete Data Collection 7.2 Evaluating Assessment Environments, Equipment, and Safety Standards 7.3 Validity, Reliability (ICC, CV%, TE), and Smallest Worthwhile Change

8 Chapter 7: Applied Assessment and Load-Monitoring Systems (Domain 3)

8.1 Movement Tracking Systems: GNSS/GPS, LPS, IMUs, and Optical Tracking 8.2 Neuromuscular Testing: Force Platforms, Isometric Pulls/Pushes, and VBT 8.3 Internal Load, Autonomic Readiness (HRV), Biomarkers, and PROMs

9 Chapter 8: Quality Assurance, Protocols, and Assessment Efficacy (Domain 3)

9.1 Applying Quality Assurance Techniques in Sport Science Data Collection 9.2 Developing, Documenting, and Standardizing Data Collection Protocols 9.3 Systematically Reevaluating Assessments, Protocols, and Interventions

10 Chapter 9: Data Governance, Storage Architecture, and Cleaning (Domain 4)

10.1 Data Privacy, Security Protocols, and Protecting Organizational Information 10.2 Designing Plans for Data Collection, Storage, Analysis, and AMS Workflows 10.3 Evaluating Data Quality, Signal Integrity, Outliers, and Cleaning Procedures

11 Chapter 10: Statistical Modeling, Feedback Delivery, and Analytical Limitations (Domain 4)

11.1 Inferential Statistics, Correlation, Regression, ANOVA, and Effect Sizes 11.2 Longitudinal Load Modeling: ACWR, EWMA, Monotony, Strain, and Change Detection 11.3 Data Visualization, Dashboard Design, and Timely Stakeholder Feedback 11.4 Communicating Analytical Limitations, Bias, and Uncertainty

9.3 Systematically Reevaluating Assessments, Protocols, and Interventions

Key Takeaways Explain This Topic AI

Assessment batteries must be audited annually against four pruning criteria—ceiling/floor effects, multicollinearity (r > 0.85), poor predictive/ecological validity, and low Decision-Impact Ratio—to eliminate 'protocol creep.'

When an intervention fails to improve a target KPI, the sport scientist must first audit implementation fidelity and dose compliance to rule out a Type III Error before concluding that the physiological training model failed.

Individual athlete training responses are classified using the Typical Error (TE) and Smallest Worthwhile Change (SWC) into True Positive Responders, Unclear/Noise changes, True Non-Responders (requiring stimulus modification), and Negative Responders (requiring fatigue/recovery audits).

Upgrading testing hardware or modifying an SOP requires a parallel cross-calibration period (n = 20–30 athletes tested on both systems) using Bland-Altman agreement and Deming/Passing-Bablok regression, paired with explicit AMS protocol version tags (v1.0 → v2.0).

Last updated: October 2026

Systematically Reevaluating Assessments, Protocols, and Interventions

A sport science monitoring system is not a static monument; it is a closed-loop cybernetic feedback system. At the conclusion of every training block or competitive season, the sport scientist must turn the analytical lens inward to audit three distinct targets: (1) the assessment battery itself , (2) the efficacy and fidelity of the prescribed training or recovery interventions , and (3) the longitudinal continuity of database records when protocols evolve .

Auditing the Assessment Battery: Combatting "Protocol Creep"

Over multiple seasons, performance departments frequently suffer from "protocol creep" —adding new tests requested by incoming coaches, medical staff, or researchers without ever retiring legacy assessments. A bloated battery wastes valuable training time, induces cumulative athlete fatigue, degrades athlete buy-in ("testing fatigue"), and overwhelms coaches with redundant dashboards.

Every assessment in the battery must undergo an annual post-season audit against four quantitative pruning criteria :

1. Ceiling and Floor Effects (Range Restriction)

If an athlete population adapts or matures to the point where > 80 % > 80\% > 80% of the squad achieves the maximum possible score (a ceiling effect , such as senior elite athletes maxing out a basic movement screen or a submaximal shuttle stage) or clusters near the minimum ( floor effect ), the between-subject standard deviation ( S D between SD_{\text{between}} S D between ​ ) collapses toward zero. Mathematically, range restriction destroys both the test's Intraclass Correlation Coefficient ( ICC \text{ICC} ICC ) and its capacity to detect positive training adaptations. The protocol must be retired or upgraded to a more demanding biomechanical progression.

2. Redundancy and Multicollinearity ( r > 0.85 r > 0.85 r > 0.85 / R 2 > 0.72 R^2 > 0.72 R 2 > 0.72 )

By computing an inter-test correlation matrix or Principal Component Analysis ( PCA \text{PCA} PCA ) across all battery metrics, the sport scientist identifies collinear tests measuring the exact same underlying construct. When two assessments exhibit r > 0.85 r > 0.85 r > 0.85 (sharing > 72 % > 72\% > 72% common variance, or showing a Variance Inflation Factor VIF > 5 – 10 \text{VIF} > 5\text{–}10 VIF > 5 – 10 —for instance, 10 m 10\text{ m} 10 m and 20 m 20\text{ m} 20 m sprint splits, or unloaded CMJ \text{CMJ} CMJ peak power and Squat Jump peak power), retaining both provides negligible unique information. The practitioner should prune the test that has the inferior signal-to-noise ratio ( TE > SWC \text{TE} > \text{SWC} TE > SWC ) , higher setup complexity, or greater injury/fatigue cost.

3. Predictive, Discriminative, and Ecological Validity

Retrospective post-season modeling should test whether baseline or in-season changes in the assessment actually correlated with on-field Key Performance Indicators ( KPIs \text{KPIs} KPIs ), discriminated elite starters from developmental squad members, or predicted soft-tissue injury incidence (e.g., Receiver Operating Characteristic Area Under the Curve AUC \text{AUC} AUC ). An injury-screening metric that repeatedly yields an AUC ≈ 0.50 – 0.55 \text{AUC} \approx 0.50\text{–}0.55 AUC ≈ 0.50 – 0.55 (no better than flipping a coin) across multiple seasons should not be used for return-to-play clearance or prophylactic grouping.

4. Actionability Audit (The "Decision-Impact Ratio")

For every test in the battery, ask: "How many times during the past season did a score on this assessment directly trigger an individualized change in training prescription, load management, recovery modality, or tactical selection?" Any metric with a Decision-Impact Ratio of zero is merely "museum data" and must either be connected to an explicit, pre-agreed coaching decision threshold or removed from the battery.

Evaluating Intervention Efficacy: Implementation Fidelity vs. Biological Responsiveness

When a 6-to-8-week training or recovery intervention fails to produce a statistically or practically meaningful shift in the squad's target KPI at retesting, the sport scientist must systematically diagnose two distinct failure pathways before discarding the underlying physiological theory.

Pathway 1: Ruling Out Type III Error (Implementation and Dose-Fidelity Failure)

In applied sport science, a Type III Error occurs when a practitioner concludes that a physiological intervention was ineffective (or asks the wrong analytical question) when, in reality, the intervention was never executed as prescribed . Before blaming the physiological stimulus design, audit Training Dose Compliance and Implementation Fidelity :

Attendance and Volume Completion: Did the athlete actually complete ≥ 85 – 90 % \ge 85\text{–}90\% ≥ 85 – 90% of the planned sessions, sets, and repetitions, or were sessions repeatedly truncated by congested match travel and medical modifications?

Intensity and Velocity Fidelity: In a maximal strength or Velocity-Based Training ( VBT \text{VBT} VBT ) block targeting 0.50 m ⋅ s − 1 0.50\text{ m}\cdot\text{s}^{-1} 0.50 m ⋅ s − 1 ( ∼ 85 % 1 RM \sim 85\%\ 1\text{RM} ∼ 85% 1 RM ), did the athletes actually lift at the prescribed load and maximal intended concentric velocity, or did they under-load the bar? In a High-Intensity Interval Training ( HIIT \text{HIIT} HIIT ) block targeting ≥ 12 min \ge 12\text{ min} ≥ 12 min above 90 % HR max ⁡ 90\%\ \text{HR}_{\max} 90% HR m a x ​ , did internal heart-rate telemetry confirm that the physiological target zone was actually achieved?

Pathway 2: Individual Responder vs. Non-Responder Classification

Even when implementation fidelity is 100 % 100\% 100% , evaluating an intervention solely by the group mean change is deeply flawed because a near-zero group change often masks polarized subgroups of strong positive responders and negative responders. By combining the test's Typical Error ( TE \text{TE} TE ) and Smallest Worthwhile Change ( SWC = 0.2 × S D between \text{SWC} = 0.2 \times SD_{\text{between}} SWC = 0.2 × S D between ​ ) into an individual responsiveness threshold—either by plotting the athlete's observed change ( Δ = X post − X pre \Delta = X_{\text{post}} - X_{\text{pre}} Δ = X post ​ − X pre ​ ) with its 90 % 90\% 90% Confidence Interval ( Δ ± 1.645 × 2 × TE \Delta \pm 1.645 \times \sqrt{2} \times \text{TE} Δ ± 1.645 × 2 ​ × TE ) against the trivial band ( ± SWC \pm \text{SWC} ± SWC ), or by requiring Δ > TE 2 + SWC 2 \Delta > \sqrt{\text{TE}^2 + \text{SWC}^2} Δ > TE 2 + SWC 2 ​ —each athlete is classified into one of four actionable categories:

Classification Profile Quantitative Criterion ( TE = 40 W , SWC = 50 W \text{TE} = 40\text{ W}, \text{SWC} = 50\text{ W} TE = 40 W , SWC = 50 W ) Underlying Physiological / Operational Diagnosis Prescribed Next-Block Action

True Positive Responder Δ = + 135 W \Delta = +135\text{ W} Δ = + 135 W ( 90 % CI : + 42 to + 228 W 90\%\ \text{CI}: +42\text{ to } +228\text{ W} 90% CI : + 42 to + 228 W ; entire CI > 0 \text{CI} > 0 CI > 0 and Δ > TE 2 + SWC 2 = 64 W \Delta > \sqrt{\text{TE}^2 + \text{SWC}^2} = 64\text{ W} Δ > TE 2 + SWC 2 ​ = 64 W ) High dose fidelity ( 95 % 95\% 95% ) + favorable neuromuscular adaptation to the prescribed stimulus Retain current modality and apply progressive overload ( + 2.5 – 5.0 % +2.5\text{–}5.0\% + 2.5 – 5.0% intensity or complexity)

Low-Fidelity Non-Responder (Type III Error) Δ = + 10 W \Delta = +10\text{ W} Δ = + 10 W ( 90 % CI : − 83 to + 103 W 90\%\ \text{CI}: -83\text{ to } +103\text{ W} 90% CI : − 83 to + 103 W ) with only 52 % 52\% 52% session compliance Intervention was never delivered at an effective biological dose due to missed sessions Do not discard the training program; solve scheduling barriers and enforce dose compliance

True Biological Non-Responder Δ = + 5 W \Delta = +5\text{ W} Δ = + 5 W (trivial change within ± SWC \pm \text{SWC} ± SWC ) despite 98 % 98\% 98% dose and velocity compliance Athlete is insensitive or already saturated with respect to the current stimulus type Shift stimulus modality (e.g., switch a force-saturated athlete from heavy squats to ballistic/plyometric velocity-deficit work)

True Negative Responder (Maladaptation) Δ = − 120 W \Delta = -120\text{ W} Δ = − 120 W ( 90 % CI : − 213 to − 27 W 90\%\ \text{CI}: -213\text{ to } -27\text{ W} 90% CI : − 213 to − 27 W ; substantial decline beyond − SWC -\text{SWC} − SWC ) Excessive concurrent match load, non-functional overreaching, sleep debt, or Low Energy Availability ( RED-S \text{RED-S} RED-S ) Immediately audit total internal/external load, sleep, and nutrition; prescribe deload/taper block

Version Control and Longitudinal Continuity When Updating Protocols

When an audit reveals that a legacy device or protocol must be upgraded (for example, replacing a contact jump mat with dual piezoelectric force plates, or migrating from a single-beam timing system with a 1.0 m 1.0\text{ m} 1.0 m setback to a dual-beam system with a 0.30 m 0.30\text{ m} 0.30 m setback), abruptly swapping systems destroys multi-year longitudinal Z-scores and individual historical baselines.

To preserve longitudinal continuity without corrupting the database, the sport scientist must execute a three-step protocol migration workflow :

Parallel Overlap / Cross-Calibration Study ( n = 20 – 30 n = 20\text{–}30 n = 20 – 30 Athletes): Before retiring the legacy protocol ( v1.0 ), test a representative sample of 20 – 30 20\text{–}30 20 – 30 athletes simultaneously (or in counterbalanced order) on both the legacy ( v1.0 ) and updated ( v2.0 ) systems across the full span of team scores.

Bland-Altman Agreement and Deming / Passing-Bablok Regression:

Perform a Bland-Altman analysis to quantify the fixed systematic bias ( d ˉ = Y ˉ v2.0 − X ˉ v1.0 \bar{d} = \bar{Y}_{\text{v2.0}} - \bar{X}_{\text{v1.0}} d ˉ = Y ˉ v2.0 ​ − X ˉ v1.0 ​ ), check for proportional bias (whether the difference grows at higher performance values), and compute the 95 % 95\% 95% Limits of Agreement ( LoA = Bias ± 1.96 × S D diff \text{LoA} = \text{Bias} \pm 1.96 \times SD_{\text{diff}} LoA = Bias ± 1.96 × S D diff ​ ) .

Fit a Deming regression or non-parametric Passing-Bablok regression ( Y v2.0 = β 0 + β 1 X v1.0 Y_{\text{v2.0}} = \beta_0 + \beta_1 X_{\text{v1.0}} Y v2.0 ​ = β 0 ​ + β 1 ​ X v1.0 ​ ), which properly accounts for random measurement error in both the legacy and new instruments (unlike Ordinary Least Squares regression, which assumes X X X is error-free and attenuates the slope β 1 \beta_1 β 1 ​ ).

Decision Rule: For example, if a legacy optical jump mat ( v1.0 ) overestimates force-plate impulse-momentum jump height ( v2.0 ) by a fixed bias of + 4.2 cm +4.2\text{ cm} + 4.2 cm with a Deming conversion equation of Y v2.0 = − 3.8 + 0.98 X v1.0 Y_{\text{v2.0}} = -3.8 + 0.98 X_{\text{v1.0}} Y v2.0 ​ = − 3.8 + 0.98 X v1.0 ​ and a 95 % LoA 95\%\ \text{LoA} 95% LoA of ± 0.9 cm \pm 0.9\text{ cm} ± 0.9 cm (narrower than the squad's SWC \text{SWC} SWC of 1.2 cm 1.2\text{ cm} 1.2 cm ), historical v1.0 records can be mathematically harmonized into v2.0 -equivalent units. Conversely, if the random scatter ( LoA = ± 3.4 cm \text{LoA} = \pm 3.4\text{ cm} LoA = ± 3.4 cm ) exceeds the SWC \text{SWC} SWC , mathematical conversion is invalid for individual athletes, and a hard baseline reset must be declared.

Athlete Management System ( AMS \text{AMS} AMS ) Schema Versioning: Never overwrite or silently concatenate raw metrics from different protocols into the same database column. Tag every record with an explicit protocol_version ( v1.0 vs. v2.0 ), hardware_model , sampling_rate_hz , and scoring_method so longitudinal visualizations automatically display a protocol-transition vertical marker or filter by version.