Study guide

4.2 Evaluating Research Studies: Design, Validity, and Practical Relevance

NSCA Certified Performance and Sport Scientist CPSS

Track your progress as you read

Finished sections get a check mark.

1 Introduction: CPSS Exam Overview, Eligibility, and Case-Study Strategy

1.1 CPSS Exam Format, Eligibility Pathways, and the 2026 Blueprint 1.2 Mastering Athlete Case Studies and Research Review Scenarios

2 Chapter 1: Scientific Process, Adaptation Theory, and Periodization (Domain 1)

2.1 Training Principles, Adaptation Models, and Injury-Risk Mitigation 2.2 Evaluating Performance Programs Using the Scientific Process 2.3 Periodization Paradigms, Fitness-Fatigue Models, and Tapering

3 Chapter 2: Scientific Disciplines in Athlete Preparation (Domain 1)

3.1 Exercise Physiology, Bioenergetics, and Systemic Adaptations 3.2 Applied Biomechanics, Force-Time Kinetics, and Force-Velocity Profiling 3.3 Motor Learning, Skill Acquisition, and Constraints-Led Practice Design 3.4 Performance Nutrition, Recovery Modalities, and Sport Psychology

4 Chapter 3: Technology Review, Research Appraisal, and Knowledge Translation (Domain 1)

4.1 Critically Reviewing Sport Performance Products, Technologies, and Trends 4.2 Evaluating Research Studies: Design, Validity, and Practical Relevance 4.3 Communicating Research Findings to Support Evidence-Based Practice

5 Chapter 4: Sport Demands Analysis, KPIs, and Resource Strategy (Domain 2)

5.1 Researching Biomechanical, Tactical, and Technical Sport Demands 5.2 Physiological Demands, Injury Epidemiology, and Environmental Constraints 5.3 Qualitative and Mixed-Methods Research for Sport Demands Analysis 5.4 Collaborating with Stakeholders to Establish Key Performance Indicators 5.5 Identifying Resources and Strategies for Short- and Long-Term Goals

6 Chapter 5: Profiling Benchmarks and Needs Analysis Communication (Domain 2)

6.1 Developing Normative Benchmarks: Z-Scores, T-Scores, and Percentiles 6.2 Multidimensional Athlete Profiling and Return-to-Performance Milestones 6.3 Communicating Needs Analysis Results to Multidisciplinary Stakeholders

7 Chapter 6: Assessment Ethics, Safety Standards, and Tool Psychometrics (Domain 3)

7.1 Ethical, Legal, and Privacy Considerations in Athlete Data Collection 7.2 Evaluating Assessment Environments, Equipment, and Safety Standards 7.3 Validity, Reliability (ICC, CV%, TE), and Smallest Worthwhile Change

8 Chapter 7: Applied Assessment and Load-Monitoring Systems (Domain 3)

8.1 Movement Tracking Systems: GNSS/GPS, LPS, IMUs, and Optical Tracking 8.2 Neuromuscular Testing: Force Platforms, Isometric Pulls/Pushes, and VBT 8.3 Internal Load, Autonomic Readiness (HRV), Biomarkers, and PROMs

9 Chapter 8: Quality Assurance, Protocols, and Assessment Efficacy (Domain 3)

9.1 Applying Quality Assurance Techniques in Sport Science Data Collection 9.2 Developing, Documenting, and Standardizing Data Collection Protocols 9.3 Systematically Reevaluating Assessments, Protocols, and Interventions

10 Chapter 9: Data Governance, Storage Architecture, and Cleaning (Domain 4)

10.1 Data Privacy, Security Protocols, and Protecting Organizational Information 10.2 Designing Plans for Data Collection, Storage, Analysis, and AMS Workflows 10.3 Evaluating Data Quality, Signal Integrity, Outliers, and Cleaning Procedures

11 Chapter 10: Statistical Modeling, Feedback Delivery, and Analytical Limitations (Domain 4)

11.1 Inferential Statistics, Correlation, Regression, ANOVA, and Effect Sizes 11.2 Longitudinal Load Modeling: ACWR, EWMA, Monotony, Strain, and Change Detection 11.3 Data Visualization, Dashboard Design, and Timely Stakeholder Feedback 11.4 Communicating Analytical Limitations, Bias, and Uncertainty

4.2 Evaluating Research Studies: Design, Validity, and Practical Relevance

Key Takeaways Explain This Topic AI

Evidence-Based Practice (EBP) integrates three equal pillars: the best available peer-reviewed scientific evidence, practitioner applied expertise, and athlete values, preferences, and competitive context.

The hierarchy of scientific evidence progresses from mechanistic/animal studies and case reports up through cross-sectional, case-control, and prospective cohort studies to randomized controlled trials (parallel-group and counterbalanced crossover trials with washout periods) and systematic reviews with meta-analyses.

Internal validity reflects confidence that the intervention caused the observed outcome (threatened by selection bias, history, maturation, familiarization/learning effects, instrumentation drift, attrition, placebo/Hawthorne effects, and regression to the mean), whereas external and ecological validity reflect whether findings generalize to elite athletes in real competitive environments.

A p-value (< 0.05) measures only the probability of observing the sample data (or more extreme data) assuming the null hypothesis is true; it never quantifies the magnitude or practical importance of an effect, which can be distorted by sample size N.

Practical relevance is established by evaluating standardized effect sizes (Cohen's d or small-sample Hedges' g on Hopkins's scale: <0.20 trivial, 0.20–0.59 small, 0.60–1.19 moderate, 1.20–1.99 large, >=2.0 very large) and 90% or 95% Confidence Intervals relative to the Smallest Worthwhile Change (SWC).

Last updated: October 2026

Evaluating Research Studies: Design, Validity, and Practical Relevance

Quick Answer: Critically appraising sport science research requires evaluating three dimensions: Study Design Hierarchy (with Systematic Reviews/Meta-Analyses and Randomized Controlled/Crossover Trials at the apex for causal inference), Internal vs. External/Ecological Validity (controlling biases like regression to the mean and familiarization while ensuring the sample and task reflect elite sport reality), and Practical Magnitude vs. Statistical Significance (interpreting standardized effect sizes—Cohen's d d d or Hedges' g g g —and 90 % / 95 % 90\%/95\% 90%/95% Confidence Intervals relative to the Smallest Worthwhile Change SWC \text{SWC} SWC rather than relying on p < 0.05 p < 0.05 p < 0.05 alone).

High-performance practitioners constantly encounter conflicting research claims regarding training periodization, injury risk factors, and ergogenic strategies. Under Domain 1, Task 1E , a Certified Performance and Sport Scientist ( CPSS \text{CPSS} CPSS ) must critically appraise interdisciplinary studies across exercise physiology, biomechanics, motor control, psychology, and nutrition to determine both their methodological validity and their practical relevance to a specific athletic population.

  • The Evidence-Based Practice ( EBP \text{EBP} EBP ) Triad and Hierarchy of Evidence
  • The Three Pillars of Evidence-Based Practice

Adapted from Sackett's foundational medical model, Evidence-Based Practice ( EBP \text{EBP} EBP ) in sport science is not blind adherence to academic journal articles. Instead, it is the deliberate integration of three overlapping domains:

Best Available Peer-Reviewed Scientific Evidence: Mechanistic, observational, and experimental findings vetted for methodological rigor.

Practitioner Applied / Clinical Expertise: The sport scientist's and coach's accumulated diagnostic skill, pattern recognition, and understanding of tissue-loading tolerances.

Athlete Values, Preferences, and Sport Context: Cultural norms, individual belief systems, competitive schedule constraints, travel load, and equipment availability.

The Hierarchy of Scientific Evidence

When evaluating causal claims (e.g., "Does intervention X X X improve sprint acceleration or reduce hamstring strain incidence?" ), study designs sit along a formal hierarchy of evidentiary strength:

Systematic Reviews and Meta-Analyses (Apex of Evidence): Use explicit, reproducible search strategies (following PRISMA guidelines) to synthesize all available primary trials and statistically pool effect sizes. When appraising a meta-analysis, a sport scientist must inspect:

Between-Study Heterogeneity ( I 2 I^2 I 2 Statistic and Cochran's Q Q Q ): Higgins's I 2 I^2 I 2 quantifies the percentage of total variation across studies due to true clinical/methodological heterogeneity rather than random sampling error (Higgins's rough guide: I 2 ≈ 25 % I^2 \approx 25\% I 2 ≈ 25% low, ≈ 50 % \approx 50\% ≈ 50% moderate, and ≈ 75 % \approx 75\% ≈ 75% or more high heterogeneity). High I 2 I^2 I 2 indicates that studies differed substantially in training status, dosing, or measurement protocols, requiring a random-effects model and subgroup moderator analysis rather than a single pooled fixed-effect estimate.

Forest Plots: Visual display of each individual study's point estimate, sample weight (box size), 95 % 95\% 95% confidence interval (horizontal whiskers), and the pooled summary diamond.

Publication Bias (The "File-Drawer Problem"): Journals preferentially publish small studies with statistically significant ( p < 0.05 p < 0.05 p < 0.05 ) positive findings while null trials remain unpublished. Audited via Funnel Plot symmetry (plotting effect size against standard error) and Egger's regression test .

Randomized Controlled Trials ( RCTs \text{RCTs} RCTs ): The gold standard for establishing primary causality . Random allocation distributes both known and unknown confounding variables evenly across intervention and control arms.

Parallel-Group RCTs \text{RCTs} RCTs : Essential for chronic adaptations (e.g., a 10-week resistance or plyometric block) where structural remodeling cannot be quickly reversed.

Randomized Counterbalanced Crossover Trials: Each athlete completes both the experimental and placebo/control conditions in a randomized sequence separated by a washout period . Because each athlete serves as their own internal control, between-subject genetic and training-history variance is eliminated, dramatically boosting statistical power in small elite samples ( n = 10 – 16 n = 10\text{–}16 n = 10 – 16 ). However, crossover designs are valid only for acute or short-term interventions (e.g., caffeine supplementation, post-activation performance enhancement [ PAPE \text{PAPE} PAPE ], or acute cooling) where the washout period completely clears carryover effects.

Prospective Cohort Studies: Observational longitudinal studies that measure baseline exposures or workload characteristics in an uninjured cohort and follow them forward over a season to record outcome incidence (calculating Relative Risk [ RR \text{RR} RR ] or Hazard Ratios [ HR \text{HR} HR ] ). Superior to retrospective designs because exposure is measured before injury occurs.

Case-Control and Cross-Sectional Studies: Case-control studies compare athletes who already have an outcome (cases, e.g., athletes with patellar tendinopathy) against matched uninjured peers (controls) to look backward at past exposures ( Odds Ratio [ OR \text{OR} OR ] )—highly vulnerable to reverse causation and recall bias. Cross-sectional studies capture a single timepoint snapshot (e.g., correlating squat 1RM with sprint time), establishing association but never causation.

Case Series and N -of- 1 N\text{-of-}1 N -of- 1 Single-Subject Designs: Detailed longitudinal tracking of an individual elite outlier (e.g., Olympic medalist rehabilitation or N -of- 1 N\text{-of-}1 N -of- 1 ABAB withdrawal trials). High contextual richness for unique athletes, though limited generalizability.

Mechanistic Laboratory / Animal / In Vitro Studies & Expert Opinion: Cell-culture, rodent muscle biopsy, or narrative opinion pieces. Useful for generating biological hypotheses, but reside at the base of the applied evidence pyramid.

Internal Validity vs. External (Ecological) Validity

A critical reviewer must weigh the tension between tight experimental control ( internal validity ) and real-world generalizability ( external and ecological validity ).

Threats to Internal Validity

Internal validity asks: Can we be confident that the observed change in the dependent variable was caused solely by the independent variable and not by a confounding factor? Classic threats include:

Selection Bias: Non-random group assignment (e.g., comparing volunteers who chose an extra morning mobility session against non-volunteers).

History and Maturation: External events occurring concurrently during the study (e.g., changes in team tactical conditioning, coaching staff, or seasonal climate) or natural biological growth and pubertal maturation in youth athletes ( Peak Height Velocity [ PHV \text{PHV} PHV ] ).

Testing / Learning (Familiarization) Effects: Apparent improvements from pre-test to post-test caused by neural motor learning and pacing familiarity with an unfamiliar assessment protocol rather than physiological adaptation. Mitigated by conducting 2 – 3 2\text{–}3 2 – 3 familiarization trials prior to baseline until the systematic change between trials approaches zero.

Instrumentation and Evaluator Drift: Shifts in force-plate zeroing, metabolic cart gas calibration, or different practitioners performing pre- vs. post-intervention skinfolds.

Statistical Regression to the Mean: A pervasive pitfall in applied sport science. When athletes are selected into a remedial intervention subgroup solely because they scored at the extreme bottom of a single baseline test (e.g., the lowest 20% of countermovement jumpers), random day-to-day biological variation and measurement error ( TE \text{TE} TE ) guarantee that their average score on a subsequent retest will naturally gravitate closer to the squad mean—even if the intervention is completely inert. Controlling for regression to the mean requires an untreated control group or averaging multiple baseline trials.

Differential Attrition (Dropouts): When athletes drop out of a high-intensity arm due to soreness or injury. Analyzing only finishers ( Per-Protocol Analysis ) artificially inflates treatment efficacy; rigorous trials report Intention-to-Treat ( ITT \text{ITT} ITT ) Analysis , analyzing every randomized participant in their originally assigned group.

Hawthorne, Placebo, and Nocebo Effects: Athletes work harder simply because they know they are being monitored ( Hawthorne effect ), improve due to positive expectancy when given an inert sham intervention ( Placebo effect ), or experience performance decrements and heightened fatigue when warned of negative side effects ( Nocebo effect ). Double-blinding (both athlete and assessor blinded) is the gold standard defense, though blinding physical training modes (e.g., heavy squats vs. sprints) is often impossible, requiring sham-controlled active comparators.

External and Ecological Validity in Sport Science

External validity refers to whether findings generalize to other populations, while ecological validity specifically evaluates whether the study's testing environment, task constraints, and loading protocols reflect the actual competitive sport environment. For example, a randomized trial demonstrating that 6 weeks of unilateral leg extensions increases quadriceps cross-sectional area by 9 % 9\% 9% in untrained college students ( Tier 1 \text{Tier 1} Tier 1 recreationally active) has high internal validity but low external and ecological validity for an in-season professional rugby squad ( Tier 4/5 \text{Tier 4/5} Tier 4/5 elite). Elite athletes possess a compressed adaptive ceiling, high concurrent metabolic and contact loads (the concurrent training interference effect ), and multi-joint neuromuscular coordination demands that differ fundamentally from isolated single-joint exercises in novices.

  • Statistical Significance vs. Practical / Clinical Relevance
  • The Limitations of p p p -Values and Null-Hypothesis Significance Testing ( NHST \text{NHST} NHST )

A p p p -value is defined strictly as the probability of observing a sample test statistic as extreme as (or more extreme than) the one obtained, assuming the null hypothesis ( H 0 H_0 H 0 ​ : true effect = 0 = 0 = 0 ) is true . Crucially:

A p p p -value does not measure the probability that the null hypothesis is true.

A p p p -value does not measure the magnitude, clinical importance, or practical relevance of an effect.

Because p p p -values depend heavily on sample size ( N N N ) (since standard error scales with 1 N \frac{1}{\sqrt{N}} N ​ 1 ​ ), two opposite errors occur in sport science:

Large-Sample Triviality: In a league-wide tracking database ( N = 3,000 N = 3{,}000 N = 3 , 000 player-matches), a difference of 0.08 km ⋅ h − 1 0.08\text{ km}\cdot\text{h}^{-1} 0.08 km ⋅ h − 1 in peak speed or a correlation of r = 0.06 r = 0.06 r = 0.06 yields p ≈ 0.001 p \approx 0.001 p ≈ 0.001 ("statistically significant"), yet is practically trivial ( r 2 = 0.36 % r^2 = 0.36\% r 2 = 0.36% shared variance).

Small-Sample Type II Error ( β \beta β ): In an elite Olympic training group ( N = 10 N = 10 N = 10 ), an intervention that improves 20 m 20\text{ m} 20 m sprint time by 0.05 s 0.05\text{ s} 0.05 s (a massive competitive edge) may yield p = 0.08 p = 0.08 p = 0.08 ("statistically non-significant") solely because the small sample size lacks statistical power ( 1 − β 1 - \beta 1 − β , conventionally targeted at ≥ 0.80 \ge 0.80 ≥ 0.80 at α = 0.05 \alpha = 0.05 α = 0.05 ) .

Standardized Effect Sizes and Confidence Intervals Relative to the SWC

To separate sample-size artifacts from true practical magnitude, sport scientists evaluate Standardized Mean Differences — Cohen's d d d or Hedges' g g g (which multiplies Cohen's d d d by a small-sample bias correction factor J ≈ 1 − 3 4 df − 1 J \approx 1 - \frac{3}{4\text{df} - 1} J ≈ 1 − 4 df − 1 3 ​ when n < 20 n < 20 n < 20 ):

d = X ˉ intervention − X ˉ control SD pooled d = \frac{\bar{X}_{\text{intervention}} - \bar{X}_{\text{control}}}{\text{SD}_{\text{pooled}}} d = SD pooled ​ X ˉ intervention ​ − X ˉ control ​ ​

In sport science, Hopkins's Modified Magnitude Scale is used alongside traditional Cohen thresholds to classify standardized effect sizes:

  • < 0.20 < 0.20 < 0.20 : Trivial
  • 0.20 – 0.59 0.20\text{–}0.59 0.20 – 0.59 : Small
  • 0.60 – 1.19 0.60\text{–}1.19 0.60 – 1.19 : Moderate
  • 1.20 – 1.99 1.20\text{–}1.99 1.20 – 1.99 : Large
  • 2.00 – 3.99 2.00\text{–}3.99 2.00 – 3.99 : Very Large
  • ≥ 4.00 \ge 4.00 ≥ 4.00 : Extremely Large

Rather than inspecting the point estimate alone, the practitioner overlays the 90 % 90\% 90% or 95 % 95\% 95% Confidence Interval ( CI \text{CI} CI ) against the Smallest Worthwhile Change ( SWC = ± 0.20 \text{SWC} = \pm 0.20 SWC = ± 0.20 standardized units, or raw SWC \text{SWC} SWC units) .

Worked Example: Interpreting a Confidence Interval / Forest Plot Table

Suppose a sport scientist reviews three studies investigating 6-week training interventions on Countermovement Jump ( CMJ \text{CMJ} CMJ ) height in collegiate basketball players, where the raw SWC \text{SWC} SWC is ± 1.0 cm \pm 1.0\text{ cm} ± 1.0 cm (standardized SWC = ± 0.20 \text{SWC} = \pm 0.20 SWC = ± 0.20 ):

Intervention Study Sample Size ( N N N ) p p p -Value Raw Mean Change ( 95 % CI 95\%\text{ CI} 95% CI ) Hedges' g g g ( 95 % CI 95\%\text{ CI} 95% CI ) Practical Interpretation Relative to SWC \text{SWC} SWC ( ± 1.0 cm \pm 1.0\text{ cm} ± 1.0 cm / g = ± 0.20 g = \pm 0.20 g = ± 0.20 )

Protocol A: Contrast Complex Training N = 28 N = 28 N = 28 p = 0.003 p = 0.003 p = 0.003 + 3.2 cm +3.2\text{ cm} + 3.2 cm ( + 1.4 to + 5.0 cm +1.4\text{ to }+5.0\text{ cm} + 1.4 to + 5.0 cm ) g = + 0.76 g = +0.76 g = + 0.76 ( + 0.33 to + 1.19 +0.33\text{ to }+1.19 + 0.33 to + 1.19 ) Clear Beneficial (Moderate) Effect: Point estimate is moderate ( g = 0.76 g = 0.76 g = 0.76 ) and the entire 95 % CI 95\%\text{ CI} 95% CI lies above the + 1.0 cm +1.0\text{ cm} + 1.0 cm ( + 0.20 +0.20 + 0.20 ) worthwhile threshold. Adopt with high confidence.

Protocol B: Commercial Vibration Insole N = 420 N = 420 N = 420 p = 0.012 p = 0.012 p = 0.012 + 0.4 cm +0.4\text{ cm} + 0.4 cm ( + 0.1 to + 0.7 cm +0.1\text{ to }+0.7\text{ cm} + 0.1 to + 0.7 cm ) g = + 0.09 g = +0.09 g = + 0.09 ( + 0.02 to + 0.16 +0.02\text{ to }+0.16 + 0.02 to + 0.16 ) Clear Trivial Effect (Despite p < 0.05 p < 0.05 p < 0.05 ): Large N N N produced statistical significance, but the entire 95 % CI 95\%\text{ CI} 95% CI falls strictly inside the [ − 1.0 , + 1.0 cm ] [-1.0, +1.0\text{ cm}] [ − 1.0 , + 1.0 cm ] triviality zone. Reject procurement.

Protocol C: Novel Eccentric Overload N = 9 N = 9 N = 9 p = 0.140 p = 0.140 p = 0.140 + 2.1 cm +2.1\text{ cm} + 2.1 cm ( − 1.5 to + 5.7 cm -1.5\text{ to }+5.7\text{ cm} − 1.5 to + 5.7 cm ) g = + 0.50 g = +0.50 g = + 0.50 ( − 0.36 to + 1.36 -0.36\text{ to }+1.36 − 0.36 to + 1.36 ) Unclear / Underpowered Effect: Point estimate looks promising ( + 2.1 cm +2.1\text{ cm} + 2.1 cm , small-to-moderate), but the wide 95 % CI 95\%\text{ CI} 95% CI spans both substantial harm ( < − 1.0 cm < -1.0\text{ cm} < − 1.0 cm ) and large benefit ( > + 1.0 cm > +1.0\text{ cm} > + 1.0 cm ). Requires more data or low-risk pilot testing before adoption.