Study guide

1.2 Hypothesis Testing & Statistical Significance

USMLE Step 3 Exam

Track your progress as you read

Finished sections get a check mark.

1 Foundations - Biostatistics & Study Design

1.1 Study Designs & Clinical Trials 1.2 Hypothesis Testing & Statistical Significance 1.3 Measures of Association & Diagnostic Accuracy

2 Foundations - Epidemiology & Literature Interpretation

Section 2.1: Epidemiology & Risk Measures Section 2.2: Interpretation of Medical Literature & Ads Section 2.3: Preventive Medicine & Screening Guidelines

3 Foundations - Ethics & Communication

3.1 Medical Ethics & Patient Autonomy 3.2 Professionalism & Communication Skills 3.3 Legal Medicine & Informed Consent

4 Foundations - Patient Safety & Systems

Section 4.1: Patient Safety & Error Reduction Section 4.2: Quality Improvement & Systems-Based Practice Section 4.3: Healthcare Delivery Systems & Transitions of Care

5 Cardiovascular Management

5.1 Ischemic Heart Disease & Myocardial Infarction 5.2 Heart Failure & Cardiomyopathies 5.3 Arrhythmias & Conduction System Disorders 5.4 Vascular Diseases & Hypertension Management

6 Respiratory Management

Section 6.1: Obstructive Lung Diseases Section 6.2: Restrictive & Interstitial Lung Diseases Section 6.3: Infectious Pulmonary Diseases Section 6.4: Pulmonary Vascular & Pleural Diseases

7 Gastrointestinal & Hepatic Management

Esophageal & Gastric Disorders Intestinal & Colorectal Disorders Hepatic, Biliary, & Pancreatic Diseases Nutritional & Malabsorptive Syndromes

8 Renal, Urinary, & Endocrine Management

Section 8.1: Acute Kidney Injury & Chronic Kidney Disease Section 8.2: Fluid, Electrolyte, & Acid-Base Disorders Section 8.3: Diabetes Mellitus & Metabolic Syndrome Section 8.4: Thyroid, Adrenal, & Pituitary Disorders

9 Neurology, Special Senses, & Behavioral Health

9.1 Cerebrovascular Disease & Stroke 9.2 Neurodegenerative & Demyelinating Disorders 9.3 Psychiatric Emergencies & Substance Use Disorders 9.4 Common Mood, Anxiety, & Psychotic Disorders

10 Obstetrics & Gynecology

10.1 Antepartum Care & Pregnancy Complications 10.2 Intrapartum Management & Postpartum Care 10.3 Gynecology & Breast Disorders

11 Pediatrics, Human Development, & Multisystem Care

11.1 Normal Development & Childhood Immunizations 11.2 Common Pediatric & Neonatal Illnesses 11.3 Multisystem & Rheumatologic Diseases

12 Skin, Musculoskeletal, & CCS Case Management

Section 12.1: Common Dermatologic Presentations Section 12.2: Acute & Chronic Musculoskeletal Injuries Section 12.3: Computer-Based Case Simulations (CCS) Strategy

1.2 Hypothesis Testing & Statistical Significance

Key Takeaways Explain This Topic AI

The null hypothesis assumes no difference, while the alternative hypothesis assumes a difference exists; the p-value is the probability of the data given a true null hypothesis.

Type I error (alpha) represents a false positive where the null hypothesis is rejected when it is true, while Type II error (beta) is a false negative where a difference is missed.

Statistical power (1 - beta) is the probability of detecting a true difference and is determined by sample size, effect size, alpha level, and variability.

A priori power calculations dictate the minimum sample size needed to detect a clinical difference, preventing underpowered studies from committing Type II errors.

Confidence intervals reflect the precision of an estimate; a 95% confidence interval excludes 0 for difference metrics and 1 for ratio metrics when a result is statistically significant.

Last updated: July 2026

Hypothesis Testing & Statistical Significance

In clinical medicine, researchers use statistical inference to draw conclusions about a broad target population based on data collected from a smaller study sample. Understanding the mathematical and conceptual foundations of hypothesis testing, error rates, statistical power, and confidence intervals is critical for evaluating whether study findings reflect true clinical phenomena or are merely the result of random chance.

Hypotheses Formulation and the P-Value

Every statistical test begins with the formulation of two mutually exclusive and exhaustive hypotheses:

Null Hypothesis ( H 0 H_0 H 0 ​ ): Assumes there is no difference, association, or effect in the underlying population (e.g., "The mean blood pressure reduction is identical between patients receiving the new drug and those receiving placebo").

Alternative Hypothesis ( H 1 H_1 H 1 ​ ): Assumes there is a true difference, association, or effect (e.g., "The new drug reduces mean blood pressure compared to placebo").

The Significance Level ( α \alpha α ) and P-Value

Alpha ( α \alpha α ): The significance level, predetermined by the researchers before data collection, representing the threshold for rejecting the null hypothesis. It represents the maximum probability of committing a Type I error that the researchers are willing to tolerate. By convention, α \alpha α is set at 0.05 (5%).

P-Value: The probability of obtaining a test statistic as extreme as, or more extreme than, the one observed, assuming that the null hypothesis is true.

A common misconception is that the p-value is the probability that the null hypothesis is true. Rather, the p-value is the probability of the data given the null hypothesis ( P ( Data ∣ H 0 ) P(\text{Data} | H_0) P ( Data ∣ H 0 ​ ) ).

If the p-value is less than the alpha level ( p < 0.05 p < 0.05 p < 0.05 ), the null hypothesis is rejected, and the result is deemed "statistically significant."

If the p-value is greater than or equal to alpha ( p ≥ 0.05 p \ge 0.05 p ≥ 0.05 ), the study fails to reject the null hypothesis. This does not prove the null hypothesis is true; it merely states that the data do not provide sufficient evidence to reject it.

Errors in Hypothesis Testing

Because statistical decisions are based on probability, there is always a risk of reaching an incorrect conclusion. These errors are classified into two types:

Study Conclusion \ True State of Nature H 0 H_0 H 0 ​ is True (No True Difference) H 0 H_0 H 0 ​ is False (True Difference Exists)

Reject H 0 H_0 H 0 ​ (Find a difference) Type I Error ( α \alpha α ) (False Positive) Correct Decision ( 1 − β 1 - \beta 1 − β ) (Power)

Fail to Reject H 0 H_0 H 0 ​ (Find no difference) Correct Decision ( 1 − α 1 - \alpha 1 − α ) Type II Error ( β \beta β ) (False Negative)

Type I Error ( α \alpha α )

A Type I error occurs when researchers reject the null hypothesis when it is actually true (a false positive). In clinical terms, this means concluding that a therapy is effective when it is not.

Clinical Impact: A Type I error can lead to the widespread adoption of ineffective—and potentially harmful—treatments, wasting resources and exposing patients to unnecessary side effects.

Control: The probability of a Type I error is directly controlled by the choice of α \alpha α . If a study uses an α \alpha α of 0.01 instead of 0.05, the threshold for significance is higher, and the probability of a false positive drops to 1%.

Type II Error ( β \beta β )

A Type II error occurs when researchers fail to reject the null hypothesis when it is actually false (a false negative). This means concluding that there is no difference between treatments when a true clinical difference exists.

Clinical Impact: A Type II error can cause researchers to discard a potentially life-saving drug or intervention because the study failed to prove its efficacy.

Control: The probability of a Type II error is denoted by β \beta β . It is inversely related to α \alpha α ; keeping all other variables constant, lowering α \alpha α (making it harder to reject the null) will increase β \beta β .

Statistical Power and Power Calculations

Statistical power ( 1 − β 1 - \beta 1 − β ) is the probability of correctly rejecting the null hypothesis when a true difference exists (i.e., detecting an effect that is genuinely present). Studies are typically designed to have a power of 80% ( β = 0.20 \beta = 0.20 β = 0.20 ) or 90% ( β = 0.10 \beta = 0.10 β = 0.10 ).

Determinants of Power

Power is determined by four interacting parameters:

Sample Size ( N N N ): The number of subjects in the study. Increasing sample size reduces standard error, which narrows the distribution curves and increases power.

Effect Size: The magnitude of the difference between the groups (e.g., mean difference, relative risk). A large effect is easier to detect than a small one, requiring fewer patients to achieve the same power.

Significance Level ( α \alpha α ): A higher α \alpha α (e.g., 0.05 vs. 0.01) increases power because the critical value threshold is easier to cross, though it increases the risk of a Type I error.

Standard Deviation (Variability): Greater variability in the population measurements widens the distribution curves, increases overlap between groups, and decreases power.

A Priori Power Calculations

Before enrolling the first patient, researchers must perform an a priori power calculation. This calculation determines the minimum sample size required to detect a pre-specified, clinically meaningful effect size at a given α \alpha α and power. If a study reports "no significant difference" but has a small sample size, it is likely underpowered , meaning it had a high risk of a Type II error and was simply blind to a real difference.

Confidence Intervals (CIs)

A Confidence Interval (CI) provides a range of values within which the true population parameter (e.g., mean difference, odds ratio, relative risk) is expected to fall with a specified degree of certainty (usually 95%).

Calculation and Precision

The general formula for a confidence interval is: Confidence Interval = Point Estimate ± ( Z × Standard Error ) \text{Confidence Interval} = \text{Point Estimate} \pm (Z \times \text{Standard Error}) Confidence Interval = Point Estimate ± ( Z × Standard Error )

Where Z Z Z is a critical value determined by the confidence level (for a 95% CI, Z ≈ 1.96 Z \approx 1.96 Z ≈ 1.96 ), and the Standard Error ( SE = SD / N \text{SE} = \text{SD}/\sqrt{N} SE = SD / N ​ ) represents the variability of the sample estimate.

Narrow CI: Indicates high precision, which is achieved with large sample sizes ( N N N ) and low standard deviation.

Wide CI: Indicates low precision (high uncertainty), typical of small sample sizes.

Confidence Level: A 99% confidence interval is wider than a 95% confidence interval because achieving higher certainty requires a wider range of values.

Interpreting Statistical Significance from CIs

Confidence intervals provide information about both statistical significance and the clinical relevance of the effect size:

For Difference Metrics (e.g., Mean Difference, Risk Difference): The null hypothesis states that the difference between groups is 0. If the 95% CI includes the value 0 (e.g., 95% CI: -0.5 to +1.2), the results are not statistically significant ( p ≥ 0.05 p \ge 0.05 p ≥ 0.05 ). If the CI excludes 0, the result is statistically significant ( p < 0.05 p < 0.05 p < 0.05 ).

For Ratio Metrics (e.g., Relative Risk, Odds Ratio, Hazard Ratio): The null hypothesis states that the ratio of risks is 1. If the 95% CI includes the value 1 (e.g., 95% CI: 0.90 to 1.35), the results are not statistically significant ( p ≥ 0.05 p \ge 0.05 p ≥ 0.05 ). If the CI excludes 1, the result is statistically significant.

Clinical vs. Statistical Significance

It is vital to distinguish statistical significance from clinical significance. A statistically significant result ( p < 0.05 p < 0.05 p < 0.05 ) does not guarantee that the finding is clinically meaningful. For example, in an extremely large trial of 100,000 patients, a new antihypertensive drug may lower systolic blood pressure by an average of 0.2 mmHg compared to placebo, achieving a p-value of 0.001. While statistically significant, a 0.2 mmHg reduction has no clinical relevance to patient health. Clinicians must always evaluate the absolute effect size and the clinical endpoints alongside the p-value.