Study guide

16.1 Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs

GRE Subject Test: Psychology Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Neuroanatomy, Neural Transmission, and Neuroscience Methods

1.1 Cellular Neuroscience: Neurons, Action Potentials, and Synaptic Transmission 1.2 Functional Neuroanatomy: Brain Structures, Hemispheric Specialization, and Cortical Lobes 1.3 Neurotransmitters, Neuromodulators, and Psychopharmacology 1.4 The Endocrine System, Autonomic Nervous System, and Behavioral Genetics 1.5 Cognitive Neuroscience Methods, Lateralization Testing, and Neuroplasticity

2 Chapter 2: Sensation and Perception

2.1 Psychophysics, Sensory Thresholds, and Signal Detection Theory 2.2 Visual Processing: Photoreceptors, Pathways, Color Vision, and Feature Detection 2.3 Auditory Systems, Pitch Perception, Gustation, and Olfaction 2.4 Somatosensation, Vestibular Senses, Perceptual Organization, and Theories of Perception

3 Chapter 3: Biological Drives, Consciousness, and Ethology

3.1 Sleep Architecture, Circadian Rhythms, and States of Consciousness 3.2 Biological Mechanisms of Hunger, Thirst, Sexual Behavior, and Emotion 3.3 Ethology and Comparative Psychology: Instincts, Imprinting, and Evolutionary Adaptation

4 Chapter 4: Principles of Conditioning and Learning

4.1 Classical Conditioning: Paradigms, Extinction, Spontaneous Recovery, and Biological Constraints 4.2 Operant Conditioning: Reinforcement Schedules, Punishment, and Behavior Modification 4.3 Observational Learning, Social Cognitive Theory, and Latent Learning

5 Chapter 5: Memory Architecture and Processes

5.1 Memory Architecture: Sensory Buffers, Working Memory, and Multi-Store Models 5.2 Long-Term Memory Systems: Episodic, Semantic, Procedural, and Implicit Memory 5.3 Retrieval Processes, Forgetting Curves, Interference Theory, and Memory Reconstruction

6 Chapter 6: Language, Thought, Decision Making, and Intelligence

6.1 Language Structure, Psycholinguistics, and Language Acquisition Theories 6.2 Concept Formation, Mental Representation, and Problem-Solving Strategies 6.3 Judgment, Heuristics, Cognitive Biases, and Decision-Making Models 6.4 Selective Attention, Divided Attention, and Executive Cognitive Control 6.5 Reading Processes, Bilingualism, and Verbal and Nonverbal Communication 6.6 Intelligence: Theories, Measurement, Heritability, and Metacognition

7 Chapter 7: Social Perception, Attribution, Attitudes, and the Self

7.1 Attribution Theories, Heider, Kelley, and Systematic Attribution Errors 7.2 Attitude Formation, Consistency Theories, and Cognitive Dissonance 7.3 Persuasion Models: Elaboration Likelihood Model and Persuasion Techniques 7.4 The Self, Social Comparison, Self-Esteem, and the Need to Belong

8 Chapter 8: Social Influence, Groups, Relationships, Culture, and Gender

8.1 Conformity, Compliance, and Obedience: Classic Studies and Mechanisms 8.2 Group Dynamics: Social Facilitation, Loafing, Polarization, and Groupthink 8.3 Stereotyping, Prejudice, Discrimination, and Intergroup Conflict 8.4 Prosocial Behavior, Bystander Intervention, Aggression, Interpersonal Attraction, and Close Relationships 8.5 Emotion, Affect, and Motivation in Social Life; Cultural and Gender Influences

9 Chapter 9: Physical, Cognitive, and Moral Development

9.1 Prenatal Development, Teratogens, Neonatal Reflexes, and Motor Milestones 9.2 Piaget's Cognitive Developmental Stages, Neo-Piagetian Theories, and Post-Piagetian Critiques 9.3 Vygotsky's Sociocultural Theory, Scaffolding, and the Zone of Proximal Development 9.4 Moral Development: Kohlberg's Stage Model and Gilligan's Care Perspective

10 Chapter 10: Social, Emotional, Gender, and Lifespan Development

10.1 Attachment Theory, Bowlby, Ainsworth's Strange Situation, and Temperament 10.2 Erikson's Psychosocial Stages and Baumrind's Parenting Typologies 10.3 Adolescent Identity Exploration, Adult Transitions, and Cognitive Changes in Aging 10.4 Emotional Development, Gender Development, Peer Relations, and Ecological Systems

11 Chapter 11: Theories of Personality

11.1 Psychodynamic and Neo-Freudian Personality Theories 11.2 Humanistic, Existential, and Phenomenological Approaches to Personality 11.3 Trait Theories: The Big Five (Five-Factor Model), Allport, Cattell, and Eysenck

12 Chapter 12: Psychological Disorders and Psychopathology

12.1 Anxiety Disorders, OCD, and Trauma- and Stressor-Related Disorders 12.2 Depressive Disorders, Bipolar and Related Disorders, and Suicide Risk 12.3 Schizophrenia Spectrum, Psychotic Disorders, and Personality Disorders 12.4 Neurodevelopmental, Eating, Substance-Related, Dissociative, Somatic Symptom, and Neurocognitive Disorders 12.5 Causes and Development of Disorders, Epidemiology, and Cultural and Gender Issues in Psychopathology

13 Chapter 13: Clinical Assessment, Treatment, Health Psychology, and Prevention

13.1 Clinical Assessment: Objective Inventories, Projective Tests, and Clinical Interviews 13.2 Psychotherapeutic Modalities: CBT, Behavioral, Psychodynamic, and Humanistic Therapies 13.3 Biological and Somatic Treatments: Psychotropic Medications and Neuromodulation 13.4 Stress, Conflict, Coping, Health Psychology, and Prevention

14 Chapter 14: Research Design and Experimental Methodology

14.1 The Scientific Method, Variables, and Operational Definitions 14.2 Between-Subjects, Within-Subjects, Quasi-Experimental, and Correlational Designs 14.3 Threats to Internal and External Validity, Demand Characteristics, and Experimenter Bias

15 Chapter 15: Statistical Analysis in Psychological Science

15.1 Descriptive Statistics, Measures of Central Tendency, Variability, and Normal Distributions 15.2 Inferential Statistics: Hypothesis Testing, Alpha Levels, Power, and Effect Sizes 15.3 Bivariate and Multivariate Techniques: t-Tests, ANOVA Models, Correlation, and Regression

16 Chapter 16: Psychometrics, Ethics, History, and Applied Psychology

16.1 Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs 16.2 Research Ethics, APA Ethical Principles, Institutional Review Boards, and Animal Welfare 16.3 Historical Foundations of Psychology, Major Schools of Thought, and Professional Milestones 16.4 Industrial-Organizational and Educational Psychology

16.1 Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs

Key Takeaways Explain This Topic AI

Classical Test Theory (CTT) decomposes observed test scores into true score and random error components (X = T + E), partitioning total variance into true and error variance (σ²_X = σ²_T + σ²_E) to define reliability as r_xx' = σ²_T / σ²_X.

The Standard Error of Measurement (SEM = s_x √(1 - r_xx')) quantifies observed score fluctuations around an examinee's true score, establishing confidence intervals where higher reliability directly shrinks measurement error.

Reliability evaluates distinct dimensions of consistency: temporal stability (test-retest), form equivalence (alternate-form), internal consistency (split-half with Spearman-Brown prophecy, Cronbach's α, KR-20), and rater agreement (Cohen's κ, Intraclass Correlation).

Validity establishes whether an instrument measures its target theoretical construct, encompassing content representation, criterion prediction (concurrent vs. predictive, corrected for attenuation by r_T_X T_Y = r_xy / √(r_xx' · r_yy')), and construct networks examined via Campbell & Fiske's Multitrait-Multimethod (MTMM) matrix.

Item analysis evaluates item difficulty (p-value, optimal ~0.50) and discrimination (D = P_U - P_L), while Item Response Theory (IRT) models probabilistic performance along latent trait θ using 1PL (Rasch difficulty), 2PL (discrimination + difficulty), and 3PL (pseudo-guessing) Item Characteristic Curves.

Last updated: October 2026

Psychometric Theory: Classical Test Theory, Reliability Types, and Validity Constructs

Psychometrics is the scientific discipline concerned with the theory and techniques of psychological measurement. Because psychological attributes—such as intelligence, extraversion, neuroticism, or perceptual speed—cannot be directly observed or physicalized like mass or length, psychometricians construct formal mathematical models to evaluate whether assessment instruments measure target constructs reliably, validly, and with minimal measurement error.

1. Classical Test Theory (CTT) and the True Score Model

Classical Test Theory (CTT) , often termed the true score model , was formalized by Charles Spearman and later refined by Harold Gulliksen and Frederic Lord. CTT posits that every observed test score ( X X X ) is an additive composite of two independent, unobservable quantities: an individual's latent true score ( T T T ) and unsystematic random measurement error ( E E E ) .

  • X = T + E X = T + E X = T + E
  • Theoretical Assumptions of CTT

Classical Test Theory rests upon several foundational axiomatic assumptions regarding random error:

Zero Expected Error: Across an infinite number of repeated administrations of parallel tests, random errors cancel each other out. The expected value (mean) of error is zero:

E ( E ) = 0 E(E) = 0 E ( E ) = 0

Uncorrelated True Scores and Errors: An examinee's true score bears no relationship to the magnitude or sign of the error affecting their score. High-ability and low-ability individuals are equally subject to positive or negative random error:

r T E = 0 r_{TE} = 0 r TE ​ = 0

Uncorrelated Errors Across Administrations: The measurement error affecting a person on one test form ( E 1 E_1 E 1 ​ ) is completely uncorrelated with the measurement error affecting them on a second test form ( E 2 E_2 E 2 ​ ):

r E 1 E 2 = 0 r_{E_1 E_2} = 0 r E 1 ​ E 2 ​ ​ = 0

Definition of the True Score: A person's true score is formally defined as the expected value (theoretical mean) of observed scores obtained if the individual were tested an infinite number of times under identical, independent conditions:

  • T = E ( X ) T = E(X) T = E ( X )
  • Variance Partitioning and the Definition of Reliability

Because true scores and random errors are assumed to be uncorrelated ( r T E = 0 r_{TE} = 0 r TE ​ = 0 ), the total observed score variance ( σ X 2 \sigma_X^2 σ X 2 ​ ) across a population of examinees partitions cleanly into the sum of true score variance ( σ T 2 \sigma_T^2 σ T 2 ​ ) and error variance ( σ E 2 \sigma_E^2 σ E 2 ​ ) :

σ X 2 = σ T 2 + σ E 2 \sigma_X^2 = \sigma_T^2 + \sigma_E^2 σ X 2 ​ = σ T 2 ​ + σ E 2 ​

From this variance decomposition, the reliability coefficient ( r x x ′ r_{xx'} r x x ′ ​ ) is mathematically defined as the proportion of observed score variance that is attributable to genuine true score variance:

r x x ′ = σ T 2 σ X 2 = σ X 2 − σ E 2 σ X 2 = 1 − σ E 2 σ X 2 r_{xx'} = \frac{\sigma_T^2}{\sigma_X^2} = \frac{\sigma_X^2 - \sigma_E^2}{\sigma_X^2} = 1 - \frac{\sigma_E^2}{\sigma_X^2} r x x ′ ​ = σ X 2 ​ σ T 2 ​ ​ = σ X 2 ​ σ X 2 ​ − σ E 2 ​ ​ = 1 − σ X 2 ​ σ E 2 ​ ​

When a test has zero measurement error ( σ E 2 = 0 \sigma_E^2 = 0 σ E 2 ​ = 0 ), reliability reaches unity ( r x x ′ = 1.0 r_{xx'} = 1.0 r x x ′ ​ = 1.0 ), meaning all observed variance reflects true differences among individuals.

When all observed differences reflect pure random noise ( σ T 2 = 0 \sigma_T^2 = 0 σ T 2 ​ = 0 ), reliability drops to zero ( r x x ′ = 0.0 r_{xx'} = 0.0 r x x ′ ​ = 0.0 ).

2. The Standard Error of Measurement (SEM)

While the reliability coefficient ( r x x ′ r_{xx'} r x x ′ ​ ) provides a single population-level metric of test consistency, clinicians and educational diagnosticians need to know how much uncertainty surrounds an individual examinee's observed score . This uncertainty is quantified by the Standard Error of Measurement ( S E M SEM SEM ) .

Mathematical Formulation

The S E M SEM SEM represents the standard deviation of observed scores an individual would obtain if tested repeatedly across an infinite number of parallel testing sessions. It is derived directly from the sample standard deviation ( s x s_x s x ​ ) and the reliability coefficient ( r x x ′ r_{xx'} r x x ′ ​ ):

S E M = s x 1 − r x x ′ SEM = s_x \sqrt{1 - r_{xx'}} SEM = s x ​ 1 − r x x ′ ​ ​ Relationship Between Reliability (r_xx') and SEM (Assuming sx = 15):

  • Reliability (r_xx') √(1 - r_xx') SEM = 15 · √(1 - r_xx')
  • ────────────────────────────────────────────────────────────
  • 1.00 0.000 0.00 (Zero measurement error)
  • 0.96 0.200 3.00 (Typical of IQ composite scores)
  • 0.91 0.300 4.50
  • 0.84 0.400 6.00
  • 0.75 0.500 7.50
  • 0.00 1.000 15.00 (Pure error; SEM equals test SD)
  • Constructing Confidence Intervals Around Observed Scores

Because CTT assumes measurement errors are normally distributed around the true score with a mean of zero and a standard deviation equal to the S E M SEM SEM , diagnosticians construct confidence intervals (CIs) to express the margin of measurement precision around an observed score ( X X X ):

Confidence Interval = X ± ( z critical × S E M ) \text{Confidence Interval} = X \pm (z_{\text{critical}} \times SEM) Confidence Interval = X ± ( z critical ​ × SEM )

68 % 68\% 68% Confidence Interval ( z = ± 1.00 z = \pm 1.00 z = ± 1.00 ): X ± 1.00 ( S E M ) X \pm 1.00(SEM) X ± 1.00 ( SEM )

95 % 95\% 95% Confidence Interval ( z = ± 1.96 z = \pm 1.96 z = ± 1.96 ): X ± 1.96 ( S E M ) X \pm 1.96(SEM) X ± 1.96 ( SEM )

99 % 99\% 99% Confidence Interval ( z = ± 2.58 z = \pm 2.58 z = ± 2.58 ): X ± 2.58 ( S E M ) X \pm 2.58(SEM) X ± 2.58 ( SEM )

Note

Consider an examinee who obtains a Full Scale IQ score of X = 110 X = 110 X = 110 on a test with standard deviation s x = 15 s_x = 15 s x ​ = 15 and reliability r x x ′ = .96 r_{xx'} = .96 r x x ′ ​ = .96 . The S E M SEM SEM is 15 1 − .96 = 15 .04 = 15 ( 0.20 ) = 3.0 15\sqrt{1 - .96} = 15\sqrt{.04} = 15(0.20) = 3.0 15 1 − .96 ​ = 15 .04 ​ = 15 ( 0.20 ) = 3.0 . A 95 % 95\% 95% confidence interval around the examinee's score is 110 ± 1.96 ( 3.0 ) = 110 ± 5.88 110 \pm 1.96(3.0) = 110 \pm 5.88 110 ± 1.96 ( 3.0 ) = 110 ± 5.88 , or approximately [ 104 , 116 ] [104, 116] [ 104 , 116 ] . This informs the diagnostician that the examinee's true intellectual capacity likely falls between 104 and 116.

3. Typology of Reliability Estimates

Reliability is not an inherent, immutable property of a test; rather, it is a property of test scores obtained from a specific sample under specific administration conditions. Different reliability methodologies isolate different sources of unwanted error variance .

Reliability Type Method of Administration Primary Source of Error Variance Isolated Statistical Index Used Primary Clinical / Research Limitation

Test-Retest Same test administered to the same examinees at two distinct time points. Time-sampling error (temporal fluctuations, mood, fatigue, illness). Pearson r r r (termed the coefficient of stability ). Vulnerable to carryover effects , practice effects, memory retrieval, and genuine developmental maturation across intervals.

Alternate-Form (Parallel-Form) Two equivalent forms of a test containing matched content, means, and variances administered to the same group. Content-sampling error (if given immediately); Content- and time-sampling error (if delayed). Pearson r r r (termed the coefficient of equivalence ). Highly expensive and laborious to construct two genuinely parallel forms; examinees may find one form systematically more difficult.

Split-Half Single test administration divided into two equivalent halves (e.g., odd-numbered vs. even-numbered items). Content-sampling error across test items. Pearson r r r between halves, corrected via the Spearman-Brown Prophecy formula . Test length is effectively halved during correlation, requiring statistical correction; splitting arbitrary items can yield conflicting coefficients.

Internal Consistency (Cronbach's α \alpha α ) Single test administration; evaluates covariance among all individual items simultaneously. Content-sampling error and item heterogeneity. Cronbach's Coefficient Alpha ( α \alpha α ) . Assumes tau-equivalence (equal item factor loadings); inflated by scale length; underestimates reliability if items are multidimensional.

Internal Consistency (KR-20) Single test administration for tests with dichotomously scored items (right/wrong, 0 / 1 0/1 0/1 ). Content-sampling error across binary items. Kuder-Richardson Formula 20 (KR-20) . Restricted exclusively to dichotomous items; cannot be used with polytomous Likert-type rating scales.

Inter-Rater (Inter-Scorer) Two or more independent raters score the exact same behavioral performance or clinical protocol. Rater-sampling error (observer bias, idiosyncratic rating severity, halo effects). Cohen's Kappa ( κ \kappa κ ) for categorical ratings; Intraclass Correlation (ICC) for continuous scores. Requires intensive rater training and standardized behavioral coding rubrics.

The Spearman-Brown Prophecy Formula

Because reliability is directly proportional to test length—longer tests sample a broader domain and allow random errors to cancel out—splitting a test into halves to calculate internal consistency artificially depresses the reliability coefficient. In 1910, Charles Spearman and William Brown independently derived the Spearman-Brown Prophecy formula to estimate the reliability of a test if its length is altered by a factor of k k k :

r S B = k ⋅ r x y 1 + ( k − 1 ) r x y r_{SB} = \frac{k \cdot r_{xy}}{1 + (k - 1)r_{xy}} r SB ​ = 1 + ( k − 1 ) r x y ​ k ⋅ r x y ​ ​

Where k = Number of items in the new test Number of items in the original test \text{Where } k = \frac{\text{Number of items in the new test}}{\text{Number of items in the original test}} Where k = Number of items in the original test Number of items in the new test ​

Correcting Split-Half Reliability: When estimating the reliability of the full test from two half-tests, the test is doubled in length, so k = 2 k = 2 k = 2 . The formula simplifies to:

r full = 2 r 1 / 2 1 + r 1 / 2 r_{\text{full}} = \frac{2 r_{1/2}}{1 + r_{1/2}} r full ​ = 1 + r 1/2 ​ 2 r 1/2 ​ ​

If the correlation between the odd and even halves of a test is r 1 / 2 = .70 r_{1/2} = .70 r 1/2 ​ = .70 , the estimated reliability of the entire test is:

r full = 2 ( .70 ) 1 + .70 = 1.40 1.70 ≈ .824 r_{\text{full}} = \frac{2(.70)}{1 + .70} = \frac{1.40}{1.70} \approx .824 r full ​ = 1 + .70 2 ( .70 ) ​ = 1.70 1.40 ​ ≈ .824

Cronbach's Alpha ( α \alpha α ) and Kuder-Richardson Formula 20 (KR-20)

Lee Cronbach generalized split-half reliability in 1951 by demonstrating that Cronbach's Alpha ( α \alpha α ) equals the mathematical mean of all possible split-half reliability coefficients for a test. Its computational formula is:

α = k k − 1 ( 1 − ∑ i = 1 k s i 2 s X 2 ) \alpha = \frac{k}{k - 1} \left(1 - \frac{\sum_{i=1}^k s_i^2}{s_X^2}\right) α = k − 1 k ​ ( 1 − s X 2 ​ ∑ i = 1 k ​ s i 2 ​ ​ )

Where k k k is the total number of items, s i 2 s_i^2 s i 2 ​ is the variance of item i i i , and s X 2 s_X^2 s X 2 ​ is the total observed test score variance.

When items are scored dichotomously (coded 1 1 1 for correct and 0 0 0 for incorrect), item variance simplifies to p i q i p_i q_i p i ​ q i ​ (where p i p_i p i ​ is the proportion answering correctly and q i = 1 − p i q_i = 1 - p_i q i ​ = 1 − p i ​ ). Substituting this into Cronbach's formula yields Kuder-Richardson Formula 20 (KR-20) :

K R - 20 = k k − 1 ( 1 − ∑ i = 1 k p i q i s X 2 ) KR\text{-}20 = \frac{k}{k - 1} \left(1 - \frac{\sum_{i=1}^k p_i q_i}{s_X^2}\right) K R - 20 = k − 1 k ​ ( 1 − s X 2 ​ ∑ i = 1 k ​ p i ​ q i ​ ​ )

Inter-Rater Reliability: Cohen's Kappa ( κ \kappa κ )

When two independent clinical raters categorize psychiatric patients into nominal diagnostic categories (e.g., Schizophrenia, Bipolar I, Major Depressive Disorder), calculating simple percentage agreement is misleading because raters will agree on a substantial percentage of cases purely by chance. Jacob Cohen developed Cohen's Kappa ( κ \kappa κ ) to correct for chance agreement:

κ = P o − P e 1 − P e \kappa = \frac{P_o - P_e}{1 - P_e} κ = 1 − P e ​ P o ​ − P e ​ ​

Where P o P_o P o ​ is the observed proportion of agreement, and P e P_e P e ​ is the expected proportion of agreement predicted by marginal probabilities under chance alone. A kappa of κ = 1.0 \kappa = 1.0 κ = 1.0 reflects perfect agreement; κ = 0 \kappa = 0 κ = 0 indicates agreement no better than chance; and κ < 0 \kappa < 0 κ < 0 reflects agreement worse than chance.

4. Validity Constructs: Content, Criterion-Related, and Construct Validity

While reliability establishes that a test measures something consistently, validity evaluates whether a test measures the specific construct it was designed to assess, and whether clinical or educational inferences drawn from test scores are scientifically justifiable. In classic psychometrics, validity is organized around the "trinitarian" model: content validity , criterion-related validity , and construct validity .

Content Validity

Content validity reflects the degree to which items on a test adequately and representatively sample the entire domain of knowledge, skills, or behaviors the test is intended to measure.

Evaluation Method: Content validity cannot be summarized by a single correlation coefficient; it is established qualitatively during test construction by assembling panels of Subject Matter Experts (SMEs) who review test specifications, item blueprints, and alignment matrices.

Lawshe's Content Validity Ratio (CVR): SMEs rate whether each item is "essential," "useful but not essential," or "not necessary." Items rated essential by a statistically significant majority are retained.

Face Validity vs. Content Validity: Face validity is not a technical psychometric validity construct. It refers merely to whether a test looks valid to the untrained examinee on superficial inspection. A test can possess high scientific content validity while lacking face validity (e.g., subtle projective stimuli or empirically keyed personality items).

Criterion-Related Validity

Criterion-related validity evaluates how well scores on a predictor test ( X X X ) correlate with an independent, external outcome measure or standard ( Y Y Y ), termed the criterion .

Concurrent Validity: The predictor test and criterion are measured at approximately the same point in time . Example: Administering a newly designed 15-minute depression screening scale and correlating its scores with a concurrent 3-hour structured diagnostic clinical interview conducted that same afternoon.

Predictive Validity: The predictor test is administered first, and the criterion outcome is measured at a future point in time . Example: Administering the SAT or GRE to predict cumulative undergraduate or graduate GPA four years later.

Standard Error of Estimate ( S E est SE_{\text{est}} S E est ​ )

In criterion-related validity, the accuracy of predicting a criterion score ( Y Y Y ) from an observed test score ( X X X ) is quantified by the Standard Error of Estimate ( S E est SE_{\text{est}} S E est ​ ) :

S E est = s y 1 − r x y 2 SE_{\text{est}} = s_y \sqrt{1 - r_{xy}^2} S E est ​ = s y ​ 1 − r x y 2 ​ ​

Where s y s_y s y ​ is the standard deviation of the criterion and r x y r_{xy} r x y ​ is the criterion validity coefficient. Note the structural distinction: the S E M SEM SEM uses test reliability ( s x 1 − r x x ′ s_x \sqrt{1 - r_{xx'}} s x ​ 1 − r x x ′ ​ ​ ) to estimate true score error, whereas the S E est SE_{\text{est}} S E est ​ uses criterion validity ( s y 1 − r x y 2 s_y \sqrt{1 - r_{xy}^2} s y ​ 1 − r x y 2 ​ ​ ) to estimate prediction error in the criterion.

The Correction for Attenuation

In the real world, both the predictor test ( X X X ) and the criterion measure ( Y Y Y ) contain unsystematic measurement error. Because random error attenuates (dampens) linear relationships, the observed correlation between two imperfect tests ( r x y r_{xy} r x y ​ ) is systematically lower than the true theoretical correlation between the underlying constructs ( r T X T Y r_{T_X T_Y} r T X ​ T Y ​ ​ ).

Charles Spearman derived the correction for attenuation formula to estimate what the correlation between two variables would be if both were measured with perfect reliability ( r x x ′ = 1.0 r_{xx'} = 1.0 r x x ′ ​ = 1.0 and r y y ′ = 1.0 r_{yy'} = 1.0 r y y ′ ​ = 1.0 ):

r T X T Y = r x y r x x ′ ⋅ r y y ′ r_{T_X T_Y} = \frac{r_{xy}}{\sqrt{r_{xx'} \cdot r_{yy'}}} r T X ​ T Y ​ ​ = r x x ′ ​ ⋅ r y y ′ ​ ​ r x y ​ ​

Upper Bound of Validity: A vital corollary tested frequently on the GRE is that a test's criterion validity cannot exceed the square root of its reliability when the criterion is perfectly reliable ( r y y ′ = 1.0 r_{yy'} = 1.0 r y y ′ ​ = 1.0 ):

r x y ( max ⁡ ) = r x x ′ r_{xy(\max)} = \sqrt{r_{xx'}} r x y ( m a x ) ​ = r x x ′ ​ ​

If a test has a reliability of r x x ′ = .64 r_{xx'} = .64 r x x ′ ​ = .64 , its maximum possible theoretical correlation with any external criterion is .64 = .80 \sqrt{.64} = .80 .64 ​ = .80 .

5. Construct Validity and the Multitrait-Multimethod (MTMM) Matrix

Construct validity is the overarching psychometric framework assessing whether an assessment instrument truly measures a theoretical psychological construct (e.g., anxiety, intelligence, working memory). It subsumes all other forms of validity evidence and is demonstrated through an ongoing process of hypothesis testing within a theoretical nomological network .

Convergent vs. Discriminant Validity

Convergent Validity: Demonstrates that the test correlates strongly with alternative measures designed to assess the same or theoretically similar constructs . (e.g., a new trait anxiety scale correlates r = .82 r = .82 r = .82 with the State-Trait Anxiety Inventory).

Discriminant (Divergent) Validity: Demonstrates that the test has low, near-zero, or negligible correlations with measures designed to assess theoretically distinct constructs . (e.g., a new trait anxiety scale correlates r = .12 r = .12 r = .12 with a measure of mechanical reasoning ability).

Campbell & Fiske's Multitrait-Multimethod (MTMM) Matrix

In 1959, Donald T. Campbell and Donald W. Fiske revolutionized construct validation by introducing the Multitrait-Multimethod (MTMM) Matrix . They argued that every psychological measurement reflects two distinct sources of variance: the trait of interest and the method used to measure it (e.g., self-report inventory, peer rating, projective test, behavioral observation). To validate a construct, one must simultaneously measure at least two distinct traits using at least two distinct measurement methods. Hypothetical MTMM Matrix (2 Traits: Trait A & Trait B; 2 Methods: M1 & M2)

  • Method 1 (Self-Report) Method 2 (Peer Rating)
  • Trait A1 Trait B1 Trait A2 Trait B2
  • ───────────────────────────────────────────────────────────────────────────
  • Method 1 Trait A1 ( .88 )
  • Trait B1 .32 ( .85 )
  • Method 2 Trait A2 [ .68 ] .25 ( .84 )
  • Trait B2 .22 [ .64 ] .30 ( .82 )
  • ───────────────────────────────────────────────────────────────────────────

Key to MTMM Coefficients:

  • ( .88 ) = Monotrait-Monomethod (Reliability Diagonal)
  • [ .68 ] = Monotrait-Heteromethod (Convergent Validity Diagonal)
  • .32 = Heterotrait-Monomethod (Method Variance / Common-Method Bias)
  • .22 = Heterotrait-Heteromethod (Discriminant Validity / Construct Uniqueness)
  • Decoding the Four MTMM Correlations

Monotrait-Monomethod: Same trait, same method. Located on the main diagonal. These are reliability coefficients (e.g., ( .88 ) (.88) ( .88 ) and ( .85 ) (.85) ( .85 ) ). They should represent the highest values in the entire matrix.

Monotrait-Heteromethod: Same trait, different methods. These are convergent validity coefficients (e.g., [ .68 ] [.68] [ .68 ] and [ .64 ] [.64] [ .64 ] ). They reflect whether Trait A measured by self-report correlates strongly with Trait A measured by peer rating. They must be statistically significant and substantially greater than zero.

Heterotrait-Monomethod: Different traits, same method (e.g., .32 .32 .32 and .30 .30 .30 ). These correlations reflect common-method bias (method variance). If these correlations are high, it indicates that shared measurement format (e.g., self-report response sets) artificially inflates relationships between unrelated constructs.

Heterotrait-Heteromethod: Different traits, different methods (e.g., .22 .22 .22 and .25 .25 .25 ). These represent pure construct uniqueness and provide evidence for discriminant validity . They should be the lowest coefficients in the matrix.

Important

MTMM Construct Validity Requirements: For construct validity to be confirmed:

Convergent validity coefficients (Monotrait-Heteromethod) must be substantially higher than discriminant validity coefficients (Heterotrait-Heteromethod).

Convergent validity coefficients must be higher than method bias coefficients (Heterotrait-Monomethod).

The pattern of trait interrelationships should remain consistent across methods.

6. Classical Item Analysis vs. Item Response Theory (IRT)

Assessment quality ultimately depends on the psychometric performance of individual test items. Psychometricians analyze items using either Classical Test Theory or Item Response Theory.

Classical Item Analysis

Classical item analysis evaluates two primary parameters: item difficulty and item discrimination .

1. Item Difficulty ( p p p -Value)

In classical testing, item difficulty is indexed by the proportion of examinees who answer the item correctly:

p = N correct N total p = \frac{N_{\text{correct}}}{N_{\text{total}}} p = N total ​ N correct ​ ​

Counterintuitive Direction: A higher p p p -value indicates an easier item (e.g., p = .90 p = .90 p = .90 means 90 % 90\% 90% got it right), whereas a lower p p p -value denotes a harder item (e.g., p = .15 p = .15 p = .15 ).

Optimal Difficulty: To maximize item variance and test score dispersion, the optimal average item difficulty is approximately p ≈ .50 p \approx .50 p ≈ .50 .

Correction for Guessing: On multiple-choice tests, optimal difficulty is adjusted halfway between the guessing probability ( g g g ) and 1.00 1.00 1.00 :

Optimal p = 1.00 + g 2 \text{Optimal } p = \frac{1.00 + g}{2} Optimal p = 2 1.00 + g ​

For a 4-option multiple-choice item ( g = .25 g = .25 g = .25 ), the optimal difficulty is 1.00 + .25 2 = .625 \frac{1.00 + .25}{2} = .625 2 1.00 + .25 ​ = .625 .

2. Item Discrimination Index ( D D D )

Item discrimination measures how effectively an item differentiates between high-scoring and low-scoring examinees. The most common classical metric is the extreme groups index ( D D D ) :

D = P U − P L D = P_U - P_L D = P U ​ − P L ​

Where P U P_U P U ​ is the proportion of examinees in the upper scoring bracket (top 27 % 27\% 27% ) answering correctly, and P L P_L P L ​ is the proportion in the lower scoring bracket (bottom 27 % 27\% 27% ) answering correctly.

D D D ranges from − 1.00 -1.00 − 1.00 to + 1.00 +1.00 + 1.00 .

D ≥ .40 D \ge .40 D ≥ .40 : Excellent discrimination; item cleanly distinguishes high performers.

D ≈ 0.00 D \approx 0.00 D ≈ 0.00 : Zero discrimination; item fails to differentiate high and low performers.

D < 0.00 D < 0.00 D < 0.00 : Negative discrimination ; more low performers answered correctly than high performers. Such items are psychometrically defective, indicating confusing distractors, typographical errors, or incorrect answer keys, and must be eliminated.

Item Response Theory (IRT) and Item Characteristic Curves

A major limitation of Classical Test Theory is that item parameters ( p p p and D D D ) are sample-dependent (an item appears hard in a low-ability sample and easy in a high-ability sample), and examinee ability scores are test-dependent (scores depend on the specific difficulty of the test administered).

Item Response Theory (IRT) overcomes these limitations by modeling the nonlinear mathematical relationship between an individual's unobservable latent trait level (denoted by Greek letter theta, θ \theta θ , where μ = 0 \mu = 0 μ = 0 and σ = 1 \sigma = 1 σ = 1 ) and the probability of correctly answering a specific item ( P ( θ ) P(\theta) P ( θ ) ). This relationship is depicted graphically as an S-shaped Item Characteristic Curve (ICC) or trace line . Item Characteristic Curve (3PL Model)

  • Probability P(θ)
  • 1.0 │ ● Upper Asymptote
  • │ ..-''
  • 0.8 │ _.-'
  • │ .-'
  • 0.6 │ .-' <-- Steepest Slope = Discrimination (a)
  • │ .-'
  • 0.4 │ .-' <-- Inflection Point θ = Difficulty (b)
  • │ _..-'
  • c --> 0.2 │...........-------'
  • (Guessing) │
  • 0.0 └─────────────────────────────────────────────────────────────
  • 3 -2 -1 0 +1 +2 +3
  • Latent Trait Level (θ)
  • The Three Logistic IRT Models

One-Parameter Logistic (1PL / Rasch Model):

Assumes all items have equal discrimination ( a a a ) and zero guessing ( c = 0 c = 0 c = 0 ).

Models only item difficulty ( b b b ) , defined as the latent trait level θ \theta θ at which an examinee has exactly a 50 % 50\% 50% probability of answering correctly ( P ( θ ) = .50 P(\theta) = .50 P ( θ ) = .50 ).

Two-Parameter Logistic (2PL Model):

Models both item difficulty ( b b b ) and item discrimination ( a a a ) .

The a a a -parameter represents the slope of the ICC at the inflection point ( b b b ). Steeper slopes reflect greater discrimination among examinees near the threshold θ \theta θ .

Three-Parameter Logistic (3PL Model):

Incorporates difficulty ( b b b ), discrimination ( a a a ), and a pseudo-guessing parameter ( c c c ) .

The c c c -parameter represents the lower horizontal asymptote of the ICC—the probability that an examinee with infinitely low ability ( θ → − ∞ \theta \rightarrow -\infty θ → − ∞ ) will endorse or guess the item correctly purely by chance.

Sample Invariance Property

The defining triumph of IRT over CTT is parameter invariance : item parameters ( a , b , c a, b, c a , b , c ) are invariant across different examinee populations, and examinee trait estimates ( θ \theta θ ) are invariant across different item subsets. This mathematical foundation enables Computerized Adaptive Testing (CAT) , where items are dynamically tailored in real time to the examinee's emerging θ \theta θ estimate.