Study guide
6.2 Stepwise Selection & Information Criteria Penalties
CAS Property & Casualty Predictive Analytics PCPA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: PCPA Exam & Project Orientation
1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading
2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing
2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56
3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM
3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling
4 Chapter 4: Model Diagnostics & Selection — Building a GLM
4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity
5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors
5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities
6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit
6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation
7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice
7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves
8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits
8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits
9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis
9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric
6.2 Stepwise Selection & Information Criteria Penalties
Key Takeaways Explain This Topic AI
Automated selection algorithms—forward addition, backward elimination, and bidirectional stepwise—iteratively evaluate predictor significance via deviance drops or score tests, but none guarantee identifying the globally optimal subset.
Stopping rules based on classical p-value thresholds (alpha_enter, alpha_stay) break down catastrophically on large insurance datasets (n > 100,000), where trivial noise fluctuations achieve extreme statistical significance (p < 0.0001).
Stepwise selection introduces severe methodological defects into actuarial models: severe Type I error inflation, post-selection hurdle bias (upwardly inflated coefficient estimates), spuriously narrow standard errors, and erratic variable flipping under collinearity.
Likelihood penalties balance goodness-of-fit against model complexity: AIC uses a fixed penalty of 2k, whereas BIC scales the penalty by sample size as k * ln(n).
In property and casualty ratemaking with massive policy volumes (n >= 100,000, ln(n) >= 11.51 >> 2), BIC imposes a far more rigorous parsimony standard than AIC, preventing over-parameterization and producing defensible rating structures.
Last updated: September 2026
Automated Subset Selection in Generalized Linear Models
In property and casualty (P&C) ratemaking, selecting an optimal subset of rating predictors is a critical balance between predictive precision, commercial interpretability, and regulatory defensibility. An insurance pricing model must capture legitimate risk differentials without over-parameterizing the rating algorithm or importing sample-specific noise into published rating manuals.
Historically, actuaries relied on automated variable selection procedures to screen candidate variables. Given p p p candidate rating variables, there are 2 p 2^p 2 p possible distinct subset models. When p p p is moderate to large (e.g., p = 40 p = 40 p = 40 , yielding 2 40 ≈ 1.1 × 10 12 2^{40} \approx 1.1 \times 10^{12} 2 40 ≈ 1.1 × 1 0 12 candidate models), exhaustive best-subset search is computationally intractable. Automated selection algorithms navigate this combinatorial space via greedy, iterative heuristics. AUTOMATED SUBSET ALGORITHMS
- │
- ┌──────────────────────────────┼──────────────────────────────┐
- ▼ ▼ ▼
- Forward Addition Backward Elimination Stepwise Bidirectional
- Start: Null Model (β₀) Start: Full Model (all p) Start: Null or Intercept
- Iteratively ADD variable Iteratively DROP variable Alternate ADD and DROP
- with largest deviance drop with smallest deviance drop Re-test existing variables
- Stops when ΔD < threshold Stops when all p ≤ α_stay Prevents trapped paths
- 1. Forward Addition
Forward selection begins with the simplest possible baseline—the null model containing only the intercept parameter β 0 \beta_0 β 0 : ln ( μ i ) = β 0 \ln(\mu_i) = \beta_0 ln ( μ i ) = β 0
At each subsequent step:
The algorithm evaluates every candidate predictor X j X_j X j not currently present in the model by fitting a single-parameter addition.
It identifies the variable that produces the largest reduction in model deviance (or equivalently, the largest increase in log-likelihood Δ ℓ \Delta \ell Δ ℓ , or the lowest p-value in a score test).
If the deviance drop exceeds a predefined threshold (or if the p-value is below a specified entry criterion α enter \alpha_{\text{enter}} α enter ), the variable is permanently added to the model.
The process repeats iteratively until no remaining candidate satisfies the entry criterion.
Actuarial Limitation : Forward addition is myopic. Because variables added in early stages cannot be subsequently removed, the algorithm cannot account for changing covariance structures as additional predictors enter. A variable that appeared highly predictive initially may become redundant once a collinear cluster of variables enters later.
2. Backward Elimination
Backward elimination begins at the opposite extreme—the saturated or full candidate model containing all p p p candidate predictors simultaneously: ln ( μ i ) = β 0 + ∑ j = 1 p β j X i j \ln(\mu_i) = \beta_0 + \sum_{j=1}^p \beta_j X_{ij} ln ( μ i ) = β 0 + j = 1 ∑ p β j X ij
At each subsequent step:
The algorithm assesses the marginal significance of each variable currently in the model using a Wald test, score test, or likelihood ratio test (LRT) drop-in-deviance.
The variable that contributes the least to the model—the one whose removal causes the smallest increase in deviance (largest p-value)—is flagged.
If this largest p-value exceeds a predefined removal criterion α stay \alpha_{\text{stay}} α stay , the variable is eliminated.
The process repeats until all variables remaining in the model satisfy the condition p ≤ α stay p \le \alpha_{\text{stay}} p ≤ α stay .
Actuarial Limitation : Backward elimination requires fitting the full model upfront. If p > n p > n p > n (common in genomics, though rarer in core P&C ratemaking), or if high-dimensional dummy coding creates severe rank-deficiency in the information matrix X T W X \mathbf{X}^T \mathbf{W} \mathbf{X} X T WX , the full model cannot be uniquely estimated by maximum likelihood.
3. Stepwise Bidirectional Selection
Stepwise bidirectional selection combines forward addition and backward elimination. Starting from either the null model or an initial baseline rating structure:
At each step, candidate variables not in the model are evaluated for addition using an entry threshold α enter \alpha_{\text{enter}} α enter .
If a variable enters, all variables currently in the model are immediately re-evaluated against a removal threshold α stay \alpha_{\text{stay}} α stay to determine if any have become redundant.
The algorithm alternates between entry and elimination steps until no variables can enter and none can be removed. STEPWISE BIDIRECTIONAL LOOP
- │
- ┌─────────────▼─────────────┐
- │ Evaluate Unselected Vars │
- └─────────────┬─────────────┘
- │
Best p-value < α_enter?
- / \
- YES / \ NO ───► STOP (Final Model)
- ▼
- Add Variable to Model
- │
- ┌──────────▼──────────┐
- │ Re-test All Vars │
- │ Currently in Model │
- └──────────┬──────────┘
- │
Worst p-value > α_stay?
- / \
- YES / \ NO
- ▼ └──► Continue to next cycle
- Remove Variable
- │
- └──► Continue to next cycle
Important
The Critical Alpha Constraint : To prevent the stepwise algorithm from entering an infinite loop—where the same variable is repeatedly added in the forward phase and removed in the backward phase—the threshold parameters must strictly satisfy: α enter < α stay \alpha_{\text{enter}} < \alpha_{\text{stay}} α enter < α stay Setting α enter = 0.05 \alpha_{\text{enter}} = 0.05 α enter = 0.05 and α stay = 0.10 \alpha_{\text{stay}} = 0.10 α stay = 0.10 is a standard convention, ensuring that a variable requires stronger evidence to enter than to remain.
Stopping Rules: P-Value Thresholds vs. Information Criteria
Automated selection routines require an objective mathematical stopping rule. Historically, statistical software utilized hypothesis test p-values based on the Likelihood Ratio Test (LRT): Δ D = D reduced − D full = 2 [ ℓ ( β ^ full ) − ℓ ( β ^ reduced ) ] ∼ χ Δ d f 2 \Delta D = D_{\text{reduced}} - D_{\text{full}} = 2 \left[ \ell(\hat{\boldsymbol{\beta}}_{\text{full}}) - \ell(\hat{\boldsymbol{\beta}}_{\text{reduced}}) \right] \sim \chi^2_{\Delta df} Δ D = D reduced − D full = 2 [ ℓ ( β ^ full ) − ℓ ( β ^ reduced ) ] ∼ χ Δ df 2
Where Δ d f \Delta df Δ df is the difference in the number of estimated parameters. If the p-value P ( χ Δ d f 2 > Δ D ) P(\chi^2_{\Delta df} > \Delta D) P ( χ Δ df 2 > Δ D ) is below α enter \alpha_{\text{enter}} α enter , the variable is retained.
The Breakdown of P-Values on Large P&C Datasets
While p-value stopping rules function acceptably on classical small-sample statistical experiments ( n = 100 n = 100 n = 100 to 1 , 000 1,000 1 , 000 ), they fail catastrophically when applied to modern insurance datasets where sample size n n n regularly exceeds 100 , 000 100,000 100 , 000 to 10 , 000 , 000 10,000,000 10 , 000 , 000 policy records.
The standard error of any estimated regression coefficient scales inversely with the square root of the effective sample size (or earned exposure): SE ( β ^ j ) ∝ 1 n \text{SE}(\hat{\beta}_j) \propto \frac{1}{\sqrt{n}} SE ( β ^ j ) ∝ n 1
As n → ∞ n \to \infty n → ∞ , SE ( β ^ j ) → 0 \text{SE}(\hat{\beta}_j) \to 0 SE ( β ^ j ) → 0 . Consequently, the Wald test statistic: z = β ^ j − 0 SE ( β ^ j ) = β ^ j n ⋅ c z = \frac{\hat{\beta}_j - 0}{\text{SE}(\hat{\beta}_j)} = \hat{\beta}_j \sqrt{n} \cdot c z = SE ( β ^ j ) β ^ j − 0 = β ^ j n ⋅ c
grows arbitrarily large for any non-zero coefficient, no matter how microscopically trivial the true underlying effect. In a personal auto portfolio with 500 , 000 500,000 500 , 000 earned car-years:
A completely irrelevant variable—such as the policyholder's favorite color or the day of the week the application was signed—might exhibit a tiny sample frequency relativity difference of 0.15 % 0.15\% 0.15% ( β ^ = 0.0015 \hat{\beta} = 0.0015 β ^ = 0.0015 ).
With n = 500 , 000 n = 500,000 n = 500 , 000 , SE ( β ^ ) ≈ 0.0003 \text{SE}(\hat{\beta}) \approx 0.0003 SE ( β ^ ) ≈ 0.0003 .
The resulting test statistic is z = 0.0015 / 0.0003 = 5.0 z = 0.0015 / 0.0003 = 5.0 z = 0.0015/0.0003 = 5.0 , yielding p ≈ 0.0000006 p \approx 0.0000006 p ≈ 0.0000006 .
Under a standard α enter = 0.05 \alpha_{\text{enter}} = 0.05 α enter = 0.05 threshold, this completely spurious, commercially absurd variable is aggressively selected. Classical hypothesis testing answers the question: "Is this coefficient exactly zero in the population?" On massive insurance datasets, virtually no physical variable has an effect size of identically zero down to the eighth decimal place. Statistical significance decouples entirely from practical actuarial significance.
Methodological Flaws of Stepwise Selection on Actuarial Datasets
Modern actuarial literature (including Frank Harrell's Regression Modeling Strategies and CAS research papers) strongly cautions against stepwise selection. Beyond the sample size dilemma, stepwise algorithms introduce four fatal methodological distortions:
1. Severe Type I Error Inflation
Stepwise selection conducts dozens or hundreds of sequential hypothesis tests without adjusting for multiple comparisons. If an actuary screens 50 candidate telematics variables at nominal α = 0.05 \alpha = 0.05 α = 0.05 , the family-wise error rate ( F W E R FWER F W ER )—the probability of falsely selecting at least one purely random noise variable—is: FWER = 1 − ( 1 − α ) m = 1 − ( 0.95 ) 50 ≈ 1 − 0.0769 = 92.31 % \text{FWER} = 1 - (1 - \alpha)^m = 1 - (0.95)^{50} \approx 1 - 0.0769 = 92.31\% FWER = 1 − ( 1 − α ) m = 1 − ( 0.95 ) 50 ≈ 1 − 0.0769 = 92.31%
The algorithm is virtually guaranteed to include false positives, mistaking sampling noise for genuine actuarial risk signals.
2. Selection Bias & The Hurdle Effect (Winner's Curse)
When a variable is selected because its test statistic cleared a significance hurdle, its estimated coefficient is systematically biased away from zero: E [ ∣ β ^ j ∣ ∣ Variable X j selected ] > ∣ β j ∣ \mathbb{E}\left[ |\hat{\beta}_j| \,\Big|\, \text{Variable } X_j \text{ selected} \right] > |\beta_j| E [ ∣ β ^ j ∣ Variable X j selected ] > ∣ β j ∣
In finite samples, an estimated coefficient β ^ j \hat{\beta}_j β ^ j equals the true effect β j \beta_j β j plus random estimation error ε j \varepsilon_j ε j . A variable is most likely to clear the entry hurdle when its random noise happens to align in the same direction as its true effect, artificially inflating the test statistic. This phenomenon—the hurdle effect or winner's curse —means that coefficients in stepwise models are systematically exaggerated. When exponentiated into rating relativities exp ( β ^ j ) \exp(\hat{\beta}_j) exp ( β ^ j ) , the model over-prices higher-risk tiers and under-prices lower-risk tiers.
3. Artificially Narrow Standard Errors & Falsely Narrow Confidence Intervals
Standard statistical software prints a regression summary table displaying standard errors, t-statistics, and confidence intervals. However, the software calculates these quantities under the assumption that the model structure was fixed a priori before looking at the data .
In reality, the degrees of freedom expended during the search across all candidate variables are ignored. The software treats the final k k k selected variables as if they were the only k k k variables ever considered, drastically understating the true parameter uncertainty. The reported standard errors are falsely narrow, p-values are deceptively small, and confidence intervals provide an illusion of precision.
4. Extreme Instability Under Collinearity
When candidate rating variables exhibit collinearity (e.g., credit score vs. insurance tier, vehicle horsepower vs. vehicle weight, territory population density vs. vehicle theft rate), stepwise algorithms become erratic. A minor perturbation in the data—such as withholding one cross-validation fold or updating the model with one new quarter of claims—causes the algorithm to select completely different subsets of variables. Such instability makes the resulting rating plan impossible to defend before state insurance regulators or explain to senior underwriting executives.
Penalized Likelihood Criteria: AIC vs. BIC
To overcome the limitations of arbitrary p-value thresholds, actuaries utilize information criteria that directly penalize the log-likelihood function for model complexity. PENALIZED LIKELIHOOD CRITERIA
- │
- ┌──────────────────────┴──────────────────────┐
- ▼ ▼
- Akaike Information (AIC) Bayesian Information (BIC)
- Formula: -2·ln(L) + 2k Formula: -2·ln(L) + k·ln(n)
- Penalty per Parameter: 2 Penalty per Parameter: ln(n)
- Objective: Minimize KL divergence Objective: Maximize posterior prob
- Target: Best out-of-sample prediction Target: Identify "true" parsimonious model
- Large Datasets: Retains too many variables Large Datasets: Heavily penalizes complexity
- Mathematical Formulation
Let ℓ ( β ^ ) \ell(\hat{\boldsymbol{\beta}}) ℓ ( β ^ ) denote the maximized log-likelihood of a GLM, k k k denote the total number of estimated parameters (including the intercept and any estimated dispersion parameter ϕ \phi ϕ ), and n n n denote the total effective sample size (number of independent policy records or earned exposure units). AIC = − 2 ℓ ( β ^ ) + 2 k = D ( y , μ ^ ) + 2 k ϕ \text{AIC} = -2\ell(\hat{\boldsymbol{\beta}}) + 2k = D(\mathbf{y}, \hat{\boldsymbol{\mu}}) + 2k\phi AIC = − 2 ℓ ( β ^ ) + 2 k = D ( y , μ ^ ) + 2 k ϕ BIC = − 2 ℓ ( β ^ ) + k ln ( n ) = D ( y , μ ^ ) + k ln ( n ) ϕ \text{BIC} = -2\ell(\hat{\boldsymbol{\beta}}) + k \ln(n) = D(\mathbf{y}, \hat{\boldsymbol{\mu}}) + k\ln(n)\phi BIC = − 2 ℓ ( β ^ ) + k ln ( n ) = D ( y , μ ^ ) + k ln ( n ) ϕ
Where D ( y , μ ^ ) D(\mathbf{y}, \hat{\boldsymbol{\mu}}) D ( y , μ ^ ) is the unscaled deviance. In both criteria, smaller values indicate superior models. The first term ( − 2 ℓ -2\ell − 2 ℓ ) rewards goodness-of-fit, while the second term ( 2 k 2k 2 k or k ln ( n ) k\ln(n) k ln ( n ) ) penalizes model complexity.
Theoretical Foundations: Efficiency vs. Consistency
Akaike Information Criterion (AIC) : Derived by Hirotugu Akaike from information theory, AIC provides an asymptotically unbiased estimator of the relative Kullback-Leibler (K-L) divergence between the fitted model and the true unknown data-generating process. AIC is asymptotically efficient : as n → ∞ n \to \infty n → ∞ , it selects the model that minimizes mean squared prediction error. However, AIC is not consistent : even as n → ∞ n \to \infty n → ∞ , the probability that AIC selects an over-parameterized model containing irrelevant variables does not converge to zero; it approaches a non-zero constant ( P ( overfitting ) ≈ 15.7 % P(\text{overfitting}) \approx 15.7\% P ( overfitting ) ≈ 15.7% for a 1-parameter addition).
Bayesian Information Criterion (BIC) : Derived by Gideon Schwarz from a Bayesian framework, BIC approximates the logarithm of the marginal likelihood (Bayes factor) under a uniform prior over candidate models. BIC is statistically consistent : as n → ∞ n \to \infty n → ∞ , the probability that BIC selects the true data-generating model approaches 1.0 1.0 1.0 (assuming the true model is in the candidate set).
The Sample Size Multiplier: Why BIC Dominates in Big P&C Data
The fundamental operational difference between AIC and BIC lies in the parameter penalty coefficient:
AIC Penalty per parameter : Exactly 2.0 2.0 2.0 , regardless of sample size.
BIC Penalty per parameter : ln ( n ) \ln(n) ln ( n ) , which grows monotonically with the sample size.
The threshold at which BIC penalizes complexity more harshly than AIC occurs when: ln ( n ) > 2 ⟺ n > e 2 ≈ 7.389 \ln(n) > 2 \iff n > e^2 \approx 7.389 ln ( n ) > 2 ⟺ n > e 2 ≈ 7.389
For any dataset with more than 7 observations, BIC imposes a steeper penalty than AIC. On large insurance portfolios, this difference becomes massive:
Portfolio Size Effective Sample Size ( n n n ) AIC Penalty ( 2 k 2k 2 k ) BIC Penalty ( k ln ( n ) k \ln(n) k ln ( n ) ) Ratio: BIC Penalty / AIC Penalty Required Δ D \Delta D Δ D for Δ k = 1 \Delta k = 1 Δ k = 1 (BIC) Equivalent χ 1 2 \chi^2_1 χ 1 2 P-value (BIC)
Small Commercial 5 , 000 5,000 5 , 000 2.00 2.00 2.00 8.52 8.52 8.52 4.26 × 4.26\times 4.26 × 8.52 8.52 8.52 0.0035 0.0035 0.0035
Mid-Size Regional 50 , 000 50,000 50 , 000 2.00 2.00 2.00 10.82 10.82 10.82 5.41 × 5.41\times 5.41 × 10.82 10.82 10.82 0.0010 0.0010 0.0010
Large Personal Lines 100 , 000 100,000 100 , 000 2.00 2.00 2.00 11.51 11.51 11.51 5.76 × 5.76\times 5.76 × 11.51 11.51 11.51 0.00069 0.00069 0.00069
Multi-State Personal Auto 1 , 000 , 000 1,000,000 1 , 000 , 000 2.00 2.00 2.00 13.82 13.82 13.82 6.91 × 6.91\times 6.91 × 13.82 13.82 13.82 0.00020 0.00020 0.00020
Mega-Carrier Telematics 10 , 000 , 000 10,000,000 10 , 000 , 000 2.00 2.00 2.00 16.12 16.12 16.12 8.06 × 8.06\times 8.06 × 16.12 16.12 16.12 0.00006 0.00006 0.00006
Practical Ratemaking Implications
Consider adding a single rating level (e.g., distinguishing between 4-door sedans and 2-door coupes, adding Δ k = 1 \Delta k = 1 Δ k = 1 ) to a personal auto frequency GLM fit on 1 , 000 , 000 1,000,000 1 , 000 , 000 earned vehicle-years:
Under AIC : The variable is added if the deviance drop satisfies Δ D > 2.00 \Delta D > 2.00 Δ D > 2.00 . Under a χ 1 2 \chi^2_1 χ 1 2 distribution, a critical value of 2.00 2.00 2.00 corresponds to a p-value of p < 0.1573 p < 0.1573 p < 0.1573 . AIC acts as a very loose filter, readily absorbing minor, borderline rating factors that capture minor sample quirks.
Under BIC : The variable is added only if the deviance drop satisfies Δ D > 13.82 \Delta D > 13.82 Δ D > 13.82 . Under a χ 1 2 \chi^2_1 χ 1 2 distribution, a critical value of 13.82 13.82 13.82 corresponds to a p-value of p < 0.0002 p < 0.0002 p < 0.0002 . BIC requires overwhelming statistical evidence before permitting an additional parameter.
In casualty insurance ratemaking, parsimony is paramount . Every additional rating tier must be implemented in policy administration systems, monitored in rate filings, and defended before state insurance commissioners. Over-parameterized models lead to unstable rating relativities, thin risk cells, and potential regulatory disapproval. Consequently, actuaries favor BIC (or regularized regression) over AIC when evaluating rating structures on large policy databases.