6.4 Under-fitting, Over-fitting & Cross-Validation

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

6.4 Under-fitting, Over-fitting & Cross-Validation

Key Takeaways Explain This Topic AI

Partitioning actuarial datasets requires three strictly segregated subsets: Training (parameter estimation), Validation (hyperparameter tuning, model architecture selection, and early stopping), and Test/Holdout (final unbiased evaluation of generalization prior to filing).

Standard random k-fold cross-validation causes catastrophic data leakage in P&C ratemaking due to macroeconomic inflation, weather catastrophe clustering, seasonal driving patterns, and long-tail claim emergence.

Out-of-time (OOT) validation structured chronologically by policy year or accident year (e.g., training on 2021–2023, validating on 2024, testing on 2025) directly replicates prospective ratemaking and detects temporal concept drift.

Stratified splitting guarantees that rare, high-severity events (shock losses exceeding $500k or litigated commercial claims) and key rating classes maintain identical empirical proportions across all data partitions.

Grouped/cluster splitting by policyholder ID, household account, or commercial fleet prevents intra-entity correlation leakage, ensuring multi-vehicle policies and longitudinal policy renewals remain strictly within a single partition.

Last updated: September 2026

7.1 Insurance Data Splitting & Cross-Validation

Exam Focus: In academic machine learning, random cross-validation is often treated as the gold standard for model evaluation. In property and casualty (P&C) actuarial practice, applying standard random splits to insurance experience data is a fatal modeling error. Because insurance data exhibits strong temporal correlation, macroeconomic loss cost inflation, weather catastrophe clustering, and hierarchical policyholder structures, naive data partitioning introduces catastrophic data leakage. Candidates must master the distinct roles of training, validation, and holdout test sets, how to design chronological out-of-time (OOT) splits that mirror prospective ratemaking, and how to execute stratified and grouped cluster sampling.

Predictive models in property and casualty insurance are developed to establish prospective rates, underwrite upcoming renewal cohorts, and optimize future marketing or claims triage. Evaluating a model on historical data requires simulating this forward-looking operational environment. If the validation architecture leaks future information or fails to preserve natural clustering, the model will produce over-optimistic performance metrics that collapse upon deployment.

1. The Three-Way Partitioning Architecture

A disciplined actuarial modeling pipeline partitions historical policy and claim records into three distinct, non-overlapping subsets: the Training Set , the Validation Set , and the Holdout Test Set . HISTORICAL INSURANCE DATASET

• │
• ┌─────────────────────────────┴─────────────────────────────┐
• ▼ ▼
• ┌───────────────────────────────────────────────┐ ┌─────────────────────────────┐
• │ Development / Modeling Sample │ │ Holdout Test Set │
• │ (80%) │ │ (20%) │
• └───────────────────────┬───────────────────────┘ └──────────────┬──────────────┘
• │ │
• ┌─────────────┴─────────────┐ │
• ▼ ▼ │
• ┌──────────────┐ ┌──────────────┐ │
• │ Training Set │ │Validation Set│ │
• │ (60%) │ │ (20%) │ │
• └──────┬───────┘ └──────┬───────┘ │
• │ │ │
• ▼ ▼ ▼
• Fits model parameters Tunes hyperparameters, Evaluates final
• (GLM coefficients, evaluates early stopping, generalization, pricing
• tree node splits) compares algorithms impact, and rate filings
• Partitioning Roles in Actuarial Workflows

Partition Primary Function Actuarial Operations & Tasks Governance & Leakage Controls

Training Set Parameter estimation and base fitting Fits GLM regression coefficients ( β ^ \hat{\boldsymbol{\beta}} β ^ ​ ), determines CART split locations, learns neural network weights, and estimates base rates. The largest partition (typically 50%–70% of historical exposure). Must never contain target information from future periods.

Validation Set Hyperparameter optimization and model selection Compares candidate GLM specifications: distribution and link choice, banding schemes for continuous predictors, which interactions to include, and how thin categorical levels are grouped. Repeatedly evaluated during development. Hyperparameters are selected based on minimum validation deviance or maximum validation lift.

Test / Holdout Set Final unbiased evaluation of generalization Provides an honest assessment of prospective model performance, computes out-of-sample Gini coefficients, constructs rate filing lift charts, and quantifies financial impact. Must be touched strictly once at the very conclusion of the modeling process. Iteratively re-tuning models based on test set feedback creates 'data snooping' leakage.

Mathematical Formulation: Empirical Risk vs. Generalization Risk

Let D = { ( x i , y i , w i ) } i = 1 n \mathcal{D} = \{(\mathbf{x}_i, y_i, w_i)\}_{i=1}^n D = {( x i ​ , y i ​ , w i ​ ) } i = 1 n ​ represent an insurance dataset where x i \mathbf{x}_i x i ​ is the rating vector, y i y_i y i ​ is the observed response (claim count, severity, or loss cost), and w i w_i w i ​ is earned exposure. The empirical training risk under deviance loss d ( y i , μ i ) d(y_i, \mu_i) d ( y i ​ , μ i ​ ) is: R train ( θ ) = 1 ∑ i ∈ D train w i ∑ i ∈ D train w i d ( y i , g − 1 ( x i T θ ) ) R_{\text{train}}(\boldsymbol{\theta}) = \frac{1}{\sum_{i \in \mathcal{D}_{\text{train}}} w_i} \sum_{i \in \mathcal{D}_{\text{train}}} w_i d(y_i, g^{-1}(\mathbf{x}_i^T \boldsymbol{\theta})) R train ​ ( θ ) = ∑ i ∈ D train ​ ​ w i ​ 1 ​ i ∈ D train ​ ∑ ​ w i ​ d ( y i ​ , g − 1 ( x i T ​ θ ))

Because optimization routines minimize R train ( θ ) R_{\text{train}}(\boldsymbol{\theta}) R train ​ ( θ ) directly, R train R_{\text{train}} R train ​ systematically underestimates the true prospective loss cost discrepancy. The generalization error must be estimated on the independent holdout partition D test \mathcal{D}_{\text{test}} D test ​ : R gen ≈ R test ( θ ^ ) = 1 ∑ i ∈ D test w i ∑ i ∈ D test w i d ( y i , g − 1 ( x i T θ ^ ) ) R_{\text{gen}} \approx R_{\text{test}}(\hat{\boldsymbol{\theta}}) = \frac{1}{\sum_{i \in \mathcal{D}_{\text{test}}} w_i} \sum_{i \in \mathcal{D}_{\text{test}}} w_i d(y_i, g^{-1}(\mathbf{x}_i^T \hat{\boldsymbol{\theta}})) R gen ​ ≈ R test ​ ( θ ^ ) = ∑ i ∈ D test ​ ​ w i ​ 1 ​ i ∈ D test ​ ∑ ​ w i ​ d ( y i ​ , g − 1 ( x i T ​ θ ^ ))

2. Temporal Validation & Out-of-Time (OOT) Splits

In standard cross-validation, observations are shuffled randomly across k k k folds. In property and casualty insurance, random cross-validation causes severe, invalidating data leakage .

• Why Random Splitting Fails in P&C Insurance HAZARDS OF RANDOM SPLITTING IN P&C
• │
• ┌────────────────────────┬────────────┴────────────┬────────────────────────┐
• ▼ ▼ ▼ ▼
• ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
• │ Macro Trend │ │ Weather Cat │ │ Seasonal │ │ Look-Ahead │
• │ Inflation │ │ Clustering │ │ Patterns │ │ Bias │
• └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
• │ │ │ │
• ▼ ▼ ▼ ▼
• Severe medical/repair Hail storm/freeze event Winter ice or hurricane Model trains on future
• inflation in 2024 leaks in June 2023 affects 500 driving season claims rates/losses to predict
• into 2021 predictions policies in train & test leak across random folds past policy outcomes

Macroeconomic Trends and Social Inflation: Loss costs are subject to persistent, non-stationary inflation. Supply-chain disruptions, medical cost increases, vehicle repair technology changes (e.g., ADAS sensors in bumpers), and litigation trends (social inflation) cause loss costs to drift over time. Random splitting interweaves 2021 and 2024 claims, allowing the model to learn 2024 price levels when predicting 2021 policies—an impossible condition in prospective ratemaking.

Weather Catastrophe Clustering: P&C claim occurrences are not independent. An atmospheric river, hurricane, or severe convective hail storm creates thousands of correlated property and auto physical damage claims across a single 48-hour window. If policies exposed to the same storm are randomly scattered between training and test folds, the model memorizes the specific geographical footprint of that historical storm rather than learning true underlying risk characteristics.

Seasonal Variations: Claim frequency varies dramatically by month (e.g., winter collision peaks in northern states, summer recreational driving surges, autumnal hurricane activity). Random splits homogenize seasonal shocks across folds, masking the model's inability to handle cyclical claim spikes.

Look-Ahead Bias: In production, an insurer prices policies written today that will incur losses over the upcoming 12 to 24 months. Random splitting utilizes future data to predict the past, directly violating the temporal direction of insurance operations.

Chronological Out-of-Time (OOT) Architecture

To faithfully replicate prospective pricing, actuaries structure partitions strictly along chronological boundaries using Policy Year or Accident Year : CHRONOLOGICAL OUT-OF-TIME SPLIT

• Policy Year: 2021 2022 2023 2024 2025
• ┌───────────────┬───────────────┬───────────────┬───────────────┬───────────────┐
• │ TRAINING SET │ VALIDATION │ HOLDOUT │
• │ (Policy Years 2021 - 2023) │ (P.Y. 2024) │ (P.Y. 2025) │
• └───────────────────────────────────────────────┴───────────────┴───────────────┘
• │ │
• ▼ ▼
• Tuning & Prospective
• Pruning Evaluation
• Actuarial Experience Alignment: Policy Year vs. Accident Year

When designing chronological splits, actuaries must respect the operational grain of the data:

Policy Year (Underwriting Year): Groups all exposures and incurred losses arising from policies written within a specific calendar year. A policy written on December 31, 2023, remains in force through December 30, 2024. Policy year splits align perfectly with rating algorithm updates, as entire contracts and their complete loss experience are assigned to a single cohort.

Accident Year (Occurrence Year): Groups all claims occurring in a calendar year, regardless of when the policy was written. While accident year data aligns with macroeconomic inflation and catastrophe events, matching exposures requires earned exposure unbundling.

Important

Claim Development Lag (IBNR / IBNER): When selecting recent periods for validation or test sets (e.g., Policy Year 2024 or 2025), actuaries must recognize that recent experience is immature. Open claims will experience reserve development (IBNER), and late-reported claims have not yet emerged (IBNR). Actuaries must either develop losses to an ultimate basis using actuarial loss development factors (LDFs) before model evaluation or truncate validation windows to mature historical periods.

Rolling-Origin (Expanding Window) Cross-Validation

Rather than relying on a single temporal split, actuaries frequently evaluate model stability across multiple time periods using Rolling-Origin (Time-Series) Cross-Validation : Fold 1: [ Train: 2019 - 2021 ] ──► [ Validate: 2022 ]

• Fold 2: [ Train: 2019 - 2022 ] ──────► [ Validate: 2023 ]
• Fold 3: [ Train: 2019 - 2023 ] ──────────► [ Validate: 2024 ]

This structure ensures that training data always precedes validation data, testing whether the model's predictive superiority remains consistent across changing underwriting cycles and inflation regimes.

3. Stratified Splitting for Rare, High-Severity Events

Insurance loss data is characterized by extreme right-skewness and excess zeros: in personal auto, 90%–95% of policies incur zero claims; in commercial general liability, less than 2% of accounts incur losses, but a single environmental liability or construction defect claim can exceed $1,000,000. THE IMBALANCED LOSS SPECTRUM

• Frequency: ███████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░
• 93% Zero Claims 6.9% Small Claims 0.1% Shock

• Loss Dollars:░░░░░░░░░░░░░░░░░░░░░░░░░░░░ █████████████████████████████
• Zero Losses (0%) Attritional (40%) Shock Losses (60%)
• Why Unstratified Splitting Distorts Model Evaluation

If an actuary performs a naive unstratified partition on a commercial auto dataset with 10 shock claims exceeding $500,000:

Pure random chance may assign 9 shock claims to the training set and only 1 to the test set.

The training set severity will appear artificially inflated, while test set deviance, average loss cost, and lift metrics will be severely distorted.

The actuary cannot determine whether poor test performance is caused by model failure or severe sampling imbalance.

Implementation of Multi-Tier Actuarial Stratification

To ensure statistical comparability across partitions, actuaries apply stratified sampling , preserving identical empirical distributions of critical risk attributes across all folds:

Binary Claim Occurrence Stratification: For frequency models, partition policies so that the proportion of claim-free policies ( N i = 0 N_i = 0 N i ​ = 0 ) versus claim-incurring policies ( N i ≥ 1 N_i \ge 1 N i ​ ≥ 1 ) is identical across training, validation, and test sets.

Severity Tier Stratification: For pure premium and severity models, partition claims across discrete loss layers:

Tier 1 (Attritional Losses): Claims with 0 < y i ≤ 50 , 000 0 < y_i \le 50,000 0 < y i ​ ≤ 50 , 000

Tier 2 (Large Losses): Claims with 50 , 000 < y i ≤ 250 , 000 50,000 < y_i \le 250,000 50 , 000 < y i ​ ≤ 250 , 000

Tier 3 (Shock / Catastrophe Losses): Claims with y i > 250 , 000 y_i > 250,000 y i ​ > 250 , 000

Line of Business / Coverage Stratification: In multi-peril products (such as Commercial Multi-Peril or Homeowners), stratify by primary coverage (e.g., Property vs. Premises Liability vs. Water Backup) to guarantee that each partition contains representative coverage mixes. # Conceptual Actuarial Stratification Variable Construction

# Combines coverage line and loss severity tier into a single composite stratum

def build_actuarial_stratum(policy):

if policy.claim_count == 0:

return f"{policy.line}_zero"

elif policy.total_loss < 50000:

return f"{policy.line}_attritional"

elif policy.total_loss < 250000:

return f"{policy.line}_large"

else:

• return f"{policy.line}_shock"
• 4. Grouped / Cluster Splitting for Hierarchical Entities

Many property and casualty portfolios contain multiple observations linked to the same underlying insured entity. Examples include:

Personal Auto Policies with Multiple Vehicles: A single household policy covering three vehicles and four drivers.

Longitudinal Policy Renewals: A commercial enterprise insured with the carrier across six consecutive annual policy terms (2018–2023).

Commercial Fleet Accounts: A trucking firm operating 250 power units, each represented as a distinct vehicle-year record.

Commercial Property Multi-Location Schedules: A retailer with 40 individual store locations insured under a single commercial property schedule. HIERARCHICAL ENTITY STRUCTURE

• │
• Commercial Account: Fleet Corp
• (Account ID: ACC-948102)
• │
• ┌─────────────────────────────┼─────────────────────────────┐
• ▼ ▼ ▼
• Vehicle 101 Vehicle 102 Vehicle 103
• (Long-Haul Semi) (Delivery Box Truck) (Service Van)
• │ │ │
• Policy Year 2022 Policy Year 2022 Policy Year 2022
• Policy Year 2023 Policy Year 2023 Policy Year 2023
• Policy Year 2024 Policy Year 2024 Policy Year 2024

MANDATE: All 9 records MUST be assigned to the same partition!

The Danger of Intra-Cluster Correlation Leakage

Individual vehicles or locations within an account share powerful unobserved risk characteristics:

The commercial fleet shares maintenance standards, driver safety protocols, geographic terminal locations, and telematics speed governances.

A personal auto household shares garaging security, neighbourhood road hazards, and shared vehicle operation between family members.

Multi-year policy renewals reflect invariant policyholder temperament, credit behavior, and property upkeep.

If records from the same entity are split randomly across training and validation sets:

The model learns the idiosyncratic risk profile of that specific household or account in the training set (e.g., memorizing specific high-hazard driver combinations or vehicle fleet age patterns).

When predicting the validation records for the same household, the model appears brilliantly accurate.

In production, when the model is applied to entirely new accounts, predictive performance collapses because the model memorized customer identity rather than learning generalizable actuarial relationships.

Algorithmic Grouping Rule

Under Grouped / Cluster Splitting , the sampling unit is the primary entity identifier ( I D entity ID_{\text{entity}} I D entity ​ ), rather than the individual policy or exposure row. All records sharing the same grouping key must reside exclusively within the training set, the validation set, or the holdout test set: S train ∩ S val = ∅ , S train ∩ S test = ∅ , S val ∩ S test = ∅ \mathcal{S}_{\text{train}} \cap \mathcal{S}_{\text{val}} = \emptyset, \quad \mathcal{S}_{\text{train}} \cap \mathcal{S}_{\text{test}} = \emptyset, \quad \mathcal{S}_{\text{val}} \cap \mathcal{S}_{\text{test}} = \emptyset S train ​ ∩ S val ​ = ∅ , S train ​ ∩ S test ​ = ∅ , S val ​ ∩ S test ​ = ∅

Where S k = { I D entity ∈ Partition k } \mathcal{S}_k = \{ID_{\text{entity}} \in \text{Partition}_k\} S k ​ = { I D entity ​ ∈ Partition k ​ } .

• 5. Actuarial Traps & Exam Pitfalls ┌────────────────────────────────────────────────────────┐
• │ Actuarial Traps in Data Splitting │
• └────────────────────────────────────────────────────────┘
• │
• ┌───────────────────────────────────────┼───────────────────────────────────────┐
• ▼ ▼ ▼
• ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
• │ Random K-Fold│ │ Immature OOT │ │ Account-Level│
• │ on Time-Data │ │ Validation │ │ Record Split │
• └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
• │ │ │
• ▼ ▼ ▼
• Leaks catastrophe weather Recent policy years have Splitting renewals or
• events and inflation into substantial IBNR/IBNER; multi-vehicle policies
• past periods; results unadjusted losses under- allows model to memorize

in production failure. state actual true risk. customer identity.

Trap 1: Submitting Rate Filings Based on In-Sample or Random Cross-Validation Metrics

Insurance department regulators and actuarial peer reviewers will challenge any filing where model separation diagnostics (such as Gini coefficients or lift charts) are calculated in-sample or using naive random k-fold CV on multi-year data. Regulators recognize that random CV masks the effects of loss inflation and catastrophe clustering, overstating rate adequacy.

Trap 2: Neglecting Loss Development on Recent Out-of-Time Partitions

When validating on the most recent completed policy year (e.g., using 2024 data evaluated in early 2025), reported losses are immature. If the actuary calculates validation deviance on raw reported losses, low-frequency, long-tail liability claims will appear absent. The model will appear to severely over-predict losses, leading the actuary to improperly deflate base rates. Actuaries must apply policy-year loss development factors or restrict OOT validation to mature years.

Trap 3: Splitting Multi-Car Policies Across Folds

A candidate builds an auto frequency model. Policies covering two cars (e.g., Mom and Teenager) have their vehicle records split: Car 1 goes to train, Car 2 goes to test. Because Car 1 and Car 2 share the same household address, territory, and policyholder credit score, the model achieves near-perfect validation accuracy by recognizing household commonalities. In production, single-car policies and new customers perform poorly.