Study guide

3.3 Outlier Diagnostics, Trimming & Winsorization

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

3.3 Outlier Diagnostics, Trimming & Winsorization

Key Takeaways Explain This Topic AI

In P&C insurance analytics, outliers must be rigorously bifurcated into operational data entry errors (which must be cleaned or removed) and legitimate heavy-tailed shock losses (which must be preserved).

Trimming extreme claim losses from pricing datasets introduces severe downward bias into pure premium calculations, leading to chronically inadequate rates and under-reserving.

Actuarial capping (Winsorization) clamps individual claim severities to a basic limit threshold, stabilizing GLM dispersion parameters and parameter estimation while preserving claim frequency.

Any loss cost reduction achieved through severity capping must be actuarially restored through an explicit excess loss loading (e.g., via Increased Limits Factors or aggregate excess pure premiums).

Robust diagnostic techniques like the Modified Z-score (based on Median Absolute Deviation) and Isolation Forests resist the masking and swamping phenomena that undermine standard Z-scores and linear fences.

Last updated: September 2026

1.2 Outlier Diagnostics, Trimming & Winsorization

Exam Focus: P&C insurance data is inherently heavy-tailed. A single $10 million commercial liability judgment or wildfire total loss can easily distort Generalized Linear Model parameter estimation. The CAS PCPA exam rigorously tests candidates on distinguishing operational data corruption from legitimate extreme losses, deploying robust anomaly diagnostics (such as MAD and Isolation Forests), and mastering the mathematical interplay between basic limits capping, Winsorization, and excess loss loadings.

Outliers in actuarial science present a fundamental operational paradox. In classical statistics, outliers are frequently viewed as contaminated observations to be detected, discarded, or minimized. In property and casualty insurance, however, extreme loss events represent the very essence of the risk transfer mechanism. Eliminating legitimate shock losses distorts tail severity distributions, underestimates expected losses, and produces rates that are inadequate under state statutory standards.

1. Outlier Diagnostics & Detection Methodologies

Actuaries evaluate anomalous data points across both predictor variables (policy characteristics, exposure measures) and target response variables (claim counts, claim dollar amounts). ┌───────────────────────────────────┐

  • │ Outlier Detection Diagnostics │
  • └───────────────────────────────────┘
  • │
  • ┌──────────────────────────┬───────────────┴───────────────┬──────────────────────────┐
  • ▼ ▼ ▼ ▼
  • ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
  • │ IQR Fences │ │ Z-Score │ │ Modified Z │ │ Isolation │
  • │ (Tukey) │ │ (Standard) │ │ (MAD) │ │ Forest │
  • └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
  • │ │ │ │
  • ▼ ▼ ▼ ▼
  • Non-parametric: Parametric: Robust to 50% Unsupervised tree
  • [Q1-1.5·IQR, Q3+1.5·IQR]. Zi = (xi - x̄) / s. contamination: ensemble. Isolates
  • Flags natural right- Distorted by masking Mi = 0.6745(xi-x̃)/MAD. anomalies via shorter

skewed insurance tails. and swamping effects. Standard for P&C. path lengths in space.

Interquartile Range (IQR) Rule (Tukey's Fences)

Tukey's method establishes non-parametric boundaries based on the empirical distribution's quartiles:

  • Interquartile Range: IQR = Q₃ - Q₁
  • Inner Fences (Mild Outliers): [Q₁ - 1.5 · IQR, Q₃ + 1.5 · IQR]
  • Outer Fences (Extreme Outliers): [Q₁ - 3.0 · IQR, Q₃ + 3.0 · IQR]

Actuarial Critique:

While robust to non-normal distributions, Tukey's fences are symmetric. P&C severity distributions (e.g., Log-Normal, Gamma, Pareto) and exposure metrics (e.g., Total Insured Value) are heavily right-skewed. Applying a standard 1.5 · IQR rule to commercial property building values will erroneously flag the upper 5% to 10% of legitimate large buildings as outliers, producing excessive false alarms.

Standard Z-Score and the Masking/Swamping Trap

The classical Z-score standardizes observations relative to the sample mean x̄ and sample standard deviation s:

Z_i = (x_i - x̄) / s

Observations with |Z_i| > 3.0 are flagged as anomalies. In insurance data, the standard Z-score suffers from two fatal mathematical vulnerabilities:

Masking Effect: A cluster of several massive losses (e.g., three separate $5 million bodily injury claims) simultaneously inflates the sample mean x̄ and dramatically inflates the sample standard deviation s. As a result, other truly anomalous losses have their calculated Z-scores pulled downward below 3.0, remaining masked and undetected.

Swamping Effect: A single colossal outlier inflates x̄ to such an extent that completely normal, uncorrupted records residing in the opposite tail are falsely flagged as outliers.

Modified Z-Score using Median Absolute Deviation (MAD)

To overcome the breakdown of sample means and variances, actuaries utilize Boris Iglewicz and David Hoaglin's (1993) Modified Z-score, which substitutes the median x̃ and Median Absolute Deviation (MAD):

MAD = median(|x_i - x̃|)

The Modified Z-score M_i is formulated as:

M_i = [0.6745 · (x_i - x̃)] / MAD

Mathematical Derivation of the 0.6745 Constant:

For a continuous standard normal distribution Φ(z), the median is 0, and the expected value of MAD corresponds to the 75th percentile of the distribution:

P(|Z| ≤ MAD) = 0.50 ==> Φ(MAD) - Φ(-MAD) = 0.50 ==> Φ(MAD) = 0.75

Because Φ⁻¹(0.75) ≈ 0.6744898, multiplying by 0.6745 ensures that MAD / 0.6745 serves as an asymptotically unbiased estimator of the population standard deviation σ when the core distribution is Gaussian:

σ̂_robust = MAD / 0.6745 ≈ 1.4826 · MAD

Breakdown Point: The Modified Z-score possesses a breakdown point of 50% . Up to half of the dataset can be extreme outliers without corrupting the scale estimator.

Decision Threshold: Observations with |M_i| > 3.5 are flagged as candidate outliers for actuarial review.

Mahalanobis Distance for Multivariate Correlated Features

Univariate screening cannot detect multidimensional inconsistencies where individual feature values are plausible, but their joint combination is impossible. For a p-dimensional feature vector x_i, Mahalanobis distance measures the distance from the multivariate centroid μ, accounting for the feature covariance matrix Σ:

D_M(x_i) = sqrt[(x_i - μ)ᵀ · Σ⁻¹ · (x_i - μ)]

Distributional Evaluation: Under multivariate normality, [D_M(x_i)]² ~ χ²(p). A record is flagged if [D_M(x_i)]² > χ²_p,0.999.

P&C Example: An auto insurance record with Driver_Age = 18 and Years_Licensed = 12 . Individually, 18 is a valid age and 12 is a valid licensing tenure. However, their multivariate combination generates a massive Mahalanobis distance, immediately exposing the record as an operational error.

Isolation Forests for Complex Insurance Anomalies

Isolation Forest (Liu, Ting, and Zhou, 2008) is an unsupervised ensemble algorithm based on decision trees that isolates anomalies rather than profiling normal data points.

Theoretical Foundations:

Splitting Mechanism: The algorithm randomly selects a feature and randomly selects a split value between the feature's minimum and maximum values.

Path Length Logic: Anomalous observations (e.g., severe multi-vehicle commercial liability claims with high litigation expenses) reside in sparse, low-density regions of feature space. Consequently, they require significantly fewer random splits to isolate into terminal leaf nodes than normal observations. Normal Policyholder (Deep Path) Anomalous Claim (Short Path)

  • [Root Node] [Root Node]
  • / | / |
  • Node Node [Leaf] Node
  • / | / | (h = 1) / |
  • Node Node Node Node Node Node
  • / | / |
  • [Leaf] [Leaf] Node [Leaf]
  • (h = 4) (h = 4)

The anomaly score s(x, n) for an observation x across an ensemble of trees on a dataset of size n is:

s(x, n) = 2^[-E(h(x)) / c(n)]

Where h(x) is the path length (number of edges traversed from root to leaf), E(h(x)) is the average path length across all isolation trees, and c(n) is the average path length of unsuccessful searches in a Binary Search Tree (BST):

c(n) = 2 · [ln(n - 1) + 0.5772156649] - [2 · (n - 1) / n]

If E(h(x)) -> 0, s -> 1: The record isolates instantaneously, indicating a definite anomaly.

If E(h(x)) -> c(n), s -> 0.5: The record exhibits structural characteristics typical of normal policies.

If E(h(x)) -> n - 1, s -> 0: The record requires maximum partitioning, indicating high centrality.

2. Distinguishing Operational Errors from Legitimate Heavy Tails

Before modifying or removing any flagged outlier, the actuary must categorize the observation into one of two fundamentally distinct categories: ┌─────────────────────────────────────────────────────────┐

  • │ Taxonomy of Insurance Outliers │
  • └─────────────────────────────────────────────────────────┘
  • │
  • ┌────────────────────────────────────────┴────────────────────────────────────────┐
  • ▼ ▼
  • ┌──────────────┐ ┌──────────────┐
  • │ Data Errors │ │ Shock Losses │
  • └──────┬───────┘ └──────┬───────┘
  • │ │
  • ▼ ▼
  • Impossible driver age (e.g. 2 or 999) • $8M commercial liability verdict
  • Negative payroll in Workers Comp • Total loss industrial fire
  • Invalid VIN checksums • Catastrophic hail/windstorm
  • Expired policy date before effective date • Structural bodily injury claim
  • │ │
  • ▼ ▼
  • Remediation: Programmatic cleaning, Remediation: NEVER delete. Cap at

field validation, or record exclusion. basic limits & add excess load.

Diagnostic Dimension Operational / Data Entry Errors Legitimate Heavy-Tailed Losses

Root Cause Broken ETL pipelines, agent typos, software glitches, missing default values Inherent stochastic volatility of property & casualty perils (Pareto/Log-Normal tails)

Examples Driver_Age = 3 , Vehicle_Value = $99,999,999 , negative workers comp payroll $5,000,000 commercial auto truck collision involving multiple fatalities

Verification Fails external validation checks (VIN decoders, postal databases, tax filings) Supported by documented adjuster reserve logs, legal settlement filings, and police reports

Modeling Action Correct via business rules, impute, or drop record from model training Never drop. Winsorize/cap at policy basic limit and explicitly load excess pure premium

3. Remediation Methods: Trimming vs. Winsorization (Capping)

Once candidate anomalies are diagnosed, the actuary must implement an appropriate remediation framework for model calibration.

Trimming (Listwise Filtering)

Trimming involves identifying records with values exceeding an upper threshold T and permanently removing those records from the modeling dataset:

D_trimmed = {(x_i, y_i) ∈ D | y_i ≤ T}

The Catastrophic Actuarial Trap of Trimming Losses:

Trimming claim dollar amounts is considered an actuarial error in rate filing. If an actuary removes all claims exceeding $250,000 when fitting a personal auto bodily injury severity model, the empirical severity mean drops precipitously:

E[Y | Y ≤ T] = [∫₀ᵀ y · f(y) dy] / F(T) < E[Y] = ∫₀^∞ y · f(y) dy

Calibrating rates to E[Y | Y ≤ T] produces baseline premiums that are systematically deficient. The insurer will fail to collect adequate premium to pay for legitimate large claims, resulting in statutory underwriting losses and reserve deficiencies.

Winsorization (Actuarial Capping)

Named after Charles Winsor, Winsorization clamps extreme values at a predetermined percentile threshold (such as the 99th or 99.5th percentile) or a standard contractual limit C:

  • Y_i,capped = min(Y_i, C) Raw Loss Distribution Winsorized (Capped) Distribution
  • f(y) f(y)
  • │ * │ *
  • │ * * │ * * Probability Mass
  • │ * * │ * * Spike at C
  • │ * * │ * * │
  • │ * * │ * * ▼
  • │* * * * * * (Long Tail) │* * * * * * ███
  • └──────────────────────────────► y └───────────────────────────┬────────► y
  • │
  • Capped
  • Limit (C)

Why Capping Preserves Actuarial Integrity:

Retains Exposure & Frequency: Unlike trimming, capping retains every single policy and claim occurrence in the dataset. The frequency model (N ~ Poisson) is completely preserved.

Stabilizes GLM Dispersion: Gamma and Inverse Gaussian severity GLMs rely on variance functions V(μ) = μ² and V(μ) = μ³. A single uncapped $10 million claim generates a massive unit deviance residual that destabilizes the Fisher scoring optimization and warps rating relativities for unrelated classes. Capping stabilizes parameter estimation.

4. Actuarial Implications: Basic Limits & Excess Loss Loading

In property and casualty ratemaking, rates are mathematically decoupled into two layers: Basic Limits Pure Premium and Excess Limits Loading .

The Fundamental Decomposition of Expected Loss

Let Y denote the uncapped claim severity random variable, and let C denote the basic limit threshold (e.g., $100,000 for personal auto bodily injury or $500,000 for commercial liability). The total expected loss decomposes mathematically into:

  • E[Y] = E[min(Y, C)] + E[max(0, Y - C)]
  • Total Pure Premium = Basic Limits Pure Premium + Excess Pure Premium
  • Pure Premium = [(E[N] / Exposure) · E[min(Y, C)]] + Excess Loss Cost Total Loss Layer
  • ┌────────────────────────────────────────────────────────────────────────┐
  • │ Excess Layer: E[max(0, Y - C)] │ ◄── Loaded via ILFs or
  • │ (Catastrophic claims, shock losses exceeding threshold C) │ aggregate excess pure premium
  • ├────────────────────────────────────────────────────────────────────────┤ ◄── Capping Threshold C
  • │ Basic Limits Layer: E[min(Y, C)] │ ◄── Modeled via GLM
  • │ (Stable, high-frequency, credible classification risk) │ (Gamma / Tweedie)
  • └────────────────────────────────────────────────────────────────────────┘

Modeling Workflow for Ratemaking:

Cap Losses at Limit C: Replace raw incurred claim amounts with Y_capped = min(Y, C).

Fit GLM to Capped Losses: Train a Gamma severity GLM or Tweedie pure premium model using Y_capped to derive refined rating relativities across rating factors (age, vehicle symbol, territory, credit tier).

Calculate Increased Limits Factors (ILFs): For policyholders purchasing coverage limits L > C, the rate is adjusted via an Increased Limits Factor derived from aggregate, line-wide loss data:

ILF(L) = E[min(Y, L)] / E[min(Y, C)]

Add Excess Loss Load: For basic limit policies, add an explicit statewide or line-wide excess loss loading factor to cover expected aggregate losses above C:

  • Excess Ratio R(C) = E[max(0, Y - C)] / E[Y] = 1 - [E[min(Y, C)] / E[Y]]
  • Frequency vs. Severity Disconnect in Outlier Handling

Actuaries must maintain strict separation between frequency and severity when managing outliers:

Severity Models: Hyper-sensitive to extreme claim dollar values. Capping at the 99th or 99.5th percentile is mandatory to prevent deviance explosion.

Frequency Models: Dependent exclusively on claim counts per unit of exposure (N_i / E_i). Large dollar amounts do not affect frequency. However, exposure outliers (e.g., an operational error recording 10,000 car-years for an individual 6-month auto policy) will severely distort frequency Poisson GLMs and must be corrected or trimmed.