3.4 Variable Transformation & Scaling

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

3.4 Variable Transformation & Scaling

Key Takeaways Explain This Topic AI

Power transformations like Box-Cox and Yeo-Johnson stabilize heteroscedastic error variance and linearize non-linear exposure relationships in P&C predictive models.

Yeo-Johnson transformation is mandatory when transforming variables that include zero or negative values, such as commercial net profit change or prior experience rating modification credits.

Feature scaling (Min-Max, Standardization, Robust scaling) is essential for distance-based algorithms and regularized GLMs (LASSO/Ridge), but has zero impact on decision tree splits or unpenalized GLM relativities.

Equal-frequency binning ensures actuarial credibility across categories, whereas decision-tree-supervised binning optimizes risk classification boundaries against deviance loss.

Rating step functions create cliff-edge pricing distortions; replacing them with natural cubic splines or shape-constrained GAMs produces smooth, monotonically compliant rating curves required by state insurance regulators.

Last updated: September 2026

1.3 Feature Transformations & Scaling

Exam Focus: Actuarial rating variables rarely enter predictive models in raw linear format. Skewed exposures, high-cardinality predictors, non-linear age curves, and regulatory mandates require sophisticated feature engineering. The CAS PCPA exam requires complete mastery of power transformations (Box-Cox vs. Yeo-Johnson), mathematical sensitivity to feature scaling across distinct algorithms (LASSO vs. tree ensembles vs. GLMs), optimal supervised binning, and spline basis functions that enforce monotonic risk relativities.

In property and casualty insurance, raw features exhibit non-linearities, heavy right-skewness, and severe exposure clustering. For instance, commercial property replacement values span several orders of magnitude, driver crash risk surges for teenagers before leveling off in adulthood, and commercial liability financial metrics can be zero or negative. Preparing these features for a Generalized Linear Model demands deliberate mathematical transformation of the predictors.

1. Power Transformations for Skewed Actuarial Features

Many statistical modeling techniques assume that continuous predictors relate linearly to the link-transformed response and possess homoscedastic variance. When raw features violate these assumptions, power transformations stabilize variance and improve model fit. ┌────────────────────────────────────────────────────────┐

│ Power Transformations │
└────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────┼─────────────────────────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Natural │ │ Box-Cox │ │ Yeo-Johnson │
│ Logarithm │ │ (Strictly >0)│ │ (All Reals) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
Compresses monetary values: Parametric family with Accommodates zero and
y = ln(x) for x > 0. optimal λ via MLE: negative insurance
Linearizes multiplicative y^(λ) = (y^λ - 1) / λ. covariates (e.g. net

relationships in log-link GLMs. Fails if y ≤ 0. profit, experience mod).

The Natural Logarithm

The natural log transformation y = ln(x) (for x > 0) is the most ubiquitous actuarial transformation. It is standard for monetary variables (Total Insured Value, annual payroll, vehicle replacement cost) and exposure durations.

Multiplicative Interpretation: In a log-link GLM (ln(μ) = β₀ + β₁ · ln(X)), exponentiating yields μ = exp(β₀) · X^(β₁). The coefficient β₁ represents the elasticity of the expected loss cost with respect to X. If β₁ = 1, the loss cost scales in direct, 1-to-1 proportion with exposure (which is why exposure enters frequency GLMs with an offset constraint β = 1).

Handling Zeroes: When features contain zeroes (e.g., prior claim count, number of minor speeding convictions), analysts sometimes apply ln(x + c), where c is a small arbitrary constant (e.g., c = 1). However, the choice of c artificially warps the shape of the curve near zero, distorting rating relativities for preferred zero-violation risks.

Box-Cox Transformation

George Box and David Cox (1964) introduced a generalized parametric family of power transformations designed to transform continuous data into approximately normal distributions:

y^(λ) = (y^λ - 1) / λ if λ ≠ 0, and ln(y) if λ = 0

Mathematical Constraints & Maximum Likelihood Estimation:

Positivity Requirement: The classical Box-Cox transformation is strictly defined only for strictly positive variables (y > 0). It cannot accommodate zero or negative values.

Optimal Parameter Estimation: The transformation parameter λ is estimated simultaneously via Maximum Likelihood Estimation (MLE) on the model residuals. The profile log-likelihood function maximized over λ is:

L(λ) = -(n / 2) · ln(σ̂²(λ)) + (λ - 1) ∑ ln(y_i)

Where σ̂²(λ) = (1 / n) ∑ [y_i^(λ) - ȳ^(λ)]².

λ Value Mathematical Transformation Common Actuarial Application
λ = 1.0 No transformation (y - 1) Symmetric, unskewed continuous variables
λ = 0.5 Square root: 2 · (sqrt(y) - 1) Poisson-like counts and dispersion stabilization
λ = 0.0 Natural logarithm: ln(y) Monetary values, building TIV, commercial revenues
λ = -0.5 Inverse square root Heavy-tailed waiting times and claim closure durations
λ = -1.0 Reciprocal: 1 - 1/y Speed or velocity metrics, operational throughput
Yeo-Johnson Transformation

To resolve Box-Cox's inability to handle non-positive values, In-Kwon Yeo and Richard Johnson (2000) formulated an extended power transformation supporting the entire real line (y ∈ ℝ):

If λ ≠ 0 and y ≥ 0: ψ(λ, y) = [(y + 1)^λ - 1] / λ
If λ = 0 and y ≥ 0: ψ(λ, y) = ln(y + 1)
If λ ≠ 2 and y < 0: ψ(λ, y) = -[(-y + 1)^(2 - λ) - 1] / (2 - λ)
If λ = 2 and y < 0: ψ(λ, y) = -ln(-y + 1)

Why Yeo-Johnson is Essential for P&C Commercial Lines:

Commercial underwriting datasets frequently contain legitimate zero and negative financial metrics:

Net Annual Operating Profit / Cash Flow: Used as an underwriting solvency predictor in commercial surety and directors & officers (D&O) liability.

Prior Year Experience Rating Modification Change: ΔMod = Mod_t - Mod_(t-1), where negative values indicate improving safety records.

Fleet Size Net Growth: Commercial auto fleets expanding (positive) or contracting (negative).

Yeo-Johnson seamlessly normalizes these mixed-sign features without requiring artificial offset constants.

2. Feature Scaling & Algorithmic Sensitivity

Feature scaling rescales independent variables to a common numerical range. While critical for some algorithms, it is entirely irrelevant for others. ┌───────────────────────────────────┐

│ Scaling Methods Comparison │
└───────────────────────────────────┘
│
┌──────────────────────────────────────────┼──────────────────────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Min-Max │ │Standardizati.│ │Robust Scaler │
│ Normalizati. │ │ (Z-Score) │ │ (IQR) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
x' = (x - xmin) / x' = (x - μ) / σ x' = (x - x̃) / IQR
(xmax - xmin) │ │
│ ▼ ▼
Maps to [0, 1]. Highly Mean = 0, StDev = 1. Resistant to extreme
sensitive to extreme Standard choice for P&C loss and exposure

outliers (tail squishing). regularized models. outliers in predictors.

Scaling Methods Formulation:

Min-Max Normalization: x' = (x - x_min) / (x_max - x_min) ∈ [0, 1] Vulnerability: Highly sensitive to outliers. A single massive commercial payroll will compress 99% of policyholders into a tight range between 0.00 and 0.02.

Standardization (Z-Score Scaling): x' = (x - μ) / σ (with mean 0, unit variance)
Robust Scaling: x' = (x - median(x)) / IQR(x)

Centers at median and scales by IQR, remaining completely impervious to outlier corruption.

Algorithmic Sensitivity Matrix

Model Architecture Scaling Required? Mathematical Theoretical Rationale

K-Nearest Neighbors (KNN) Mandatory Distance metric d(u,v) = sqrt[∑ (u_j - v_j)²] is dominated by unscaled features with large ranges (e.g., building TIV in millions vs. driver age in decades).

Shrinkage / penalized fits Mandatory A penalty applied uniformly to coefficient magnitudes is not scale-invariant: a predictor measured in dollars carries a tiny coefficient and escapes the penalty, while one measured in units carries a large coefficient and is shrunk.

Unpenalized GLM (Poisson, Gamma, Tweedie) Irrelevant In standard maximum likelihood estimation, scaling a predictor X_j by constant c simply scales the resulting MLE coefficient to β̂_j / c. The fitted linear predictor η = Xβ and predicted loss costs remain mathematically identical.

Reported relativities Presentation only Scaling does not change predictions, but it changes what "a one-unit increase" means. Express continuous predictors in units a reader recognises — TIV in $100,000s, payroll in $1,000s — so the exponentiated coefficient is a meaningful relativity.

3. Discretization & Binning of Continuous Actuarial Predictors

Actuaries frequently discretize continuous variables into discrete intervals (bins) to reflect rating territories, driver age brackets, or vehicle symbol groupings. ┌─────────────────────────────────────┐

│ Binning Methodologies │
└─────────────────────────────────────┘
│
┌──────────────────────────────────────────┼──────────────────────────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Equal-Width │ │Equal-Freq. │ │ Supervised │
│ Binning │ │ (Quantiles) │ │ (Tree-Based) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
Interval width Δ = (xmax-xmin)/k. Each bin contains N/k records. Decision tree optimizes
Creates sparse or empty Ensures equal credibility, splits based on Poisson/
tail buckets in skewed P&C but clumps distinct high- Gamma deviance while enforcing

exposure distributions. hazard risks into single bin. minimum exposure constraints.

1. Equal-Width Binning

Divides the range [x_min, x_max] into k intervals of identical width Δ = (x_max - x_min) / k.

Actuarial Flaw: Because insurance data is skewed, the vast majority of policyholders fall into the first two bins, while upper bins have near-zero exposure, creating credibility deficiencies and volatile rating relativities.

2. Equal-Frequency (Quantile) Binning

Sorts data and establishes cutoffs such that each bin contains exactly 1/k of the total exposure or policy count (e.g., deciles or ventiles).

Advantage: Guarantees sufficient earned car-years or payroll in every bucket, ensuring that credibility criteria (e.g., Classical or Bühlmann credibility) are satisfied.

Disadvantage: Bins in the extreme tails span massive numerical ranges, masking distinct high-hazard risk profiles.

3. Supervised Binning via Decision Trees

Uses a shallow single-variable decision tree (e.g., CART) with a Poisson or Gamma deviance splitting criterion to identify optimal cutoffs.

Algorithm: The tree iteratively splits the feature to maximize the reduction in Poisson deviance:

ΔD = 2 ∑ [y_i · ln(y_i / μ̂_parent) - (y_i - μ̂_parent)] - ∑ {k ∈ {L, R}} 2 ∑ {i ∈ k} [y_i · ln(y_i / μ̂_k) - (y_i - μ̂_k)]

Constraints: The actuary enforces a strict minimum exposure constraint per leaf node (e.g., minimum 1,000 earned car-years), ensuring every discovered bin is statistically credible and actuarially defensible.

4. Rating Step Functions, Splines & Monotonic Regulatory Curves

Traditional insurance rating manuals employ discrete step functions for continuous rating variables. For instance, driver age is traditionally banded into rating classes (16–20, 21–24, 25–29, 30–64, 65+). Rating Relativities: Step Functions vs. Smooth Monotonic Splines

Relativity
│
2.5 ┼───█████████
│ (Age 16-20)
2.0 ┼ █████████ ◄── Cliff-Edge Discontinuity
│ (Age 21-24) (Rate drops 20% on 21st birthday)
1.5 ┼ █████████
│ (Age 25-29) • • • • • • • Smooth Spline Curve
1.0 ┼───────────────────────────────────────────────────────── (Monotonic GAM)
└──────┬────────────┬──────────────┬──────────────┬──────► Driver Age
16 21 25 30
The Problem of "Cliff-Edge" Step Functions

Notch Effects: A step function creates artificial rate discontinuities. An insured turning 25 experiences an immediate 20% premium reduction on their birthday, even though their underlying driving maturity evolved continuously.

Adverse Selection and Gaming: Policyholders and independent agents exploit arbitrary boundaries (e.g., misreporting driving experience or garaging zip codes to fall just inside a cheaper rating bucket).

Natural Cubic Splines and B-Splines in Actuarial GLMs

To eliminate cliff edges, modern actuarial ratemaking replaces step functions with continuous spline basis functions within Generalized Additive Models (GAMs):

ln(μ) = β₀ + f(Age) + ∑ β_j · Z_j

Where f(X) is represented as a linear combination of spline basis functions b_k(X):

f(X) = ∑ γ_k · b_k(X)

Natural Cubic Splines: Cubic polynomials between internal knots k₁, k₂, ..., k_m, constrained to be strictly linear beyond the boundary knots (k₁ and k_m). This linearity constraint prevents erratic polynomial oscillations (Runge's phenomenon) in the sparse data tails.

Continuous Derivatives: Natural cubic splines possess continuous first and second derivatives (C² continuity), guaranteeing completely smooth rate transitions across all ages.

Regulatory Defensibility and Monotonicity Constraints

State insurance regulators (such as California CDI, Texas TDI, and Florida OIR) review rating algorithms under statutory standards requiring that rates be neither excessive, inadequate, nor unfairly discriminatory. A common cause of regulatory rate filing objection is non-monotonicity caused by data noise .

The Regulatory Objection:

Suppose an unconstrained spline or unregularized high-degree polynomial fits the raw empirical loss cost for driver age. Due to local sample noise, the model might estimate that 23-year-olds have higher claim frequency than 22-year-olds, but lower than 24-year-olds.

Regulators will reject this filing as actuarially unsound and unfairly discriminatory because there is no causal or actuarial justification for 23-year-olds being riskier than 22-year-olds.

The Solution: Monotonic P-Splines and Shape-Constrained GAMs (SC-GAMs)

Actuaries enforce mathematical shape constraints directly into the spline optimization:

d f(X) / dX ≤ 0 for all X ∈ [16, 65]

By constraining the derivative of the driver age spline to be strictly non-positive (f'(x) ≤ 0), the model guarantees a monotonically decreasing risk curve from teenage years through maturity. This simultaneously satisfies actuarial soundness, eliminates cliff edges, and complies with state insurance department filing standards.