Study guide

5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping

Key Takeaways Explain This Topic AI

High-cardinality categoricals such as ZIP code, vehicle make-model and workers compensation class code explode the parameter count under one-hot encoding and leave most cells with too little exposure to estimate.

Naive in-sample target encoding leaks the target into the predictor: a level observed once is encoded with its own loss, and the model memorises rather than generalises.

Credibility-weighted target encoding shrinks a level's own mean toward the overall mean in proportion to its volume, and out-of-fold computation keeps the encoding out of the records it is applied to.

A GLM assigns full credibility to every categorical level, so thin levels must be grouped by the actuary; the model flags the problem through large standard errors but does not fix it.

The base level should be the largest-exposure level, because every other level's standard error is measured relative to the base and a thin base inflates all of them.

Last updated: September 2026

The High-Cardinality Challenge in P&C Insurance

In property and casualty insurance, some of the most powerful risk discriminators are discrete categorical variables with hundreds, thousands, or tens of thousands of unique levels. These high-cardinality categorical predictors are central to risk classification:

Geographic Classifiers : 5-digit United States ZIP codes contain over 42 , 000 42,000 42 , 000 active levels. In homeowners insurance, geographic location captures critical exposure differentials: distance to coastline (hurricane hazard), brush density (wildfire risk), local fire department responsiveness, and municipal building code enforcement.

Vehicle Classifiers : Vehicle identification numbers (VIN) decode into Make, Model, Sub-model, and Trim, creating 3 , 000 3,000 3 , 000 to 10 , 000 10,000 10 , 000 distinct vehicle classifications. These capture engine performance, advanced driver assistance systems (ADAS), repair parts costs, and crashworthiness.

Commercial Business Classifiers : Workers' compensation class codes, North American Industry Classification System (NAICS), and Standard Industrial Classification (SIC) codes exceed 1 , 000 1,000 1 , 000 categories, differentiating occupational hazards from clerical office work to high-rise structural steel erection.

The Failure of Traditional One-Hot (Dummy) Encoding

Standard statistical modeling encodes categorical predictors by creating K − 1 K - 1 K − 1 binary dummy variables for a factor with K K K levels. When applied to high-cardinality features in GLMs, this approach fails completely: HIGH-CARDINALITY ONE-HOT FAILURE MODES

  • │
  • ┌─────────────────────────────┼─────────────────────────────┐
  • ▼ ▼ ▼
  • Dimensionality Explosion Degrees of Freedom Loss Empty / Sparse Cells
  • 42,000 ZIP codes create Consumes tens of thousands Rural ZIPs with 1-2 policies
  • 41,999 new columns in X; of degrees of freedom; yield infinite or undefined
  • matrix memory exhausts drastically inflates variance MLE coefficients (separation)

Dimensionality Explosion : Encoding 42 , 000 42,000 42 , 000 ZIP codes adds 41 , 999 41,999 41 , 999 columns to the design matrix X \mathbf{X} X . Storing and manipulating this sparse matrix requires massive memory and slows IRLS convergence to a crawl.

Loss of Degrees of Freedom : Estimating tens of thousands of parameters depletes model degrees of freedom ( n − p n - p n − p ), drastically inflating standard errors across all coefficients.

Complete Separation & Extreme Sparse Cells : In casualty portfolios, many levels are sparsely populated. A rural ZIP code might have only 2 2 2 insured policyholders with 0.8 0.8 0.8 earned car-years and 0 0 0 claims. In a Poisson frequency GLM, the maximum likelihood estimate for a category with zero claims diverges to − ∞ -\infty − ∞ : β ^ sparse = ln ⁡ ( 0 exposure ) = − ∞ \hat{\beta}_{\text{sparse}} = \ln\left( \frac{0}{\text{exposure}} \right) = -\infty β ^ ​ sparse ​ = ln ( exposure 0 ​ ) = − ∞

Conversely, if a rare vehicle trim with 1 1 1 exposure suffered a catastrophic total loss, MLE estimates a massive positive coefficient. The model suffers from severe estimation instability.

Target Encoding (Mean Encoding) & The Target Leakage Hazard

To compress high-cardinality categories into a manageable representation, machine learning practitioners frequently employ target encoding (also known as mean encoding or likelihood encoding).

Conceptual Definition

Target encoding replaces each categorical level c ∈ { 1 , 2 , … , K } c \in \{1, 2, \dots, K\} c ∈ { 1 , 2 , … , K } with the expected value of the target response Y Y Y within that level: S ^ c = E [ Y ∣ X = c ] \hat{S}_c = \mathbb{E}[Y \mid X = c] S ^ c ​ = E [ Y ∣ X = c ]

For a frequency model , S ^ c \hat{S}_c S ^ c ​ is the empirical claim frequency in level c c c : S ^ c = ∑ i ∈ c y i ∑ i ∈ c e i \hat{S}_c = \frac{\sum_{i \in c} y_i}{\sum_{i \in c} e_i} S ^ c ​ = ∑ i ∈ c ​ e i ​ ∑ i ∈ c ​ y i ​ ​

For a pure premium / loss cost model , S ^ c \hat{S}_c S ^ c ​ is the average loss per earned exposure in level c c c : S ^ c = ∑ i ∈ c Loss i ∑ i ∈ c e i \hat{S}_c = \frac{\sum_{i \in c} \text{Loss}_i}{\sum_{i \in c} e_i} S ^ c ​ = ∑ i ∈ c ​ e i ​ ∑ i ∈ c ​ Loss i ​ ​

Target encoding replaces a K K K -level categorical variable with a single continuous 1D numeric feature . This preserves monotonic ordering with respect to the loss cost and consumes exactly one degree of freedom in a GLM instead of the K − 1 K-1 K − 1 that dummy coding would require.

The Fatal Flaw: Target Leakage and Noise Memorization

Despite its simplicity, naive target encoding is disastrously flawed because it induces severe target leakage .

If the sample mean S ^ c \hat{S}_c S ^ c ​ is calculated using the target response of the entire training dataset—including observation i i i itself—the model leaks the response into the predictor: S ^ c ( i ) = y i + ∑ j ∈ c , j ≠ i y j n c \hat{S}_{c(i)} = \frac{y_i + \sum_{j \in c, j \ne i} y_j}{n_c} S ^ c ( i ) ​ = n c ​ y i ​ + ∑ j ∈ c , j  = i ​ y j ​ ​

Consider an auto insurance portfolio with a rare vehicle trim code c = Vintage_Trim_42 c = \text{Vintage\_Trim\_42} c = Vintage_Trim_42 containing exactly n c = 1 n_c = 1 n c ​ = 1 policyholder. Suppose this policyholder suffered a severe 80 , 000 c l a i m ( 80,000 claim ( 80 , 000 c l aim ( y_i = 80,000$).

Under naive target encoding, this vehicle trim is assigned an encoded feature value of S ^ c = 80 , 000 \hat{S}_c = 80,000 S ^ c ​ = 80 , 000 .

A downstream GLM or tree ensemble immediately recognizes that whenever Feature = 80 , 000 \text{Feature} = 80,000 Feature = 80 , 000 , the target is $80,000, assigning this single predictor a massive positive weight.

The model appears to achieve near-perfect in-sample fit ( R 2 ≈ 1.0 R^2 \approx 1.0 R 2 ≈ 1.0 , deviance drops drastically). In reality, the algorithm has simply memorized target noise .

When deployed on unseen prospective policyholders, future insureds driving Vintage Trim 42 are hit with absurdly uncompetitive premiums, while the model fails to predict losses on other risks.

Credibility-Weighted Target Encoding & Bühlmann Shrinkage

Decades before modern data science discovered the vulnerability of naive target encoding, casualty actuaries solved the problem of sparse classification cells using Bühlmann credibility theory .

Actuarial Credibility Formulation

To prevent small, sparse categories from memorizing noise, credibility-weighted target encoding shrinks the empirical sample mean of category c c c toward the global portfolio average: S ^ c cred = Z c y ˉ c + ( 1 − Z c ) y ˉ global \hat{S}_c^{\text{cred}} = Z_c \bar{y}_c + (1 - Z_c) \bar{y}_{\text{global}} S ^ c cred ​ = Z c ​ y ˉ ​ c ​ + ( 1 − Z c ​ ) y ˉ ​ global ​

Where:

y ˉ c = ∑ i ∈ c y i ∑ i ∈ c e i \bar{y}_c = \frac{\sum_{i \in c} y_i}{\sum_{i \in c} e_i} y ˉ ​ c ​ = ∑ i ∈ c ​ e i ​ ∑ i ∈ c ​ y i ​ ​ is the empirical target mean of category c c c .

y ˉ global = ∑ i = 1 N y i ∑ i = 1 N e i \bar{y}_{\text{global}} = \frac{\sum_{i=1}^N y_i}{\sum_{i=1}^N e_i} y ˉ ​ global ​ = ∑ i = 1 N ​ e i ​ ∑ i = 1 N ​ y i ​ ​ is the grand portfolio mean across all policies.

Z c ∈ [ 0 , 1 ] Z_c \in [0, 1] Z c ​ ∈ [ 0 , 1 ] is the Bühlmann credibility factor .

The Bühlmann Credibility Factor

In classical Bühlmann credibility, the credibility weight assigned to the category's own experience is defined as: Z c = n c n c + k Z_c = \frac{n_c}{n_c + k} Z c ​ = n c ​ + k n c ​ ​

Where n c n_c n c ​ is the exposure (or policy count) in category c c c , and k k k is the structural ratio of the Expected Value of the Process Variance (EPV) to the Variance of the Hypothetical Means (VHM): k = E [ Var ( Y ∣ Θ ) ] Var ( E [ Y ∣ Θ ] ) = EPV VHM k = \frac{\mathbb{E}[\text{Var}(Y \mid \Theta)]}{\text{Var}(\mathbb{E}[Y \mid \Theta])} = \frac{\text{EPV}}{\text{VHM}} k = Var ( E [ Y ∣ Θ ]) E [ Var ( Y ∣ Θ )] ​ = VHM EPV ​ BÜHLMANN CREDIBILITY SHRINKAGE SPECTRUM

  • Sparse Category (n_c << k) Dense Category (n_c >> k)
  • ────────────────────────── ─────────────────────────
  • Z_c ≈ 0 Z_c ≈ 1
  • Encoded Value -> Global Mean Encoded Value -> Category Mean
  • Safe, robust, prevents noise Reflects distinct credibility
  • 0.0 ├───────────────────────●───────────────────────┤ 1.0
  • Z_c = 0 Z_c = 1
  • (Pure Global Mean) (Pure Category Mean)

For Sparse Categories ( n c → 0 n_c \to 0 n c ​ → 0 ) : As n c ≪ k n_c \ll k n c ​ ≪ k , Z c → 0 Z_c \to 0 Z c ​ → 0 . The category's sparse, volatile empirical mean is discarded, and its encoded value reverts almost entirely to the reliable portfolio baseline y ˉ global \bar{y}_{\text{global}} y ˉ ​ global ​ . An isolated $80,000 claim in a 1-policy ZIP code is shrunk heavily back to the state average, preventing rate distortion.

For Dense Categories ( n c → ∞ n_c \to \infty n c ​ → ∞ ) : As n c ≫ k n_c \gg k n c ​ ≫ k , Z c → 1 Z_c \to 1 Z c ​ → 1 . The category possesses full credibility, and its encoded value reflects its own robust empirical loss experience.

Out-of-Fold (OOF) Target Encoding Pipeline

Even with credibility shrinkage, target leakage can still occur if an observation's own target is included in y ˉ c \bar{y}_c y ˉ ​ c ​ . To achieve complete leakage isolation, actuaries combine credibility shrinkage with an Out-of-Fold (OOF) calculation architecture: OUT-OF-FOLD (OOF) ENCODING PIPELINE

  • Training Data (Partitioned into M = 5 Folds)
  • ┌──────────┬──────────┬──────────┬──────────┬──────────┐
  • │ Fold 1 │ Fold 2 │ Fold 3 │ Fold 4 │ Fold 5 │
  • └────┬─────┴──────────┴──────────┴──────────┴──────────┘
  • │
  • ▼

To encode Fold 1:

  • 1. Compute category means & Bühlmann Z using strictly Folds 2, 3, 4, 5
  • 2. Apply credibility formula: S_c = Z·ȳ_c + (1-Z)·ȳ_global
  • 3. Map values onto Fold 1 observations

4. Observation i never sees its own y_i in its feature value!

Partition the training dataset into M M M disjoint folds (e.g., M = 5 M = 5 M = 5 or 10 10 10 ).

For each fold m ∈ { 1 , 2 , … , M } m \in \{1, 2, \dots, M\} m ∈ { 1 , 2 , … , M } :

Exclude fold m m m .

Compute category sample means y ˉ c ( − m ) \bar{y}_{c}^{(-m)} y ˉ ​ c ( − m ) ​ , exposures n c ( − m ) n_c^{(-m)} n c ( − m ) ​ , and the grand mean y ˉ global ( − m ) \bar{y}_{\text{global}}^{(-m)} y ˉ ​ global ( − m ) ​ strictly from the remaining M − 1 M - 1 M − 1 folds.

Calculate credibility-weighted values S ^ c cred , ( − m ) \hat{S}_c^{\text{cred}, (-m)} S ^ c cred , ( − m ) ​ using only out-of-fold data.

Map these values onto the observations in fold m m m .

For scoring prospective validation/test data or production policies, compute the credibility-weighted target encodings using the entire training dataset .

Additive Smoothing & Jitter

To provide an extra layer of regularization in machine learning trees, practitioners inject small Gaussian noise (jitter) into the encoded feature during training: S ~ c = S ^ c cred + ε \tilde{S}_c = \hat{S}_c^{\text{cred}} + \varepsilon S ~ c ​ = S ^ c cred ​ + ε , where ε ∼ N ( 0 , σ noise 2 ) \varepsilon \sim \mathcal{N}(0, \sigma^2_{\text{noise}}) ε ∼ N ( 0 , σ noise 2 ​ ) . This prevents decision trees from performing exact threshold splits on discrete credibility values.

Choosing the Base Level

Every categorical variable in a GLM is fitted relative to a base level , whose relativity is fixed at 1.000 and whose effect is absorbed into the intercept. The choice does not change the model's predictions, but it changes every reported coefficient and every standard error, so it is a presentation decision with real consequences.

Choose the base level with the largest exposure . Two reasons:

Standard errors are measured against the base. Each non-base coefficient's standard error reflects uncertainty in both that level and the base. A base level with thin data inflates the standard errors of every other level, making the whole variable look insignificant.

Readers anchor on the base. A relativity table whose 1.000 sits on a level containing 2% of exposure gives the reader no intuition for what "average" means.

  • Base level chosen Effect
  • Largest-exposure level Tight standard errors, intuitive relativities
  • Alphabetically first level Arbitrary; often thin, inflating every standard error
  • Smallest level Worst case: unstable base, every relativity unreliable
  • Grouping Levels Before Fitting

A GLM assigns full credibility to every level of a categorical variable. The estimate for a territory with 8 claims is simply that territory's own experience, with no shrinkage toward the mean. The model warns you — the standard error will be large and the p-value high — but it does nothing about it.

That makes grouping a modelling decision the actuary must make, not one the software makes:

Group by similarity of indicated effect , not alphabetically or geographically, and check that the grouping holds up out of sample.

Set an exposure or claim-count floor below which a level cannot carry its own coefficient. State the floor in the report.

Use a residual "other" level for the genuinely small remainder, and check that it is not absorbing levels with materially different experience.

Re-check after grouping : deviance should barely rise while the number of parameters falls sharply.

Comparing the Options

  • Approach Degrees of freedom Leakage risk Best for
  • One-hot / dummy coding K - 1 None Low-cardinality variables with credible levels
  • Grouped levels G - 1, with G much smaller than K None Moderate cardinality with thin tails
  • Naive target encoding 1 Severe Never, in-sample

Credibility-weighted, out-of-fold target encoding 1 Controlled High-cardinality codes such as ZIP or class code

For a PCPA project, grouping plus a clearly stated exposure floor is usually the right answer: it is defensible in a sentence, it satisfies the rubric criterion on variable treatment, and it avoids the leakage traps that target encoding introduces.