Study guide

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts

Key Takeaways Explain This Topic AI

Discrimination metrics measure whether a model ranks risks correctly; calibration measures whether the predicted dollar amounts are right, and a model can pass one while failing the other.

The portfolio Actual-to-Expected ratio, sum(actual losses) / sum(predicted losses), shows overall rate balance but can hide large offsetting errors between segments.

Plots of actual against predicted must be built on holdout data, aggregated into equal-weight buckets, and drawn on a log scale, because loss data spans orders of magnitude.

Points crossing the 45-degree line — high at the left and low at the right — mean predictions are too compressed, so the rating plan under-segments even when its overall level is correct.

A loss ratio chart sorts risks by predicted loss ratio and plots actual loss ratio by bucket; a perfect current rating plan would produce a flat line, so any spread is evidence of mispricing the model has found.

Last updated: September 2026

Calibration: The Complement to Discrimination

While discrimination metrics (such as the Gini coefficient and ROC AUC covered in Section 7.4) determine how effectively a model separates low-risk policies from high-risk policies, calibration metrics evaluate whether the predicted dollar amounts equal true expected loss costs .

In casualty actuarial practice, an uncalibrated model is dangerous:

If rates are set according to an under-predicting model (Actual-to-Expected ratio > 1.0 > 1.0 > 1.0 ), collected premiums will be inadequate to cover incurred claims and expenses, resulting in underwriting losses, combined ratio deterioration, and regulatory solvency intervention.

If rates are set according to an over-predicting model (Actual-to-Expected ratio < 1.0 < 1.0 < 1.0 ), premiums will be commercially uncompetitive, causing policyholder defection, declining renewal retention, and adverse selection.

Rigorous model validation requires testing calibration at two distinct levels: global calibration across the total portfolio and marginal calibration across individual rating dimensions.

Global Calibration: The Actual-to-Expected (A/E) Ratio

The most fundamental calibration metric is the portfolio-wide Actual-to-Expected (A/E) ratio (also termed the Actual-over-Expected loss ratio): A/E global = ∑ i = 1 n y i ∑ i = 1 n y ^ i = ∑ i = 1 n y i ∑ i = 1 n e i μ ^ i \text{A/E}_{\text{global}} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n \hat{y}_i} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n e_i \hat{\mu}_i} A/E global ​ = ∑ i = 1 n ​ y ^ ​ i ​ ∑ i = 1 n ​ y i ​ ​ = ∑ i = 1 n ​ e i ​ μ ^ ​ i ​ ∑ i = 1 n ​ y i ​ ​

Where:

y i y_i y i ​ is the actual observed aggregate loss dollars incurred by policy i i i .

y ^ i = e i μ ^ i \hat{y}_i = e_i \hat{\mu}_i y ^ ​ i ​ = e i ​ μ ^ ​ i ​ is the model-predicted aggregate loss dollars for policy i i i , calculated as earned exposure e i e_i e i ​ multiplied by the predicted pure premium μ ^ i \hat{\mu}_i μ ^ ​ i ​ .

Actuarial Interpretation of the Global A/E Ratio

A/E = 1.00 \text{A/E} = 1.00 A/E = 1.00 (Perfect Aggregate Calibration) :

The total dollar volume of losses predicted by the model exactly matches the total dollar volume of losses observed in the validation experience. The portfolio base rate is properly centered.

A/E > 1.00 \text{A/E} > 1.00 A/E > 1.00 (Systematic Under-prediction) :

Actual losses exceed predicted losses. The model underestimates overall risk costs. If rates are implemented without upward base rate adjustment, the insurer will experience an underwriting loss.

A/E < 1.00 \text{A/E} < 1.00 A/E < 1.00 (Systematic Over-prediction) :

Predicted losses exceed actual losses. The model overestimates risk costs, which will lead to uncompetitive pricing in the marketplace.

The Fallacy of Relying Solely on Global Calibration

Important

A global A/E ratio of exactly 1.00 1.00 1.00 is a necessary but not sufficient condition for model validity. Aggregate balance can easily disguise catastrophic offsetting miscalibrations between rating classes.

Consider an auto bodily injury model evaluated on a holdout portfolio of 50 , 000 50,000 50 , 000 car-years:

  • Total Actual Losses: $ 25 , 000 , 000 \text{\textdollar}25,000,000 $ 25 , 000 , 000
  • Total Predicted Losses: $ 25 , 000 , 000 \text{\textdollar}25,000,000 $ 25 , 000 , 000

Global A/E = $ 25 , 000 , 000 $ 25 , 000 , 000 = 1.000 \text{Global A/E} = \frac{\text{\textdollar}25,000,000}{\text{\textdollar}25,000,000} = 1.000 Global A/E = $ 25 , 000 , 000 $ 25 , 000 , 000 ​ = 1.000 (Appears flawless)

However, decomposing the portfolio into youthful vs. mature drivers reveals severe localized failure:

Driver Class Earned Exposure Actual Losses Predicted Losses Segment A/E Actuarial Consequence

Youthful (Age < 25) 10,000 car-years $12,500,000 $8,333,333 1.50 Underpriced by 33 % 33\% 33% ; severe adverse selection and combined ratio losses.

Mature (Age 25+) 40,000 car-years $12,500,000 $16,666,667 0.75 Overpriced by 25 % 25\% 25% ; competitors will capture all low-risk adult drivers.

Total Portfolio 50,000 car-years $25,000,000 $25,000,000 1.00 Offsetting errors mask severe cross-subsidies.

Because the youthful driver undercharge exactly balances the mature driver overcharge in total dollars, the global A/E ratio fails to detect the distortion. To uncover these structural defects, actuaries must analyze marginal calibration .

Marginal Calibration: One-Way Actual vs. Expected Plots

Marginal calibration diagnostics (commonly known in casualty ratemaking as one-way actual vs. expected plots or univariate lift graphs ) evaluate model fidelity across discrete categories or continuous intervals of every candidate rating variable.

Mathematical Formulation

Let X j X_j X j ​ represent a rating variable (e.g., Driver Age, Vehicle Body Type, Territory, Home Age, Construction Class). We partition the domain of X j X_j X j ​ into K K K distinct bins b ∈ { 1 , 2 , … , K } b \in \{1, 2, \dots, K\} b ∈ { 1 , 2 , … , K } . For continuous variables, bins are created using deciles of earned exposure or standardized actuarial age groupings.

For each bin b b b , the actuary computes three quantities on an independent holdout or out-of-time dataset:

Actual Loss Cost (Actual Pure Premium) : ALC b = ∑ i ∈ b y i ∑ i ∈ b e i \text{ALC}_b = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i} ALC b ​ = ∑ i ∈ b ​ e i ​ ∑ i ∈ b ​ y i ​ ​

Predicted Loss Cost (Expected Pure Premium) : PLC b = ∑ i ∈ b e i μ ^ i ∑ i ∈ b e i \text{PLC}_b = \frac{\sum_{i \in b} e_i \hat{\mu}_i}{\sum_{i \in b} e_i} PLC b ​ = ∑ i ∈ b ​ e i ​ ∑ i ∈ b ​ e i ​ μ ^ ​ i ​ ​

Marginal Segment A/E Ratio : A/E b = ALC b PLC b = ∑ i ∈ b y i ∑ i ∈ b e i μ ^ i \text{A/E}_b = \frac{\text{ALC}_b}{\text{PLC}_b} = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i \hat{\mu}_i} A/E b ​ = PLC b ​ ALC b ​ ​ = ∑ i ∈ b ​ e i ​ μ ^ ​ i ​ ∑ i ∈ b ​ y i ​ ​

Dual-Axis Visual Architecture

A standard actuarial one-way plot uses a dual-axis layout:

Primary Left Y-Axis (Loss Cost Scale) : Two overlaid lines tracking ALC b \text{ALC}_b ALC b ​ (solid line with observed markers) and PLC b \text{PLC}_b PLC b ​ (dashed model curve).

Secondary Right Y-Axis (Exposure Volume) : A vertical bar chart showing total earned exposure e b e_b e b ​ in each bin, providing visual credibility assessment. ONE-WAY ACTUAL VS. EXPECTED: DRIVER AGE BINS

  • Loss Cost ($)
  • [Primary Axis] Earned Exposure (Car-Years)
  • [Secondary Axis, Bars]
  • $1,200 ┼───────────────────────────────────────────────────┬ 25,000
  • │ * │
  • $1,000 ┼───\───────────────────────────────────────────────┼ 20,000
  • │ \* * │
  • $800 ┼─────\*────────────────────────────────────╱───────┼ 15,000
  • │ \* *╱ │
  • $600 ┼───────\──* *──╱─────────┼ 10,000
  • │ \ \ ╱ ╱ ██ │
  • $400 ┼─────────\──\───────────────────────╱──╱────██─────┼ 5,000
  • │ ██ \──*───*───*─────*───*───*──╱ ██ │
  • $200 ┼───██─────██──██──██──██──██──██──██────────██─────┼ 0
  • └───┴──────┴───┴───┴───┴───┴───┴───┴─────────┴─────┘
  • 16-20 21-25 26-30 31-40 41-50 51-60 61-70 71-80 81+
  • [Low] [--- Moderate to High Exposure ---] [Low]

Legend:

  • * ── * : Actual Loss Cost (Observed)
  • - - : Predicted Loss Cost (Model)
  • █████ : Earned Exposure (Car-Years)
  • Diagnostic Patterns and Actuarial Remedies

Well-Calibrated Factor : The predicted line closely tracks the actual loss cost markers across all high-exposure bins, with segment A/E ratios hovering tightly between 0.95 0.95 0.95 and 1.05 1.05 1.05 .

Missing Non-Linearity (Functional Misspecification) :

If the actual loss costs display a steep U-shaped curve across age bins (elevated losses for teen drivers, low losses for middle-aged drivers, rising losses for drivers aged 75+) while the model line is linear, the model has misspecified the predictor's functional form. Remedy : Replace the linear term with restricted cubic splines, fractional polynomials, or non-linear tree splits.

Unmodeled Interaction Effect :

If an entire segment of a one-way plot (e.g., urban sports car drivers) exhibits actual loss costs substantially above predicted values, a multi-way interaction exists between territory and vehicle class that the additive linear predictor failed to capture. Remedy : Include an interaction term β terr × veh \beta_{\text{terr} \times \text{veh}} β terr × veh ​ or utilize gradient-boosted trees to detect higher-order feature splits.

Credibility-Driven Fluctuations :

Large divergences in bins with minimal exposure (e.g., drivers aged 85+ representing < 1 % < 1\% < 1% of total car-years) typically reflect sampling noise rather than model misspecification. Remedy : Apply actuarial credibility shrinkage (such as Bühlmann-Straub credibility) or collapse small edge bins.