Study guide
7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts
CAS Property & Casualty Predictive Analytics PCPA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: PCPA Exam & Project Orientation
1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading
2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing
2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56
3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM
3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling
4 Chapter 4: Model Diagnostics & Selection — Building a GLM
4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity
5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors
5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities
6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit
6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation
7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice
7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves
8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits
8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits
9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis
9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric
7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts
Key Takeaways Explain This Topic AI
Discrimination metrics measure whether a model ranks risks correctly; calibration measures whether the predicted dollar amounts are right, and a model can pass one while failing the other.
The portfolio Actual-to-Expected ratio, sum(actual losses) / sum(predicted losses), shows overall rate balance but can hide large offsetting errors between segments.
Plots of actual against predicted must be built on holdout data, aggregated into equal-weight buckets, and drawn on a log scale, because loss data spans orders of magnitude.
Points crossing the 45-degree line — high at the left and low at the right — mean predictions are too compressed, so the rating plan under-segments even when its overall level is correct.
A loss ratio chart sorts risks by predicted loss ratio and plots actual loss ratio by bucket; a perfect current rating plan would produce a flat line, so any spread is evidence of mispricing the model has found.
Last updated: September 2026
Calibration: The Complement to Discrimination
While discrimination metrics (such as the Gini coefficient and ROC AUC covered in Section 7.4) determine how effectively a model separates low-risk policies from high-risk policies, calibration metrics evaluate whether the predicted dollar amounts equal true expected loss costs .
In casualty actuarial practice, an uncalibrated model is dangerous:
If rates are set according to an under-predicting model (Actual-to-Expected ratio > 1.0 > 1.0 > 1.0 ), collected premiums will be inadequate to cover incurred claims and expenses, resulting in underwriting losses, combined ratio deterioration, and regulatory solvency intervention.
If rates are set according to an over-predicting model (Actual-to-Expected ratio < 1.0 < 1.0 < 1.0 ), premiums will be commercially uncompetitive, causing policyholder defection, declining renewal retention, and adverse selection.
Rigorous model validation requires testing calibration at two distinct levels: global calibration across the total portfolio and marginal calibration across individual rating dimensions.
Global Calibration: The Actual-to-Expected (A/E) Ratio
The most fundamental calibration metric is the portfolio-wide Actual-to-Expected (A/E) ratio (also termed the Actual-over-Expected loss ratio): A/E global = ∑ i = 1 n y i ∑ i = 1 n y ^ i = ∑ i = 1 n y i ∑ i = 1 n e i μ ^ i \text{A/E}_{\text{global}} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n \hat{y}_i} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n e_i \hat{\mu}_i} A/E global = ∑ i = 1 n y ^ i ∑ i = 1 n y i = ∑ i = 1 n e i μ ^ i ∑ i = 1 n y i
Where:
y i y_i y i is the actual observed aggregate loss dollars incurred by policy i i i .
y ^ i = e i μ ^ i \hat{y}_i = e_i \hat{\mu}_i y ^ i = e i μ ^ i is the model-predicted aggregate loss dollars for policy i i i , calculated as earned exposure e i e_i e i multiplied by the predicted pure premium μ ^ i \hat{\mu}_i μ ^ i .
Actuarial Interpretation of the Global A/E Ratio
A/E = 1.00 \text{A/E} = 1.00 A/E = 1.00 (Perfect Aggregate Calibration) :
The total dollar volume of losses predicted by the model exactly matches the total dollar volume of losses observed in the validation experience. The portfolio base rate is properly centered.
A/E > 1.00 \text{A/E} > 1.00 A/E > 1.00 (Systematic Under-prediction) :
Actual losses exceed predicted losses. The model underestimates overall risk costs. If rates are implemented without upward base rate adjustment, the insurer will experience an underwriting loss.
A/E < 1.00 \text{A/E} < 1.00 A/E < 1.00 (Systematic Over-prediction) :
Predicted losses exceed actual losses. The model overestimates risk costs, which will lead to uncompetitive pricing in the marketplace.
The Fallacy of Relying Solely on Global Calibration
Important
A global A/E ratio of exactly 1.00 1.00 1.00 is a necessary but not sufficient condition for model validity. Aggregate balance can easily disguise catastrophic offsetting miscalibrations between rating classes.
Consider an auto bodily injury model evaluated on a holdout portfolio of 50 , 000 50,000 50 , 000 car-years:
- Total Actual Losses: $ 25 , 000 , 000 \text{\textdollar}25,000,000 $ 25 , 000 , 000
- Total Predicted Losses: $ 25 , 000 , 000 \text{\textdollar}25,000,000 $ 25 , 000 , 000
Global A/E = $ 25 , 000 , 000 $ 25 , 000 , 000 = 1.000 \text{Global A/E} = \frac{\text{\textdollar}25,000,000}{\text{\textdollar}25,000,000} = 1.000 Global A/E = $ 25 , 000 , 000 $ 25 , 000 , 000 = 1.000 (Appears flawless)
However, decomposing the portfolio into youthful vs. mature drivers reveals severe localized failure:
Driver Class Earned Exposure Actual Losses Predicted Losses Segment A/E Actuarial Consequence
Youthful (Age < 25) 10,000 car-years $12,500,000 $8,333,333 1.50 Underpriced by 33 % 33\% 33% ; severe adverse selection and combined ratio losses.
Mature (Age 25+) 40,000 car-years $12,500,000 $16,666,667 0.75 Overpriced by 25 % 25\% 25% ; competitors will capture all low-risk adult drivers.
Total Portfolio 50,000 car-years $25,000,000 $25,000,000 1.00 Offsetting errors mask severe cross-subsidies.
Because the youthful driver undercharge exactly balances the mature driver overcharge in total dollars, the global A/E ratio fails to detect the distortion. To uncover these structural defects, actuaries must analyze marginal calibration .
Marginal Calibration: One-Way Actual vs. Expected Plots
Marginal calibration diagnostics (commonly known in casualty ratemaking as one-way actual vs. expected plots or univariate lift graphs ) evaluate model fidelity across discrete categories or continuous intervals of every candidate rating variable.
Mathematical Formulation
Let X j X_j X j represent a rating variable (e.g., Driver Age, Vehicle Body Type, Territory, Home Age, Construction Class). We partition the domain of X j X_j X j into K K K distinct bins b ∈ { 1 , 2 , … , K } b \in \{1, 2, \dots, K\} b ∈ { 1 , 2 , … , K } . For continuous variables, bins are created using deciles of earned exposure or standardized actuarial age groupings.
For each bin b b b , the actuary computes three quantities on an independent holdout or out-of-time dataset:
Actual Loss Cost (Actual Pure Premium) : ALC b = ∑ i ∈ b y i ∑ i ∈ b e i \text{ALC}_b = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i} ALC b = ∑ i ∈ b e i ∑ i ∈ b y i
Predicted Loss Cost (Expected Pure Premium) : PLC b = ∑ i ∈ b e i μ ^ i ∑ i ∈ b e i \text{PLC}_b = \frac{\sum_{i \in b} e_i \hat{\mu}_i}{\sum_{i \in b} e_i} PLC b = ∑ i ∈ b e i ∑ i ∈ b e i μ ^ i
Marginal Segment A/E Ratio : A/E b = ALC b PLC b = ∑ i ∈ b y i ∑ i ∈ b e i μ ^ i \text{A/E}_b = \frac{\text{ALC}_b}{\text{PLC}_b} = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i \hat{\mu}_i} A/E b = PLC b ALC b = ∑ i ∈ b e i μ ^ i ∑ i ∈ b y i
Dual-Axis Visual Architecture
A standard actuarial one-way plot uses a dual-axis layout:
Primary Left Y-Axis (Loss Cost Scale) : Two overlaid lines tracking ALC b \text{ALC}_b ALC b (solid line with observed markers) and PLC b \text{PLC}_b PLC b (dashed model curve).
Secondary Right Y-Axis (Exposure Volume) : A vertical bar chart showing total earned exposure e b e_b e b in each bin, providing visual credibility assessment. ONE-WAY ACTUAL VS. EXPECTED: DRIVER AGE BINS
- Loss Cost ($)
- [Primary Axis] Earned Exposure (Car-Years)
- [Secondary Axis, Bars]
- $1,200 ┼───────────────────────────────────────────────────┬ 25,000
- │ * │
- $1,000 ┼───\───────────────────────────────────────────────┼ 20,000
- │ \* * │
- $800 ┼─────\*────────────────────────────────────╱───────┼ 15,000
- │ \* *╱ │
- $600 ┼───────\──* *──╱─────────┼ 10,000
- │ \ \ ╱ ╱ ██ │
- $400 ┼─────────\──\───────────────────────╱──╱────██─────┼ 5,000
- │ ██ \──*───*───*─────*───*───*──╱ ██ │
- $200 ┼───██─────██──██──██──██──██──██──██────────██─────┼ 0
- └───┴──────┴───┴───┴───┴───┴───┴───┴─────────┴─────┘
- 16-20 21-25 26-30 31-40 41-50 51-60 61-70 71-80 81+
- [Low] [--- Moderate to High Exposure ---] [Low]
Legend:
- * ── * : Actual Loss Cost (Observed)
- - - : Predicted Loss Cost (Model)
- █████ : Earned Exposure (Car-Years)
- Diagnostic Patterns and Actuarial Remedies
Well-Calibrated Factor : The predicted line closely tracks the actual loss cost markers across all high-exposure bins, with segment A/E ratios hovering tightly between 0.95 0.95 0.95 and 1.05 1.05 1.05 .
Missing Non-Linearity (Functional Misspecification) :
If the actual loss costs display a steep U-shaped curve across age bins (elevated losses for teen drivers, low losses for middle-aged drivers, rising losses for drivers aged 75+) while the model line is linear, the model has misspecified the predictor's functional form. Remedy : Replace the linear term with restricted cubic splines, fractional polynomials, or non-linear tree splits.
Unmodeled Interaction Effect :
If an entire segment of a one-way plot (e.g., urban sports car drivers) exhibits actual loss costs substantially above predicted values, a multi-way interaction exists between territory and vehicle class that the additive linear predictor failed to capture. Remedy : Include an interaction term β terr × veh \beta_{\text{terr} \times \text{veh}} β terr × veh or utilize gradient-boosted trees to detect higher-order feature splits.
Credibility-Driven Fluctuations :
Large divergences in bins with minimal exposure (e.g., drivers aged 85+ representing < 1 % < 1\% < 1% of total car-years) typically reflect sampling noise rather than model misspecification. Remedy : Apply actuarial credibility shrinkage (such as Bühlmann-Straub credibility) or collapse small edge bins.