Study guide

4.3 Overdispersion, Quasi-Poisson & Negative Binomial

CAS Property & Casualty Predictive Analytics PCPA

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: PCPA Exam & Project Orientation

1.1 PCPA Exam Format, Scoring & Registration 1.2 The PCPA Project: Windows, Deliverables & Grading

2 Chapter 2: Dealing with Data — Sourcing, Importing & Assessing

2.1 Requirements Gathering & Assessing Data Relevance 2.2 Importing, Joining & Reshaping Data Sets 2.3 Exploratory Data Analysis & Evaluating a Data Set 2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

3 Chapter 3: Dealing with Data — Conditioning P&C Data for a GLM

3.1 Exposure Bases, Weights & Data Grain 3.2 Missing Data Mechanisms & Imputation Strategies 3.3 Outlier Diagnostics, Trimming & Winsorization 3.4 Variable Transformation & Scaling

4 Chapter 4: Model Diagnostics & Selection — Building a GLM

4.1 Exponential Dispersion Family & Link Functions 4.2 Poisson Regression & Exposure Offsets 4.3 Overdispersion, Quasi-Poisson & Negative Binomial 4.4 Gamma & Inverse Gaussian Models for Claim Severity

5 Chapter 5: Model Diagnostics & Selection — Model Form & Predictors

5.1 Tweedie Compound Poisson-Gamma Modeling 5.2 Binomial & Logistic GLMs for Binary Outcomes 5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping 5.4 Interaction Terms & Non-Linear Effects 5.5 Interpreting Coefficients & Multiplicative Relativities

6 Chapter 6: Model Diagnostics & Selection — Evaluating and Improving the Fit

6.1 Deviance, Likelihood Ratio Tests & Residual Diagnostics 6.2 Stepwise Selection & Information Criteria Penalties 6.3 Multicollinearity, Correlated Predictors & Aliasing 6.4 Under-fitting, Over-fitting & Cross-Validation

7 Chapter 7: Model Diagnostics & Selection — Validation & Model Choice

7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts 7.2 Simple Quantile Plots & Measuring Lift 7.3 Double Lift Charts & Competing Model Evaluation 7.4 Gini Index, Lorenz Curves & ROC Curves

8 Chapter 8: Model Interpretation & Presentation — Tables, Charts & Exhibits

8.1 Designing Tables & Statistical Exhibits 8.2 Chart Selection & Graphical Perception 8.3 How Charts Mislead 8.4 One-Way, Two-Way & Partial Dependence Exhibits

9 Chapter 9: Model Interpretation & Presentation — Communicating the Analysis

9.1 Communicating Methodology & Modeling Decisions 9.2 Explaining Why a Variable Was or Was Not Included 9.3 Communicating to Non-Technical Audiences 9.4 The PCPA Technical Report: Structure & Rubric

4.3 Overdispersion, Quasi-Poisson & Negative Binomial

Key Takeaways Explain This Topic AI

Overdispersion occurs when the empirical variance of claim counts exceeds the mean (Var(Y) > E[Y]), violating the Poisson equidispersion assumption.

The Pearson dispersion statistic phi_hat = chi^2 / (n - p) diagnoses overdispersion; when phi_hat significantly exceeds 1.0, Poisson coefficient estimates remain consistent, but standard errors are underestimated, inflating Type I error rates.

Overdispersion in P&C insurance stems primarily from unobserved policyholder risk heterogeneity (omitted variables) and claim contagion/clustering (e.g., convective storms, multi-claimant accidents).

The Negative Binomial (NB2) model arises as a continuous Poisson-Gamma mixture where the Poisson parameter lambda is multiplied by a Gamma-distributed error term, yielding variance Var(Y) = mu + alpha * mu^2.

While Zero-Inflated Poisson (ZIP) and Zero-Inflated Negative Binomial (ZINB) models separate structural zeros from sampling zeros, standard Poisson or Negative Binomial models with exposure offsets are preferred in regulated ratemaking due to regulatory transparency and exposure consistency.

Last updated: September 2026

The Problem of Overdispersion in Insurance Data

In classical Poisson regression, the conditional variance is constrained to equal the conditional mean ( Var ( Y ∣ x ) = E [ Y ∣ x ] = μ \text{Var}(Y \mid \mathbf{x}) = \mathbb{E}[Y \mid \mathbf{x}] = \mu Var ( Y ∣ x ) = E [ Y ∣ x ] = μ ). In empirical property and casualty insurance datasets, this condition is almost never satisfied. Instead, actuaries encounter overdispersion : Var ( Y ∣ x ) > E [ Y ∣ x ] \text{Var}(Y \mid \mathbf{x}) > \mathbb{E}[Y \mid \mathbf{x}] Var ( Y ∣ x ) > E [ Y ∣ x ]

Underdispersion ( Var ( Y ) < E [ Y ] \text{Var}(Y) < \mathbb{E}[Y] Var ( Y ) < E [ Y ] ) is virtually non-existent in general insurance, as loss occurrences are characterized by volatility, clustering, and latent risk factors.

Root Causes of Overdispersion in P&C Portfolios

Overdispersion does not imply that the data are corrupt; rather, it reflects genuine physical and behavioral characteristics of insurance risks:

1. Unobserved Policyholder Heterogeneity (Omitted Variable Bias)

No matter how granular an insurer's rating classification is, observable variables (such as driver age, territory, vehicle make, and credit score) capture only a fraction of true risk variance. Within any rating cell, unobservable traits—such as driver aggressiveness, reaction speed, daily route congestion, vehicle maintenance, and texting while driving—create substantial latent risk differences. This unmodeled variation causes claim counts to scatter far more widely than a homogeneous Poisson process predicts.

2. Claim Contagion and Event Clustering

The Poisson process assumes that individual claim occurrences are mutually independent. In insurance, this independence assumption is routinely violated:

Catastrophic Weather Events : A single hailstorm, convective wind event, or winter freeze can generate thousands of comprehensive auto or homeowners claims simultaneously across an entire territory.

Multi-Claimant Accidents : A single severe multi-vehicle collision can trigger simultaneous bodily injury, property damage, and collision claims from multiple parties.

Commercial Fleets : Multiple vehicles operating in the same logistics fleet may be exposed to shared hazardous working conditions or poor corporate maintenance policies.

3. Exposure Measurement Inaccuracies

Earned vehicle-years or house-years treat exposure as uniform over time. In reality, miles driven fluctuate drastically by season, remote-work status, or economic conditions, introducing extra-Poisson variance.

Diagnosing and Measuring Overdispersion

To detect and quantify overdispersion in a fitted Poisson GLM, actuaries evaluate the Pearson Chi-Square statistic χ 2 \chi^2 χ 2 : χ 2 = ∑ i = 1 n ( y i − μ ^ i ) 2 V ( μ ^ i ) = ∑ i = 1 n ( y i − μ ^ i ) 2 μ ^ i \chi^2 = \sum_{i=1}^n \frac{(y_i - \hat{\mu}_i)^2}{V(\hat{\mu}_i)} = \sum_{i=1}^n \frac{(y_i - \hat{\mu}_i)^2}{\hat{\mu}_i} χ 2 = i = 1 ∑ n ​ V ( μ ^ ​ i ​ ) ( y i ​ − μ ^ ​ i ​ ) 2 ​ = i = 1 ∑ n ​ μ ^ ​ i ​ ( y i ​ − μ ^ ​ i ​ ) 2 ​

Where:

y i y_i y i ​ is the observed claim count for policy i i i .

μ ^ i = e i exp ⁡ ( x i T β ^ ) \hat{\mu}_i = e_i \exp(\mathbf{x}_i^T \hat{\boldsymbol{\beta}}) μ ^ ​ i ​ = e i ​ exp ( x i T ​ β ^ ​ ) is the fitted expected claim count.

V ( μ ^ i ) = μ ^ i V(\hat{\mu}_i) = \hat{\mu}_i V ( μ ^ ​ i ​ ) = μ ^ ​ i ​ is the Poisson variance function.

The Pearson dispersion statistic ϕ ^ \hat{\phi} ϕ ^ ​ is calculated by dividing χ 2 \chi^2 χ 2 by the residual degrees of freedom: ϕ ^ = χ 2 n − p \hat{\phi} = \frac{\chi^2}{n - p} ϕ ^ ​ = n − p χ 2 ​

Where n n n is the number of observations and p p p is the number of estimated parameters (including the intercept).

Interpreting the Dispersion Statistic

ϕ ^ ≈ 1.0 \hat{\phi} \approx 1.0 ϕ ^ ​ ≈ 1.0 : Equidispersion holds; the standard Poisson model is adequate.

ϕ ^ > 1.0 \hat{\phi} > 1.0 ϕ ^ ​ > 1.0 (e.g., 1.5 , 2.0 , 5.0 + 1.5, 2.0, 5.0+ 1.5 , 2.0 , 5.0 + ): Significant overdispersion exists.

ϕ ^ < 1.0 \hat{\phi} < 1.0 ϕ ^ ​ < 1.0 : Underdispersion (rare in insurance applications).

Statistical Consequences of Ignoring Overdispersion

When a model exhibits overdispersion, fitting a standard Poisson GLM produces two major statistical consequences:

Point Estimates β ^ \hat{\boldsymbol{\beta}} β ^ ​ Remain Consistent : As long as the conditional mean function E [ Y ∣ x ] = exp ⁡ ( x T β ) \mathbb{E}[Y \mid \mathbf{x}] = \exp(\mathbf{x}^T \boldsymbol{\beta}) E [ Y ∣ x ] = exp ( x T β ) is correctly specified, maximum likelihood estimates of the regression coefficients remain unbiased and asymptotically consistent.

Standard Errors SE ( β ^ ) \text{SE}(\hat{\boldsymbol{\beta}}) SE ( β ^ ​ ) are Underestimated : The standard Poisson model assumes ϕ = 1.0 \phi = 1.0 ϕ = 1.0 . If the true dispersion is ϕ = 2.25 \phi = 2.25 ϕ = 2.25 , the true standard errors are 2.25 = 1.5 \sqrt{2.25} = 1.5 2.25 ​ = 1.5 times larger than reported by the software. True SE ( β ^ j ) ≈ ϕ ^ ⋅ Reported SE ( β ^ j ) \text{True } \text{SE}(\hat{\beta}_j) \approx \sqrt{\hat{\phi}} \cdot \text{Reported } \text{SE}(\hat{\beta}_j) True SE ( β ^ ​ j ​ ) ≈ ϕ ^ ​ ​ ⋅ Reported SE ( β ^ ​ j ​ )

Underestimating standard errors leads to:

Artificially inflated z z z -statistics ( z = β ^ / SE z = \hat{\beta} / \text{SE} z = β ^ ​ / SE ).

Deceptively small p p p -values.

Artificially narrow confidence intervals.

Severe Type I error inflation : Variables that have no true relationship with claim frequency will test as "statistically significant" at the 5 % 5\% 5% level, leading to the inclusion of spurious rating variables in rate filings.

The Negative Binomial Model (NB2 Parameterization)

The most common method for resolving overdispersion in insurance frequency modeling is the Negative Binomial regression model , specifically the NB2 formulation .

Derivation as a Continuous Poisson-Gamma Mixture

To account for unobserved policyholder heterogeneity, suppose that the conditional claim count for policy i i i , given a latent risk effect ϵ i \epsilon_i ϵ i ​ , follows a Poisson distribution: Y i ∣ ϵ i ∼ Poisson ( λ i ∗ ) Y_i \mid \epsilon_i \sim \text{Poisson}(\lambda_i^*) Y i ​ ∣ ϵ i ​ ∼ Poisson ( λ i ∗ ​ )

Where the hazard rate λ i ∗ \lambda_i^* λ i ∗ ​ is modulated by an unobserved random multiplier ϵ i \epsilon_i ϵ i ​ : λ i ∗ = μ i ⋅ ϵ i \lambda_i^* = \mu_i \cdot \epsilon_i λ i ∗ ​ = μ i ​ ⋅ ϵ i ​

Here, μ i = e i exp ⁡ ( x i T β ) \mu_i = e_i \exp(\mathbf{x}_i^T \boldsymbol{\beta}) μ i ​ = e i ​ exp ( x i T ​ β ) represents the mean frequency predicted by observable rating variables, and ϵ i \epsilon_i ϵ i ​ represents unobserved policyholder risk characteristics.

To ensure that E [ Y i ] = μ i \mathbb{E}[Y_i] = \mu_i E [ Y i ​ ] = μ i ​ , we assume that ϵ i \epsilon_i ϵ i ​ follows a Gamma distribution with mean 1 and variance α \alpha α : ϵ i ∼ Gamma ( shape = 1 α , scale = α ) \epsilon_i \sim \text{Gamma}\left(\text{shape} = \frac{1}{\alpha}, \text{scale} = \alpha\right) ϵ i ​ ∼ Gamma ( shape = α 1 ​ , scale = α ) E [ ϵ i ] = ( 1 α ) α = 1 \mathbb{E}[\epsilon_i] = \left(\frac{1}{\alpha}\right) \alpha = 1 E [ ϵ i ​ ] = ( α 1 ​ ) α = 1 Var ( ϵ i ) = ( 1 α ) α 2 = α \text{Var}(\epsilon_i) = \left(\frac{1}{\alpha}\right) \alpha^2 = \alpha Var ( ϵ i ​ ) = ( α 1 ​ ) α 2 = α

Where α > 0 \alpha > 0 α > 0 is the overdispersion parameter .

Deriving the Unconditional Mean and Variance

By the Law of Total Expectation : E [ Y i ] = E ϵ [ E Y ∣ ϵ [ Y i ∣ ϵ i ] ] = E ϵ [ μ i ϵ i ] = μ i E [ ϵ i ] = μ i ⋅ 1 = μ i \mathbb{E}[Y_i] = \mathbb{E}_\epsilon [\mathbb{E}_{Y|\epsilon}[Y_i \mid \epsilon_i]] = \mathbb{E}_\epsilon [\mu_i \epsilon_i] = \mu_i \mathbb{E}[\epsilon_i] = \mu_i \cdot 1 = \mu_i E [ Y i ​ ] = E ϵ ​ [ E Y ∣ ϵ ​ [ Y i ​ ∣ ϵ i ​ ]] = E ϵ ​ [ μ i ​ ϵ i ​ ] = μ i ​ E [ ϵ i ​ ] = μ i ​ ⋅ 1 = μ i ​

By the Law of Total Variance : Var ( Y i ) = E ϵ [ Var ( Y i ∣ ϵ i ) ] + Var ϵ ( E [ Y i ∣ ϵ i ] ) \text{Var}(Y_i) = \mathbb{E}_\epsilon [\text{Var}(Y_i \mid \epsilon_i)] + \text{Var}_\epsilon (\mathbb{E}[Y_i \mid \epsilon_i]) Var ( Y i ​ ) = E ϵ ​ [ Var ( Y i ​ ∣ ϵ i ​ )] + Var ϵ ​ ( E [ Y i ​ ∣ ϵ i ​ ]) Var ( Y i ) = E ϵ [ μ i ϵ i ] + Var ϵ ( μ i ϵ i ) \text{Var}(Y_i) = \mathbb{E}_\epsilon [\mu_i \epsilon_i] + \text{Var}_\epsilon (\mu_i \epsilon_i) Var ( Y i ​ ) = E ϵ ​ [ μ i ​ ϵ i ​ ] + Var ϵ ​ ( μ i ​ ϵ i ​ ) Var ( Y i ) = μ i E [ ϵ i ] + μ i 2 Var ( ϵ i ) \text{Var}(Y_i) = \mu_i \mathbb{E}[\epsilon_i] + \mu_i^2 \text{Var}(\epsilon_i) Var ( Y i ​ ) = μ i ​ E [ ϵ i ​ ] + μ i 2 ​ Var ( ϵ i ​ ) Var ( Y i ) = μ i + α μ i 2 \text{Var}(Y_i) = \mu_i + \alpha \mu_i^2 Var ( Y i ​ ) = μ i ​ + α μ i 2 ​

Key Mathematical Insights of NB2

Quadratic Variance Function : In the NB2 model, the variance function is V ( μ ) = μ + α μ 2 V(\mu) = \mu + \alpha \mu^2 V ( μ ) = μ + α μ 2 . The extra-Poisson variance grows quadratically with the mean, providing heavy right-tail flexibility for higher-risk policies.

Convergence to Poisson : As the overdispersion parameter α → 0 \alpha \to 0 α → 0 , the quadratic term vanishes: lim ⁡ α → 0 Var ( Y i ) = μ i \lim_{\alpha \to 0} \text{Var}(Y_i) = \mu_i α → 0 lim ​ Var ( Y i ​ ) = μ i ​

The Negative Binomial distribution converges smoothly back to the Poisson distribution.

3. Likelihood Ratio Test for Overdispersion : To test whether a Negative Binomial model is statistically superior to a Poisson model, actuaries test H 0 : α = 0 H_0: \alpha = 0 H 0 ​ : α = 0 versus H 1 : α > 0 H_1: \alpha > 0 H 1 ​ : α > 0 . Because α = 0 \alpha = 0 α = 0 lies on the boundary of the parameter space, the asymptotic distribution of the Likelihood Ratio Test statistic is a 50 : 50 50:50 50 : 50 mixture of a point mass at 0 and a χ 1 2 \chi^2_1 χ 1 2 ​ distribution (denoted 0.5 χ 0 2 + 0.5 χ 1 2 0.5 \chi^2_0 + 0.5 \chi^2_1 0.5 χ 0 2 ​ + 0.5 χ 1 2 ​ ). The critical value at the 5 % 5\% 5% significance level is 2.706 2.706 2.706 (rather than the standard 3.841 3.841 3.841 ).

Zero-Inflated Models: ZIP and ZINB

In personal auto and homeowners lines, 90 % 90\% 90% to 98 % 98\% 98% of policies experience zero claims during an annual term. When empirical data exhibit a higher proportion of zeros than predicted by a standard Poisson distribution, analysts often consider Zero-Inflated models :

  • Zero-Inflated Poisson (ZIP)
  • Zero-Inflated Negative Binomial (ZINB)
  • The Two-State Mixture Concept

Zero-inflated models posit that the observed zeros arise from two fundamentally different latent mechanisms:

Structural Zeros ("Always Zero" State) : A subpopulation of policyholders with zero probability of experiencing a loss ( P ( Y = 0 ) = 1.0 P(Y = 0) = 1.0 P ( Y = 0 ) = 1.0 ). Examples might include a vehicle stored in a secure garage and never driven, or a vacant property with utilities disconnected.

Sampling Zeros ("At-Risk" State) : Policyholders who are actively exposed to loss, but by chance experienced no claims during the observation period. Their counts follow a standard Poisson or Negative Binomial process.

Mathematical Formulation

Let π i \pi_i π i ​ denote the probability of belonging to the structural zero state. The probability mass function of a ZIP model is: P ( Y i = 0 ) = π i + ( 1 − π i ) e − μ i P(Y_i = 0) = \pi_i + (1 - \pi_i) e^{-\mu_i} P ( Y i ​ = 0 ) = π i ​ + ( 1 − π i ​ ) e − μ i ​ P ( Y i = y ) = ( 1 − π i ) μ i y e − μ i y ! , y ∈ { 1 , 2 , 3 , … } P(Y_i = y) = (1 - \pi_i) \frac{\mu_i^y e^{-\mu_i}}{y!}, \quad y \in \{1, 2, 3, \dots\} P ( Y i ​ = y ) = ( 1 − π i ​ ) y ! μ i y ​ e − μ i ​ ​ , y ∈ { 1 , 2 , 3 , … }

ZIP models require estimating two separate regression equations simultaneously:

A logistic regression for the zero-inflation probability: logit ( π i ) = z i T γ \text{logit}(\pi_i) = \mathbf{z}_i^T \boldsymbol{\gamma} logit ( π i ​ ) = z i T ​ γ .

A log-link Poisson regression for the count mean: ln ⁡ ( μ i ) = x i T β \ln(\mu_i) = \mathbf{x}_i^T \boldsymbol{\beta} ln ( μ i ​ ) = x i T ​ β .

Why Regulators and Actuaries Prefer Negative Binomial Over ZIP

Despite their statistical elegance, Zero-Inflated models are rarely used in filed P&C rating algorithms for compelling practical and regulatory reasons:

Incompatibility with Multiplicative Rating Engines : Standard insurance rating manuals require a single multiplicative chain: Rate = Base × F 1 × F 2 \text{Rate} = \text{Base} \times F_1 \times F_2 Rate = Base × F 1 ​ × F 2 ​ . A ZIP model produces two separate sets of non-linear coefficients (one for π i \pi_i π i ​ and one for μ i \mu_i μ i ​ ) that cannot be consolidated into a traditional rating table.

The Exposure Scaling Fallacy : In insurance, a short policy period (e.g., a 1-month policy) naturally produces a very high probability of zero claims ( e − μ ⋅ ( 1 / 12 ) ≈ 1.0 e^{-\mu \cdot (1/12)} \approx 1.0 e − μ ⋅ ( 1/12 ) ≈ 1.0 ). A ZIP model misinterprets this short exposure window as evidence that the driver belongs to the "structurally immune" class, distorting rate equity.

Regulatory Skepticism : Insurance regulators require clear actuarial justification for rating differentials. Explaining to an insurance department why a policyholder's rate is split into an "immunity likelihood" and an "at-risk severity" invites regulatory objections under unfair discrimination statutes.

Occam's Razor : An appropriately specified Negative Binomial GLM with a ln ⁡ ( exposure ) \ln(\text{exposure}) ln ( exposure ) offset naturally handles large proportions of zeros caused by exposure variation and heterogeneity without inventing a fictitious "immune" driver category.

Comparison of Frequency Distribution Models

  • Feature Poisson Quasi-Poisson Negative Binomial (NB2) Zero-Inflated Poisson (ZIP)
  • Mean Function E [ Y ] \mathbb{E}[Y] E [ Y ] μ \mu μ μ \mu μ μ \mu μ ( 1 − π ) μ (1 - \pi)\mu ( 1 − π ) μ

Variance Function Var ( Y ) \text{Var}(Y) Var ( Y ) μ \mu μ ϕ μ \phi \mu ϕ μ μ + α μ 2 \mu + \alpha \mu^2 μ + α μ 2 μ ( 1 − π ) ( 1 + μ π ) \mu(1-\pi)(1 + \mu \pi) μ ( 1 − π ) ( 1 + μ π )

  • Full Likelihood Defined? Yes No (Quasi-likelihood) Yes Yes
  • Handles Overdispersion? No Adjusts SEs only Yes (Mixture model) Yes (Excess zeros only)
  • AIC/BIC Available? Yes No Yes Yes
  • Rating Engine Deployment Excellent Excellent Excellent Poor (Requires 2 equations)
  • Regulatory Acceptance Universal Universal Universal Low / Disfavored