11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

CFA Level 2 Chartered Financial Analyst

Track your progress as you read

Finished sections get a check mark.

1 Introduction & CFA Level II Exam Strategy

1.1 CFA Level II Exam Overview & Item-Set Vignette Strategy 1.2 2026 Curriculum Structure & 10 Topic Weight Bands 1.3 300-Hour Study Plan, Calculator Mastery & Practical Skills Modules (PSM)

2 Ethical and Professional Standards

2.1 Code of Ethics & The Seven Standards of Professional Conduct 2.2 Vignette Analysis: Conflicts of Interest & Disclosure (Standard VI) 2.3 Duties to Clients (Standard III) & Duties to Employers (Standard IV) 2.4 Standard V: Investment Analysis, Recommendations & Actions and Standard VII: Responsibilities as a CFA Member or Candidate 2.5 Application of the Code and Standards: Level II Case Method 2.6 Recommended Procedures: Compliance Programs, Supervisory Systems & Firm Policies

3 Quantitative Methods

3.1 Multiple Linear Regression, ANOVA & Hypothesis Testing 3.2 Regression Diagnostics: Multicollinearity, Heteroskedasticity & Serial Correlation 3.3 Time-Series Analysis: AR Models, Nonstationarity & Cointegration 3.4 Machine Learning, Deep Learning & Big Data Projects 3.5 Extensions of Multiple Regression: Influence Analysis, Qualitative Variables & Logistic Regression 3.6 Big Data Projects: Wrangling, Exploration, Feature Engineering & Model Training

4 Economics for Valuation

4.1 Foreign Exchange Parity Relations & Equilibrium Exchange Rates 4.2 Economic Growth Theories & Production Function Analysis 4.3 Forward Points, the Carry Trade & Policy Effects on Exchange Rates 4.4 Growth Determinants: Natural Resources, Demographics, Convergence & Trade Openness

5 Financial Statement Analysis

5.1 Intercorporate Investments: Equity Method & Business Combinations 5.2 Employee Compensation: Defined Benefit Pension Plans & Share-Based Payments 5.3 Multinational Operations: Current Rate vs Temporal Translation 5.4 Analysis of Financial Institutions & The CAMELS Framework 5.5 Evaluating the Quality of Financial Reports & Accounting Adjustments 5.6 Integration of Financial Statement Analysis Techniques

6 Corporate Issuers

6.1 Analysis of Dividends: Policy Theories, Signaling & Taxation Systems 6.2 Share Repurchases: Methods, EPS and Book Value Effects & Payout Sustainability 6.3 Cost of Capital: Advanced Topics 6.4 Corporate Governance, Board Oversight & ESG Integration 6.5 Corporate Restructuring: Types, Motivations, Evaluation & Valuation 6.6 Restructuring Financial Effects: EPS, Net Debt to EBITDA, WACC & Divestitures

7 Equity Valuation

7.1 Equity Valuation Framework & Required Rate of Return Estimation 7.2 Discounted Dividend Valuation: Gordon Growth, Multistage & The H-Model 7.3 Free Cash Flow Valuation: FCFF and FCFE Modeling 7.4 Market-Based Valuation: Price and Enterprise Value Multiples 7.5 Residual Income Valuation 7.6 Private Company Valuation: Income, Market & Asset-Based Approaches 7.7 Choosing and Combining Models: Definitions of Value, Sum-of-the-Parts & Cross-Border Comparability

8 Fixed Income Analysis

8.1 The Term Structure and Interest Rate Dynamics: Spot, Forward & Swap Curves 8.2 The Arbitrage-Free Valuation Framework & Binomial Interest Rate Trees 8.3 Valuation and Analysis of Bonds with Embedded Options: OAS & Duration 8.4 Credit Analysis Models: Structural vs Reduced-Form Approaches 8.5 Credit Default Swaps (CDS): Mechanics, Pricing & Trading Strategies 8.6 Managing Yield-Curve Risk: Key Rate Durations, One-Sided Durations & Yield Volatility Structure

9 Derivatives & Risk Management

9.1 Pricing and Valuation of Forward Commitments: Equity, Fixed Income & FX 9.2 Swap Pricing and Valuation: Interest Rate, Currency & Equity Swaps 9.3 Contingent Claims: Binomial Option Pricing Model & Arbitrage Replication 9.4 The Black-Scholes-Merton (BSM) Model, Option Greeks & Dynamic Hedging 9.5 Interest Rate Options, Swaptions & the Black Model

10 Alternative Investments

10.1 Commodities and Commodity Derivatives: Sectors, Participants, Futures Returns & Indexes 10.2 Private Real Estate Investments: Direct Capitalization, DCF & Cost Approach 10.3 Public Real Estate Investments: REIT Structures, NAV, FFO & AFFO 10.4 Hedge Fund Strategies: Classification, Implementation & Portfolio Role 10.5 Real Estate Investment Features, Due Diligence & Index Construction Biases

11 Portfolio Management

11.1 Using Multifactor Models: Macroeconomic, Fundamental & Statistical Risk 11.2 Measuring and Managing Market Risk: VaR, CVaR & Stress Testing 11.3 Economics & Investment Markets: Business Cycles, Term Structure & Asset Allocation 11.4 Analysis of Active Portfolio Management & The Fundamental Law 11.5 Exchange-Traded Funds (ETFs): Creation, Arbitrage, Tracking & Applications 11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

Key Takeaways Explain This Topic AI

A backtest runs three repeated steps: specify the strategy and its rules, simulate the investment using a rolling window of formation and holding periods, and analyse the resulting return distribution.

A rolling-window procedure re-estimates parameters and re-forms the portfolio each period using only data available at that time, which is what prevents look-ahead bias.

Survivorship bias, look-ahead bias, and data-snooping bias are the three structural problems in backtests, and repeated testing on the same sample inflates apparent significance.

Historical scenario analysis reprices the current portfolio using the actual returns of a past stress episode, so it is limited to events that have occurred and to the assets that existed then.

Monte Carlo simulation generates outcomes from assumed distributions and so can explore events never observed, but it inherits every assumption embedded in those distributions.

Last updated: August 2026

11.6 Backtesting and Simulation: Procedures, Metrics, Biases & Scenario Analysis

Blueprint note: Backtesting and Simulation is a full Level II Portfolio Management learning module with eight learning outcomes. Vignettes describe a quantitative researcher's testing process and ask what is wrong with it, or present backtest output and ask what conclusion it supports.

1. Objectives of Backtesting

Backtesting approximates the real-life investment process of a strategy using historical data, to learn how it would have performed and, more importantly, what risks it carries .

Legitimate objectives:

• assess the risk-return profile of a strategy before committing capital;
• understand which environments the strategy performs well and badly in;
• estimate turnover, capacity, and transaction costs ;
• test the sensitivity of the result to parameter choices;

compare candidate strategies on a consistent basis.

What backtesting cannot do:

• prove that a strategy will work in future, since it reports one realisation of history;
• capture structural breaks — a regime that has not occurred in the sample is not in the test;

reflect the market impact of the strategy's own trading once it is scaled up.

The core assumption is that the future resembles the past . Where that assumption is weakest — after a regulatory change, a structural shift in market microstructure, or a change in monetary regime — backtest results are least informative.

2. The Backtesting Procedure

Backtesting is not a single pass over history. It is a rolling-window procedure repeated period after period:

Step 1 — Strategy design. Specify the investment hypothesis, the universe, the ranking or signal rules, position sizing, the rebalancing frequency, and the transaction cost assumption. Everything must be fixed before the test runs.

Step 2 — Historical investment simulation. For each period:

use a formation period ending at time t t t to estimate parameters and rank securities, using only information available at time t t t ;

• construct the portfolio at time t t t ;
• hold it over the holding period to t + 1 t+1 t + 1 and record the realised return;

roll the window forward one period and repeat.

The rolling window is what makes the test honest. Estimating a factor loading on the full sample and then "testing" the strategy over that same sample uses information from the future, which is look-ahead bias in its purest form.

Step 3 — Output analysis. Compute the distribution of realised returns and the risk metrics below, then examine the visuals.

Metrics reported

• Metric What it adds
• Average return and standard deviation The first two moments; necessary but insufficient
• Sharpe ratio Return per unit of total risk; misleading for skewed strategies
• Sortino ratio Return per unit of downside deviation; preferred where returns are negatively skewed

Maximum drawdown The largest peak-to-trough decline; the metric that determines whether a strategy is survivable in practice

• Skewness and kurtosis Reveal the fat left tails that the first two moments hide
• Turnover and transaction costs Determine whether a paper return survives implementation
• Hit rate and information ratio Consistency of the signal relative to a benchmark
• Visuals that carry information

Cumulative return chart — reveals whether performance came from a few short episodes or accumulated steadily. A strategy whose entire excess return arrived in two months of one year is not a strategy.

Rolling excess return or rolling Sharpe — shows whether the edge has decayed over time, which is common after a strategy is published.

Drawdown chart — shows the depth and, critically, the duration of drawdowns.

Return distribution histogram — reveals skew and fat tails directly.

• 3. The Three Structural Biases
• Survivorship bias

The sample includes only entities that exist today. Funds that closed and companies that were delisted or went bankrupt are absent, so the measured return of the universe is biased upward and the measured risk downward . It is most severe in hedge fund databases, where reporting is voluntary and failed funds stop reporting, and in equity universes constructed from a current index membership list.

Remedy: use a point-in-time database that records index membership and security existence as of each historical date , including delisted securities and their delisting returns.

Look-ahead bias

Using information that was not available at the time of the decision. Three specific forms:

Reporting lag. Using a fiscal-year-end book value on the fiscal year-end date, when it was published three months later. The standard remedy is to lag fundamental data by a conservative period, often four to six months.

Restated data. Databases store the restated financials, not the figures as originally reported. A strategy that screens on the restated numbers uses knowledge of the restatement.

Index membership. Assuming a security was in the index before its addition was announced.

Data snooping (data mining)

Repeatedly testing variations on the same dataset until one appears significant. With enough attempts, a spurious result is certain: testing 20 independent specifications at a 5% significance level produces roughly one "significant" finding by chance alone.

Its symptoms and remedies:

Symptom: a strategy with many parameters that were each tuned to the sample; performance that collapses on any small change to the parameters.

Remedy 1: hold out an out-of-sample period that is never examined until the strategy is final.

Remedy 2: require a higher significance threshold when many tests have been run, and report the number of specifications tested.

Remedy 3: demand an economic rationale for the signal before accepting a statistical result. A relationship with no plausible mechanism is more likely to be noise.

Remedy 4: test on a different market or period entirely.

Related implementation problems that a clean backtest must also address: ignoring transaction costs and market impact, assuming execution at the closing price, ignoring short-sale constraints and borrow costs, and assuming unlimited capacity in illiquid securities.

4. Historical Scenario Analysis

Historical scenario analysis takes the actual returns of a defined past stress episode and applies them to the current portfolio, answering "what would today's positions have done in that event".

Common scenarios: the 1987 crash, the 1998 sovereign and hedge fund crisis, the 2000–2002 technology decline, the 2007–2009 global financial crisis, the 2013 taper episode, the 2020 pandemic shock, and the 2022 rate shock.

• Strength Limitation
• Uses internally consistent factor moves that actually occurred together Limited to events that have occurred
• Requires no distributional assumption Assets that did not exist then have no history in that episode
• Communicates well to committees and boards Correlations in the episode may not repeat
• Reveals cross-asset behaviour, including correlation breakdown Cannot address a novel risk

The specific insight this exercise supplies, and that VaR does not, is correlation behaviour under stress . A portfolio that looks diversified on average correlations may show that in 2008 every risky asset moved together, so the diversification the risk model reported was not available when it was needed.

5. Monte Carlo Simulation

Monte Carlo simulation generates many possible paths by drawing from assumed distributions for the underlying risk factors, then computes the outcome for each path and analyses the resulting distribution.

Procedure:

Specify the risk factors and the functional form linking them to portfolio value.

Choose distributions for each factor and specify the correlation structure between them.

Draw a large number of random scenarios — commonly tens or hundreds of thousands.

Reprice the portfolio in each scenario.

Analyse the distribution: mean, percentiles, value at risk, conditional value at risk, and the probability of specified outcomes.

Multivariate distributions matter. Drawing each factor independently understates joint tail risk. The standard approach specifies a covariance matrix, and more sophisticated implementations use copulas to model tail dependence — the empirical fact that extreme moves occur together more often than a multivariate normal assumption implies.

Distributional choice is the dominant assumption. Using a normal distribution understates tail frequency; using a Student's t with low degrees of freedom, or bootstrapping from historical returns, produces fatter tails and materially higher tail risk estimates.

Monte Carlo versus historical simulation

• Historical simulation Monte Carlo simulation
• Source of scenarios Actual past returns, resampled Draws from assumed distributions
• Distributional assumption None imposed; the sample's own shape is used Explicit and must be chosen
• Can generate unobserved events No — limited to what occurred Yes
• Captures actual joint behaviour Yes , as it occurred Only as far as the correlation structure specifies

Main weakness The sample may not contain the relevant stress; short samples are unstable Results are only as good as the assumed distributions; a false sense of precision

Computational cost Low High

Neither dominates. The disciplined practice is to run both: historical simulation for realism about joint behaviour, Monte Carlo to explore the space of events that has not yet occurred.

6. Sensitivity Analysis

Sensitivity analysis varies one input at a time — or a small number jointly — and reports how the conclusion changes. In a backtesting context it answers whether the result is robust or an artefact of a specific parameter choice.

What to vary:

Strategy parameters: the lookback window, the rebalancing frequency, the number of holdings, the signal threshold.

Cost assumptions: double the assumed transaction cost and see whether the strategy still works.

Distributional assumptions: re-run a Monte Carlo with a Student's t instead of a normal distribution and compare the tail estimates.

Sample period: split the history into sub-periods and test each.

The interpretive rule: a strategy whose performance survives reasonable variation in its parameters is more likely to reflect a genuine effect. A strategy that works at a 60-day lookback but fails at 50 and 70 days has been fitted to the sample , and the narrow peak is the signature of data snooping.

Illustration. A researcher reports a backtested information ratio of 1.4 for a momentum strategy using a 12-month formation period, a one-month holding period, and 10 basis points of assumed transaction cost. Sensitivity analysis shows:

• Variation Information ratio
• Base case 1.40
• 11-month formation 1.31
• 13-month formation 1.28
• Quarterly rebalancing 1.12
• Transaction cost of 30 basis points 0.42
• Excluding the smallest quintile by market capitalisation 0.55

The strategy is robust to the formation window but not to realistic transaction costs or to the exclusion of small, illiquid stocks. The honest conclusion is that the reported information ratio is largely a small-cap liquidity premium net of an implausible cost assumption — which is exactly the conclusion a well-built item set will ask for.

Level II traps in this module

Treating a single full-sample regression as a backtest; a backtest requires a rolling window .

Using restated financial data, or year-end data on the year-end date, both of which are look-ahead bias.

Reporting a Sharpe ratio for a negatively skewed strategy without a drawdown or Sortino measure.

Believing Monte Carlo is more reliable than historical simulation because it produces more scenarios; more scenarios drawn from a wrong distribution do not add information.

Concluding a strategy is validated because it survived one out-of-sample period after twenty in-sample specifications were tested.