Study guide
9.2 Discrete-Time Markov Chains: Transition Matrices and Steady-State Probabilities
FE Industrial and Systems Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview, CBT Strategy & Handbook Navigation
1.1 FE Industrial Exam Structure, CBT Logistics, and Pacing 1.2 Using the NCEES FE Reference Handbook and Approved Calculators
2 Chapter 2: Mathematics for Industrial Engineers
2.1 Analytic Geometry, Coordinate Systems, and Trigonometry 2.2 Calculus: Differential, Integral, and Series Approximations 2.3 Linear Algebra: Matrix Operations, Systems of Equations, and Vectors
3 Chapter 3: Engineering Sciences Core
3.1 Thermodynamics Fundamentals and Fluid Flow Principles 3.2 Statics, Dynamics, and Mechanics of Materials Core 3.3 Electrical Circuits, DC/AC Analysis, and Power in Industrial Systems
4 Chapter 4: Ethics and Professional Practice
4.1 NCEES Model Rules of Professional Conduct and Technical Society Ethics 4.2 Engineering Licensure, Public Welfare, and Professional Liability 4.3 Engineering Contracts, Intellectual Property, and Conflicts of Interest
5 Chapter 5: Engineering Economics
5.1 Time Value of Money, Discount Factors, and Equivalence 5.2 Evaluating Alternatives: Present Worth, Equivalent Annual Cost, and IRR 5.3 Incremental Rate of Return, Replacement Analysis, and Break-Even 5.4 Depreciation Methods, MACRS, Income Taxes, and Inflation
6 Chapter 6: Probability Foundations & Distributions
6.1 Basic Probability Concepts, Conditional Probability, and Bayes' Theorem 6.2 Discrete Probability Distributions 6.3 Continuous Probability Distributions
7 Chapter 7: Statistical Inference, Regression & Design of Experiments
7.1 Sampling Distributions, Central Limit Theorem, and Confidence Intervals 7.2 Hypothesis Testing: Means, Proportions, and Variances 7.3 Linear Regression, Correlation Analysis, and Design of Experiments (ANOVA)
8 Chapter 8: Data, Logic, Modeling & Linear Programming
8.1 Optimization Modeling: Decision Variables, Objectives, and Constraints 8.2 Linear Programming: Graphical Solutions and Simplex Fundamentals 8.3 Duality, Sensitivity Analysis, and Integer Linear Programming 8.4 Data, Logic Development, and Analytics
9 Chapter 9: Stochastic Models & Simulation
9.1 Queuing Theory: Little's Law, M/M/1, and Multi-Server Systems 9.2 Discrete-Time Markov Chains: Transition Matrices and Steady-State Probabilities 9.3 Discrete-Event Simulation Principles and Monte Carlo Modeling
10 Chapter 10: Engineering Management, Project Planning & Performance Measurement
10.1 Organizational Structure, Leadership, and Worker Motivation Theories 10.2 Project Management: WBS, Critical Path Method (CPM), PERT, and Crashing 10.3 Earned Value Management (EVM), Decision Trees, and Risk Preference 10.4 Performance Measurement: Productivity, KPIs, Wage Systems, and the Balanced Scorecard
11 Chapter 11: Manufacturing & Production Systems
11.1 Manufacturing Processes, Material Processing, and Tolerancing 11.2 Assembly Line Balancing, Takt Time, Cycle Time, and Throughput 11.3 Sustainable Manufacturing, Waste Minimization, and Life-Cycle Assessment
12 Chapter 12: Production Planning, Inventory Control & Sequencing
12.1 Demand Forecasting Methods and Forecast Error Tracking 12.2 Aggregate Planning, Master Production Scheduling (MPS), and MRP 12.3 Deterministic Inventory Models: EOQ, EPQ, and Safety Stock 12.4 Job Sequencing, Dispatching Rules, and Flow Shop Scheduling
13 Chapter 13: Lean Systems & Theory of Constraints
13.1 Lean Foundations: Value Stream Mapping, Seven Wastes, and 5S 13.2 Pull Systems, Kanban, Just-In-Time (JIT), and SMED Setup Reduction 13.3 Theory of Constraints: Bottleneck Identification and Drum-Buffer-Rope
14 Chapter 14: Facilities Planning & Material Handling
14.1 Facility Location Analysis: Qualitative Factor Rating and Centroid Method 14.2 Facility Layout Design and Flow Analysis 14.3 Material Handling Equipment, Warehousing, and AS/RS
15 Chapter 15: Supply Chain Design & Logistics
15.1 Supply Chain Network Architecture and Transportation Modes 15.2 Distribution Strategies, Cross-Docking, and Vehicle Routing 15.3 The Bullwhip Effect, Risk Pooling, and Supply Chain Coordination
16 Chapter 16: Human Factors & Cognitive Ergonomics
16.1 Cognitive Engineering, Human-Machine Interface, Displays, and Controls 16.2 Anthropometric Principles, Percentile Design, and Biomechanical Posture 16.3 Environmental Ergonomics: Industrial Noise, Lighting, and Heat Stress
17 Chapter 17: Physical Ergonomics & Workplace Safety
17.1 Physical Ergonomics: The NIOSH Manual Lifting Equation 17.2 Cumulative Trauma Disorders and Postural Assessment 17.3 Workplace Safety: OSHA Standards, Hazard Analysis, and Control Hierarchy
18 Chapter 18: Work Design & Measurement
18.1 Methods Engineering, Process Charting, and Principles of Motion Economy 18.2 Direct Stopwatch Time Study: Observed Time, Performance Rating, and Allowances 18.3 Predetermined Time Systems, Work Sampling, and Learning Curves
19 Chapter 19: Quality Management, Six Sigma & Analytical Tools
19.1 Quality Philosophies, Total Quality Management, and Cost of Quality 19.2 Six Sigma DMAIC Methodology and Quality Function Deployment 19.3 The Seven Basic Quality Tools, Root Cause Analysis, and Taguchi Loss Function
20 Chapter 20: Statistical Process Control
20.1 Statistical Process Control Principles, Variation, and Run Rules 20.2 Variable Control Charts: X-bar & R, X-bar & S, and Individual-Moving Range 20.3 Attribute Control Charts: Defectives (p, np) and Defects (c, u)
21 Chapter 21: Process Capability & Acceptance Sampling
21.1 Process Capability Indices and Defect Levels 21.2 Acceptance Sampling Concepts and Operating Characteristic Curves 21.3 Sampling Risks, AQL, LTPD, and Standard Sampling Plans
22 Chapter 22: Systems Engineering & Life-Cycle Management
22.1 The Systems Engineering V-Model, Stakeholder Requirements, and Functional Architecture 22.2 Configuration Management, Interface Control, and Verification & Validation 22.3 Life-Cycle Costing, Maintainability, and System Availability
23 Chapter 23: Reliability Engineering & Risk Analysis
23.1 Reliability Mathematics, Hazard Rates, and the Bathtub Curve 23.2 System Reliability Configurations: Series, Parallel, and k-out-of-n Systems 23.3 Failure Modes and Effects Analysis and Fault Tree Analysis
9.2 Discrete-Time Markov Chains: Transition Matrices and Steady-State Probabilities
Key Takeaways Explain This Topic AI
The Markov property (memoryless property) dictates that the future state probability distribution of a stochastic process depends strictly upon its current state and is conditionally independent of all past historical states: P(X_{n+1} = j | X_n = i, ..., X_0 = i_0) = P(X_{n+1} = j | X_n = i).
A transition probability matrix P = [p_ij] is row-stochastic, meaning all entries are non-negative (p_ij ≥ 0) and each row sums to unity (Σ_j p_ij = 1), representing complete conditional distributions over next-step states.
The Chapman-Kolmogorov equations establish that n-step transition probabilities are computed via matrix multiplication: P^(n) = P^n, allowing projection of state distributions over time via π^(n) = π^(0) * P^n.
States in a Markov chain are classified by accessibility, communication, recurrence (positive recurrent vs transient), absorbing nature (p_ii = 1), and periodicity; an irreducible, aperiodic, positive-recurrent chain is ergodic.
The steady-state (stationary) distribution π represents the long-run proportion of time the process spends in each state, determined by solving the linear system π * P = π subject to the normalization constraint Σ π_i = 1.
Last updated: September 2026
Discrete-time Markov chains (DTMCs) model dynamic stochastic systems that transition between countable operational states over sequential discrete time steps. In industrial and systems engineering, Markov chains are indispensable tools for modeling machine wear and degradation, predicting preventative maintenance intervals, analyzing warranty claims, evaluating inventory replenishment policies, and forecasting customer brand-switching market shares. The NCEES FE Reference Handbook outlines core Markovian definitions under its Industrial and Systems Engineering section. Mastery of transition matrices, Chapman-Kolmogorov equations, state classification, and steady-state vector solutions is essential for solving stochastic process problems on the FE exam.
1. The Markov Property and Stochastic Processes
A stochastic process is a mathematical collection of random variables { X n , n ∈ T } \{X_n, n \in T\} { X n , n ∈ T } indexed by time or step parameter n n n . In a discrete-time stochastic process with discrete state space S = { 0 , 1 , 2 , … , M } S = \{0, 1, 2, \dots, M\} S = { 0 , 1 , 2 , … , M } , X n X_n X n represents the state of the system at step n n n .
The Markov Property (Memoryless Property)
A discrete-time stochastic process possesses the Markov property if the conditional probability distribution of the future state X n + 1 X_{n+1} X n + 1 depends solely upon the present state X n X_n X n and is completely conditionally independent of the past states X n − 1 , X n − 2 , … , X 0 X_{n-1}, X_{n-2}, \dots, X_0 X n − 1 , X n − 2 , … , X 0 :
P ( X n + 1 = j ∣ X n = i , X n − 1 = i n − 1 , … , X 0 = i 0 ) = P ( X n + 1 = j ∣ X n = i ) = p i j P(X_{n+1} = j \mid X_n = i, X_{n-1} = i_{n-1}, \dots, X_0 = i_0) = P(X_{n+1} = j \mid X_n = i) = p_{ij} P ( X n + 1 = j ∣ X n = i , X n − 1 = i n − 1 , … , X 0 = i 0 ) = P ( X n + 1 = j ∣ X n = i ) = p ij
Informally: Given the present, the future is conditionally independent of the past. Historical trajectories that led the system to its current state provide no additional predictive power regarding its next transition.
Time-Homogeneity
A Markov chain is time-homogeneous (or stationary) if the transition probabilities between states remain constant across all time steps n n n :
P ( X n + 1 = j ∣ X n = i ) = P ( X 1 = j ∣ X 0 = i ) = p i j for all n ≥ 0 P(X_{n+1} = j \mid X_n = i) = P(X_1 = j \mid X_0 = i) = p_{ij} \quad \text{for all } n \ge 0 P ( X n + 1 = j ∣ X n = i ) = P ( X 1 = j ∣ X 0 = i ) = p ij for all n ≥ 0
All standard FE exam Markov problems assume time-homogeneous chains unless explicitly noted otherwise.
- 2. State Space, Transition Matrices, and Transition Diagrams
- The Transition Probability Matrix ( P P P )
For a Markov chain with M M M discrete states S = { 1 , 2 , … , M } S = \{1, 2, \dots, M\} S = { 1 , 2 , … , M } , the transition probabilities p i j p_{ij} p ij are arranged into a square M × M M \times M M × M transition probability matrix P P P :
P = [ p 11 p 12 … p 1 M p 21 p 22 … p 2 M ⋮ ⋮ ⋱ ⋮ p M 1 p M 2 … p M M ] P = \begin{bmatrix} p_{11} & p_{12} & \dots & p_{1M} \\ p_{21} & p_{22} & \dots & p_{2M} \\ \vdots & \vdots & \ddots & \vdots \\ p_{M1} & p_{M2} & \dots & p_{MM} \end{bmatrix} P = p 11 p 21 ⋮ p M 1 p 12 p 22 ⋮ p M 2 … … ⋱ … p 1 M p 2 M ⋮ p MM
Where p i j p_{ij} p ij represents the conditional probability of transitioning from current state i i i to next state j j j in one step:
p i j = P ( X n + 1 = j ∣ X n = i ) p_{ij} = P(X_{n+1} = j \mid X_n = i) p ij = P ( X n + 1 = j ∣ X n = i )
Fundamental Properties of Matrix P P P
Non-negativity : Every element is a valid probability: p i j ≥ 0 p_{ij} \ge 0 p ij ≥ 0 for all i , j i, j i , j .
Row-Stochastic Condition : Each row i i i represents a complete conditional probability distribution over all mutually exclusive and collectively exhaustive destination states. Therefore, each row must sum to exactly 1.0 : ∑ j = 1 M p i j = 1 for each row i = 1 , 2 , … , M \sum_{j=1}^M p_{ij} = 1 \quad \text{for each row } i = 1, 2, \dots, M ∑ j = 1 M p ij = 1 for each row i = 1 , 2 , … , M (Note: Columns do not need to sum to 1.0; a matrix where both rows and columns sum to 1.0 is doubly stochastic).
State Transition Diagrams
A Markov chain can be visualized as a directed graph where:
Nodes represent the distinct physical states { 1 , 2 , … , M } \{1, 2, \dots, M\} { 1 , 2 , … , M } .
Directed Arcs from node i i i to node j j j represent permissible transitions with non-zero probability p i j > 0 p_{ij} > 0 p ij > 0 .
Self-Loops from node i i i back to node i i i represent the probability p i i p_{ii} p ii of remaining in the current state. ┌──────────── p12 ────────────┐
- │ ▼
- ┌───────────┐ ┌───────────┐
- p11 ↺ ─── │ State 1 │ ◄─────────────── │ State 2 │ ─── ↻ p22
- └───────────┘ p21 └───────────┘
- │ ▲
- │ p23 │
- └──────────────┬──────────────┘
- ▼
- ┌───────────┐
- p33 ↺ ──── │ State 3 │ ───> p31 (to State 1)
- └───────────┘
- 3. Multi-Step Transitions and Chapman-Kolmogorov Equations
To predict the state of the system multiple steps into the future, we evaluate the n n n -step transition probability , denoted p i j ( n ) p_{ij}^{(n)} p ij ( n ) :
p i j ( n ) = P ( X m + n = j ∣ X m = i ) p_{ij}^{(n)} = P(X_{m+n} = j \mid X_m = i) p ij ( n ) = P ( X m + n = j ∣ X m = i )
Chapman-Kolmogorov Equations
The probability of transitioning from state i i i to state j j j in n + m n + m n + m steps equals the sum of probabilities across all possible intermediate states k k k occupied after n n n steps:
p i j ( n + m ) = ∑ k = 1 M p i k ( n ) p k j ( m ) p_{ij}^{(n+m)} = \sum_{k=1}^M p_{ik}^{(n)} p_{kj}^{(m)} p ij ( n + m ) = ∑ k = 1 M p ik ( n ) p kj ( m )
In matrix notation, this relationship reduces to ordinary matrix multiplication:
P ( n + m ) = P ( n ) P ( m ) P^{(n+m)} = P^{(n)} P^{(m)} P ( n + m ) = P ( n ) P ( m )
By induction, the matrix of n n n -step transition probabilities is simply the transition matrix P P P raised to the n n n -th power:
P ( n ) = P n = P × P × ⋯ × P ⏟ n times P^{(n)} = P^n = \underbrace{P \times P \times \dots \times P}_{n \text{ times}} P ( n ) = P n = n times P × P × ⋯ × P
Unconditional State Probability Propagation
Let π ( 0 ) = [ π 1 ( 0 ) , π 2 ( 0 ) , … , π M ( 0 ) ] \pi^{(0)} = [\pi_1^{(0)}, \pi_2^{(0)}, \dots, \pi_M^{(0)}] π ( 0 ) = [ π 1 ( 0 ) , π 2 ( 0 ) , … , π M ( 0 ) ] be a row vector representing the initial probability distribution over states at time n = 0 n = 0 n = 0 . The probability distribution across states at time step n n n is given by:
π ( n ) = π ( 0 ) P n \pi^{(n)} = \pi^{(0)} P^n π ( n ) = π ( 0 ) P n
Where π j ( n ) = P ( X n = j ) \pi_j^{(n)} = P(X_n = j) π j ( n ) = P ( X n = j ) is the unconditional probability that the system occupies state j j j at step n n n .
4. Classification of States in Markov Chains
Understanding the structural classification of states is necessary to determine whether a Markov chain reaches a unique steady-state equilibrium.
State Property Mathematical Definition Physical Interpretation
Accessible ( i → j i \to j i → j ) p i j ( n ) > 0 p_{ij}^{(n)} > 0 p ij ( n ) > 0 for some n ≥ 0 n \ge 0 n ≥ 0 State j j j can eventually be reached from state i i i
Communicating ( i ↔ j i \leftrightarrow j i ↔ j ) i → j i \to j i → j and j → i j \to i j → i States can reach each other mutually (equivalence relation)
Irreducible All states communicate with each other System consists of a single communicating class; no traps
Absorbing p i i = 1 p_{ii} = 1 p ii = 1 and p i j = 0 p_{ij} = 0 p ij = 0 for all j ≠ i j \neq i j = i Once entered, the state can never be left (e.g., catastrophic scrap/failure)
Transient Probability of return f i i < 1 f_{ii} < 1 f ii < 1 System will eventually exit the state forever (finite number of visits)
Recurrent (Persistent) Probability of return f i i = 1 f_{ii} = 1 f ii = 1 System is guaranteed to return to the state infinitely often
Periodic (Period d > 1 d > 1 d > 1 ) d ( i ) = gcd { n ≥ 1 : p i i ( n ) > 0 } > 1 d(i) = \gcd\{n \ge 1 : p_{ii}^{(n)} > 0\} > 1 d ( i ) = g cd { n ≥ 1 : p ii ( n ) > 0 } > 1 Return to state i i i is only possible in multiples of d d d steps
Aperiodic ( d = 1 d = 1 d = 1 ) gcd { n ≥ 1 : p i i ( n ) > 0 } = 1 \gcd\{n \ge 1 : p_{ii}^{(n)} > 0\} = 1 g cd { n ≥ 1 : p ii ( n ) > 0 } = 1 Return timing is not locked into fixed integer cycles
Ergodic Recurrent, aperiodic, and finite mean recurrence time Possesses a unique stationary steady-state distribution
Exam Key : If any diagonal entry p i i > 0 p_{ii} > 0 p ii > 0 (a state has a self-loop), that state is automatically aperiodic ( d = 1 d = 1 d = 1 ). Furthermore, if an irreducible Markov chain has at least one aperiodic state, the entire chain is aperiodic.
5. Stationary and Steady-State Probability Distributions
In long-run industrial operations, engineers are rarely interested in transient fluctuations from a specific starting state; they require the long-run proportion of time the process spends in each state.
Mathematical Definition of Stationary Distribution
A row vector π = [ π 1 , π 2 , … , π M ] \pi = [\pi_1, \pi_2, \dots, \pi_M] π = [ π 1 , π 2 , … , π M ] is called a stationary (steady-state) distribution if it satisfies:
π P = π \pi P = \pi π P = π ∑ i = 1 M π i = 1 , with π i ≥ 0 for all i \sum_{i=1}^M \pi_i = 1, \quad \text{with } \pi_i \ge 0 \text{ for all } i ∑ i = 1 M π i = 1 , with π i ≥ 0 for all i
In expanded linear system form:
π j = ∑ i = 1 M π i p i j for j = 1 , 2 , … , M \pi_j = \sum_{i=1}^M \pi_i p_{ij} \quad \text{for } j = 1, 2, \dots, M π j = ∑ i = 1 M π i p ij for j = 1 , 2 , … , M π 1 + π 2 + ⋯ + π M = 1 \pi_1 + \pi_2 + \dots + \pi_M = 1 π 1 + π 2 + ⋯ + π M = 1
Physical and Long-Run Interpretation
For an irreducible, aperiodic, finite Markov chain (an ergodic chain):
lim n → ∞ P n = [ π 1 π 2 … π M π 1 π 2 … π M ⋮ ⋮ ⋱ ⋮ π 1 π 2 … π M ] \lim_{n \to \infty} P^n = \begin{bmatrix} \pi_1 & \pi_2 & \dots & \pi_M \\ \pi_1 & \pi_2 & \dots & \pi_M \\ \vdots & \vdots & \ddots & \vdots \\ \pi_1 & \pi_2 & \dots & \pi_M \end{bmatrix} lim n → ∞ P n = π 1 π 1 ⋮ π 1 π 2 π 2 ⋮ π 2 … … ⋱ … π M π M ⋮ π M
Every row of P n P^n P n converges to the exact same steady-state vector π \pi π . This implies:
lim n → ∞ P ( X n = j ∣ X 0 = i ) = π j \lim_{n \to \infty} P(X_n = j \mid X_0 = i) = \pi_j lim n → ∞ P ( X n = j ∣ X 0 = i ) = π j , completely independent of the starting state i i i .
π j \pi_j π j represents the long-run fraction of time the system spends in state j j j .
The mean recurrence time (expected number of steps between consecutive visits to state j j j ) is: μ j j = 1 π j \mu_{jj} = \frac{1}{\pi_j} μ jj = π j 1
Systematic Solution Procedure for the FE Exam
The matrix equation π P = π \pi P = \pi π P = π can be rewritten as the homogeneous linear system π ( P − I ) = 0 \pi(P - I) = 0 π ( P − I ) = 0 . Because the rows of P P P sum to 1.0 1.0 1.0 , the matrix ( P − I ) (P - I) ( P − I ) is singular (its determinant is zero), meaning the system π ( P − I ) = 0 \pi(P - I) = 0 π ( P − I ) = 0 has infinitely many dependent solutions. To find the unique solution:
Write out the M M M linear equations from π P = π \pi P = \pi π P = π .
Discard any one redundant equation from the set.
Replace the discarded equation with the normalization equation : ∑ i = 1 M π i = 1 \sum_{i=1}^M \pi_i = 1 ∑ i = 1 M π i = 1 .
Solve the resulting system of M M M independent linear equations for π 1 , π 2 , … , π M \pi_1, \pi_2, \dots, \pi_M π 1 , π 2 , … , π M .
- 6. Industrial Applications of Markov Chains
- Machine Reliability and Preventative Maintenance
Equipment condition degrades stochastically over time. A common model discretizes machine health into:
- State 1: Operational / Excellent (new tool, tight tolerances)
- State 2: Degraded / Acceptable (surface finish drifting, minor chatter)
- State 3: Failed / Out of Specification (broken tool, machine halted)
By evaluating the steady-state probability vector π \pi π , reliability engineers compute:
- Long-run machine availability: A = π 1 + π 2 A = \pi_1 + \pi_2 A = π 1 + π 2
- Long-run machine downtime: D = π 3 D = \pi_3 D = π 3
- Average hourly maintenance cost: Cost = ∑ i π i C i \text{Cost} = \sum_{i} \pi_i C_i Cost = ∑ i π i C i
- Customer Brand Switching
Marketing and operations teams model consumer brand loyalty using Markov chains. If state 1 1 1 represents purchasing Company A's product and state 2 2 2 represents purchasing Competitor B's product, the steady-state probabilities π 1 \pi_1 π 1 and π 2 \pi_2 π 2 dictate long-run equilibrium market shares.
Inventory Replenishment Policies
In ( s , S ) (s, S) ( s , S ) inventory control, when inventory drops to reorder point s s s , an order is triggered to restore stock to target S S S . Modeling stock levels as a Markov chain allows exact calculation of stockout probabilities π 0 \pi_0 π 0 and average holding costs.
- 7. Step-by-Step Worked Engineering Examples
- Worked Example 9.2.1: Three-State CNC Degradation and Availability Analysis
Problem : An automated CNC mill is inspected at the start of each 8-hour shift and categorized into one of three states:
- State 1: Optimal operating condition
- State 2: Minor tool wear (operates with acceptable tolerances)
- State 3: Failed (shut down; requires emergency maintenance)
The one-shift transition probability matrix is:
P = [ 0.80 0.15 0.05 0.00 0.70 0.30 0.90 0.00 0.10 ] P = \begin{bmatrix} 0.80 & 0.15 & 0.05 \\ 0.00 & 0.70 & 0.30 \\ 0.90 & 0.00 & 0.10 \end{bmatrix} P = 0.80 0.00 0.90 0.15 0.70 0.00 0.05 0.30 0.10
Verify that P P P is a valid transition probability matrix.
Formulate the steady-state equations and determine the long-run probability distribution π = [ π 1 , π 2 , π 3 ] \pi = [\pi_1, \pi_2, \pi_3] π = [ π 1 , π 2 , π 3 ] .
What is the long-run availability of the CNC mill (percentage of shifts it can operate)?
What is the mean time between failures (MTBF in shifts)?
Solution :
Verify Matrix Properties :
All entries p i j ≥ 0 p_{ij} \ge 0 p ij ≥ 0 .
- Row 1: 0.80 + 0.15 + 0.05 = 1.00 0.80 + 0.15 + 0.05 = 1.00 0.80 + 0.15 + 0.05 = 1.00
- Row 2: 0.00 + 0.70 + 0.30 = 1.00 0.00 + 0.70 + 0.30 = 1.00 0.00 + 0.70 + 0.30 = 1.00
- Row 3: 0.90 + 0.00 + 0.10 = 1.00 0.90 + 0.00 + 0.10 = 1.00 0.90 + 0.00 + 0.10 = 1.00
All rows sum to 1.00 1.00 1.00 . P P P is a valid row-stochastic matrix.
Formulate and Solve Steady-State Equations :
Set up π P = π \pi P = \pi π P = π : [ π 1 , π 2 , π 3 ] [ 0.80 0.15 0.05 0.00 0.70 0.30 0.90 0.00 0.10 ] = [ π 1 , π 2 , π 3 ] [\pi_1, \pi_2, \pi_3] \begin{bmatrix} 0.80 & 0.15 & 0.05 \\ 0.00 & 0.70 & 0.30 \\ 0.90 & 0.00 & 0.10 \end{bmatrix} = [\pi_1, \pi_2, \pi_3] [ π 1 , π 2 , π 3 ] 0.80 0.00 0.90 0.15 0.70 0.00 0.05 0.30 0.10 = [ π 1 , π 2 , π 3 ]
Multiply components: ( 1 ) π 1 = 0.80 π 1 + 0.90 π 3 (1) \quad \pi_1 = 0.80 \pi_1 + 0.90 \pi_3 ( 1 ) π 1 = 0.80 π 1 + 0.90 π 3 ( 2 ) π 2 = 0.15 π 1 + 0.70 π 2 (2) \quad \pi_2 = 0.15 \pi_1 + 0.70 \pi_2 ( 2 ) π 2 = 0.15 π 1 + 0.70 π 2 ( 3 ) π 3 = 0.05 π 1 + 0.30 π 2 + 0.10 π 3 (3) \quad \pi_3 = 0.05 \pi_1 + 0.30 \pi_2 + 0.10 \pi_3 ( 3 ) π 3 = 0.05 π 1 + 0.30 π 2 + 0.10 π 3 ( 4 ) π 1 + π 2 + π 3 = 1 ( Normalization ) (4) \quad \pi_1 + \pi_2 + \pi_3 = 1 \quad (\text{Normalization}) ( 4 ) π 1 + π 2 + π 3 = 1 ( Normalization )
Express all variables in terms of π 1 \pi_1 π 1 :
From equation (2): π 2 − 0.70 π 2 = 0.15 π 1 ⟹ 0.30 π 2 = 0.15 π 1 ⟹ π 2 = 0.15 0.30 π 1 = 0.50 π 1 \pi_2 - 0.70 \pi_2 = 0.15 \pi_1 \implies 0.30 \pi_2 = 0.15 \pi_1 \implies \pi_2 = \frac{0.15}{0.30} \pi_1 = 0.50 \pi_1 π 2 − 0.70 π 2 = 0.15 π 1 ⟹ 0.30 π 2 = 0.15 π 1 ⟹ π 2 = 0.30 0.15 π 1 = 0.50 π 1
From equation (1): π 1 − 0.80 π 1 = 0.90 π 3 ⟹ 0.20 π 1 = 0.90 π 3 ⟹ π 3 = 0.20 0.90 π 1 = 2 9 π 1 \pi_1 - 0.80 \pi_1 = 0.90 \pi_3 \implies 0.20 \pi_1 = 0.90 \pi_3 \implies \pi_3 = \frac{0.20}{0.90} \pi_1 = \frac{2}{9} \pi_1 π 1 − 0.80 π 1 = 0.90 π 3 ⟹ 0.20 π 1 = 0.90 π 3 ⟹ π 3 = 0.90 0.20 π 1 = 9 2 π 1
Substitute into the normalization equation (4): π 1 + 0.50 π 1 + 2 9 π 1 = 1 \pi_1 + 0.50 \pi_1 + \frac{2}{9} \pi_1 = 1 π 1 + 0.50 π 1 + 9 2 π 1 = 1 π 1 ( 1 + 1 2 + 2 9 ) = 1 ⟹ π 1 ( 18 + 9 + 4 18 ) = 1 ⟹ π 1 ( 31 18 ) = 1 \pi_1 \left( 1 + \frac{1}{2} + \frac{2}{9} \right) = 1 \implies \pi_1 \left( \frac{18 + 9 + 4}{18} \right) = 1 \implies \pi_1 \left( \frac{31}{18} \right) = 1 π 1 ( 1 + 2 1 + 9 2 ) = 1 ⟹ π 1 ( 18 18 + 9 + 4 ) = 1 ⟹ π 1 ( 18 31 ) = 1 π 1 = 18 31 ≈ 0.5806 ( 58.06 % ) \pi_1 = \frac{18}{31} \approx 0.5806 \quad (58.06\%) π 1 = 31 18 ≈ 0.5806 ( 58.06% )
Compute π 2 \pi_2 π 2 and π 3 \pi_3 π 3 : π 2 = 0.50 ( 18 31 ) = 9 31 ≈ 0.2903 ( 29.03 % ) \pi_2 = 0.50 \left( \frac{18}{31} \right) = \frac{9}{31} \approx 0.2903 \quad (29.03\%) π 2 = 0.50 ( 31 18 ) = 31 9 ≈ 0.2903 ( 29.03% ) π 3 = 2 9 ( 18 31 ) = 4 31 ≈ 0.1290 ( 12.90 % ) \pi_3 = \frac{2}{9} \left( \frac{18}{31} \right) = \frac{4}{31} \approx 0.1290 \quad (12.90\%) π 3 = 9 2 ( 31 18 ) = 31 4 ≈ 0.1290 ( 12.90% )
Verification: 18 / 31 + 9 / 31 + 4 / 31 = 31 / 31 = 1.0000 18/31 + 9/31 + 4/31 = 31/31 = 1.0000 18/31 + 9/31 + 4/31 = 31/31 = 1.0000 .
Compute Long-Run Machine Availability :
The mill operates when it is in State 1 or State 2: Availability = π 1 + π 2 = 18 31 + 9 31 = 27 31 ≈ 0.8710 ( 87.10 % ) \text{Availability} = \pi_1 + \pi_2 = \frac{18}{31} + \frac{9}{31} = \frac{27}{31} \approx 0.8710 \quad (87.10\%) Availability = π 1 + π 2 = 31 18 + 31 9 = 31 27 ≈ 0.8710 ( 87.10% )
Compute Mean Recurrence Time for Failure (MTBF) : μ 33 = 1 π 3 = 1 4 / 31 = 31 4 = 7.75 shifts \mu_{33} = \frac{1}{\pi_3} = \frac{1}{4/31} = \frac{31}{4} = 7.75 \text{ shifts} μ 33 = π 3 1 = 4/31 1 = 4 31 = 7.75 shifts
Worked Example 9.2.2: Two-Step Transition and State Propagation
Problem : An automated assembly tool exhibits two operational states: State 1 (Calibrated) and State 2 (Out of Calibration). The transition matrix between production batches is:
P = [ 0.90 0.10 0.40 0.60 ] P = \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix} P = [ 0.90 0.40 0.10 0.60 ]
If the tool is known to be Calibrated at batch 0 ( π ( 0 ) = [ 1.0 , 0.0 ] \pi^{(0)} = [1.0, 0.0] π ( 0 ) = [ 1.0 , 0.0 ] ), find the probability that the tool is Out of Calibration at batch 2 ( p 12 ( 2 ) p_{12}^{(2)} p 12 ( 2 ) ).
Find the long-run steady-state probability distribution π \pi π .
Solution :
Compute Two-Step Transition Matrix P 2 P^2 P 2 : P 2 = P × P = [ 0.90 0.10 0.40 0.60 ] [ 0.90 0.10 0.40 0.60 ] P^2 = P \times P = \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix} \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix} P 2 = P × P = [ 0.90 0.40 0.10 0.60 ] [ 0.90 0.40 0.10 0.60 ] p 11 ( 2 ) = ( 0.90 ) ( 0.90 ) + ( 0.10 ) ( 0.40 ) = 0.81 + 0.04 = 0.85 p_{11}^{(2)} = (0.90)(0.90) + (0.10)(0.40) = 0.81 + 0.04 = 0.85 p 11 ( 2 ) = ( 0.90 ) ( 0.90 ) + ( 0.10 ) ( 0.40 ) = 0.81 + 0.04 = 0.85 p 12 ( 2 ) = ( 0.90 ) ( 0.10 ) + ( 0.10 ) ( 0.60 ) = 0.09 + 0.06 = 0.15 p_{12}^{(2)} = (0.90)(0.10) + (0.10)(0.60) = 0.09 + 0.06 = 0.15 p 12 ( 2 ) = ( 0.90 ) ( 0.10 ) + ( 0.10 ) ( 0.60 ) = 0.09 + 0.06 = 0.15 p 21 ( 2 ) = ( 0.40 ) ( 0.90 ) + ( 0.60 ) ( 0.40 ) = 0.36 + 0.24 = 0.60 p_{21}^{(2)} = (0.40)(0.90) + (0.60)(0.40) = 0.36 + 0.24 = 0.60 p 21 ( 2 ) = ( 0.40 ) ( 0.90 ) + ( 0.60 ) ( 0.40 ) = 0.36 + 0.24 = 0.60 p 22 ( 2 ) = ( 0.40 ) ( 0.10 ) + ( 0.60 ) ( 0.60 ) = 0.04 + 0.36 = 0.40 p_{22}^{(2)} = (0.40)(0.10) + (0.60)(0.60) = 0.04 + 0.36 = 0.40 p 22 ( 2 ) = ( 0.40 ) ( 0.10 ) + ( 0.60 ) ( 0.60 ) = 0.04 + 0.36 = 0.40 P 2 = [ 0.85 0.15 0.60 0.40 ] P^2 = \begin{bmatrix} 0.85 & 0.15 \\ 0.60 & 0.40 \end{bmatrix} P 2 = [ 0.85 0.60 0.15 0.40 ]
The probability that the tool is in State 2 after 2 batches given it started in State 1 is: P ( X 2 = 2 ∣ X 0 = 1 ) = p 12 ( 2 ) = 0.15 ( 15.0 % ) P(X_2 = 2 \mid X_0 = 1) = p_{12}^{(2)} = 0.15 \quad (15.0\%) P ( X 2 = 2 ∣ X 0 = 1 ) = p 12 ( 2 ) = 0.15 ( 15.0% )
Compute Steady-State Probabilities :
π P = π ⟹ [ π 1 , π 2 ] [ 0.90 0.10 0.40 0.60 ] = [ π 1 , π 2 ] \pi P = \pi \implies [\pi_1, \pi_2] \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix} = [\pi_1, \pi_2] π P = π ⟹ [ π 1 , π 2 ] [ 0.90 0.40 0.10 0.60 ] = [ π 1 , π 2 ]
Equation 1: π 1 = 0.90 π 1 + 0.40 π 2 ⟹ 0.10 π 1 = 0.40 π 2 ⟹ π 1 = 4 π 2 \pi_1 = 0.90 \pi_1 + 0.40 \pi_2 \implies 0.10 \pi_1 = 0.40 \pi_2 \implies \pi_1 = 4 \pi_2 π 1 = 0.90 π 1 + 0.40 π 2 ⟹ 0.10 π 1 = 0.40 π 2 ⟹ π 1 = 4 π 2
Normalization: π 1 + π 2 = 1 ⟹ 4 π 2 + π 2 = 1 ⟹ 5 π 2 = 1 \pi_1 + \pi_2 = 1 \implies 4 \pi_2 + \pi_2 = 1 \implies 5 \pi_2 = 1 π 1 + π 2 = 1 ⟹ 4 π 2 + π 2 = 1 ⟹ 5 π 2 = 1 π 2 = 1 5 = 0.20 ( 20.0 % ) , π 1 = 0.80 ( 80.0 % ) \pi_2 = \frac{1}{5} = 0.20 \quad (20.0\%), \quad \pi_1 = 0.80 \quad (80.0\%) π 2 = 5 1 = 0.20 ( 20.0% ) , π 1 = 0.80 ( 80.0% )
In the long run, the tool is calibrated 80 % 80\% 80% of the time and out of calibration 20 % 20\% 20% of the time.
8. NCEES Reference Handbook Tips & Realistic Exam Traps
Row Vector vs. Column Vector Trap : The steady-state system is π P = π \pi P = \pi π P = π , where π \pi π is a row vector multiplying P P P from the left. Do not write P π = π P \pi = \pi P π = π ; that formulation treats π \pi π as a column vector and will multiply columns incorrectly unless you transpose P P P ( P T π T = π T P^T \pi^T = \pi^T P T π T = π T ).
Omitting the Normalization Equation : Attempting to solve π ( P − I ) = 0 \pi(P - I) = 0 π ( P − I ) = 0 directly using matrix inversion ( [ P − I ] − 1 [P - I]^{-1} [ P − I ] − 1 ) will result in a divide-by-zero error on your calculator because ( P − I ) (P - I) ( P − I ) is singular. You must manually replace one of the dependent rows with the normalization equation ∑ π i = 1 \sum \pi_i = 1 ∑ π i = 1 .
Row Sum Verification : Before doing any Markov calculations, always sum the elements of each row in the given matrix. If a row does not sum to 1.0 1.0 1.0 , you may have transcribed a column as a row, or the problem statement defines transitions differently.
Absorbing State Distractor : A state with p i i = 1.0 p_{ii} = 1.0 p ii = 1.0 is absorbing: probability that reaches it never leaves. If a chain has exactly one absorbing state reachable from every transient state, its long-run probability is 1.0 1.0 1.0 . If the chain has two or more absorbing states (as in the 4-state chain earlier in this section), the long-run distribution splits among them according to the starting state and requires absorption-probability analysis, not the ergodic π P = π \pi P = \pi π P = π method.