2.5 Markov Chains: Transition Matrices, Steady-State Probabilities & Absorbing States

PE Industrial and Systems Exam

Track your progress as you read

Finished sections get a check mark.

1 Exam Overview, Strategy & Systems Foundations

1.1 PE Industrial Exam Format, Official Specification & Reference Handbook Strategy 1.2 System Analysis Tools, Process Definition, Requirements Engineering & QFD 1.3 Performance Measures: Output, Productivity, OEE, Cost & Environmental Sustainability 1.4 Process Types (Discrete, Continuous, Manufacturing, Service) & Value Analysis/Engineering

2 Optimization, Queueing, Simulation, Networks & Markov Chains

2.1 Linear Programming Formulation, Graphical Solutions & Sensitivity Analysis 2.2 Queueing Theory Models (M/M/1, M/M/s) & Little's Law Applications 2.3 Discrete-Event Simulation, Random Variates & Model Validation 2.4 Graph Theory & Network Models: Shortest Path, Minimal Spanning Tree & Maximum Flow 2.5 Markov Chains: Transition Matrices, Steady-State Probabilities & Absorbing States 2.6 Decision Trees, Expected Value & Bottleneck Analysis (Theory of Constraints)

3 Project Management & Planning

3.1 Project Scheduling with CPM and PERT: Critical Path, Float, Probability & Crashing 3.2 Project Planning & Control: Scope and WBS, Gantt Charts, Critical Chain (CCPM), Risk & Earned Value

4 Facility Layout, Space, Capacity & Material Handling

4.1 Systematic Layout Planning (SLP) & Activity Relationship (REL) Analysis 4.2 Workstation Design, Aisle Sizing & Space Requirements Planning 4.3 Capacity Analysis: Machine Requirements, Scrap Allowances & Personnel Requirements 4.4 Material Handling Equipment Selection, Unit Loads & Handling Principles

5 Site Location, Warehousing & Facility Economics

5.1 Facility Location Models: Center of Gravity & Minisum / Minimax Distance 5.2 Warehousing Systems, Storage Assignment & Order Picking Operations 5.3 Facility Life-Cycle Economics, Replacement Analysis & Rearrangement Costs

6 Demand Forecasting, Production Planning, MRP & JIT

6.1 Demand Forecasting Methods: Moving Averages, Exponential Smoothing & Error Metrics 6.2 Aggregate Planning Strategies & Master Production Scheduling (MPS) 6.3 Material Requirements Planning (MRP), Bill of Materials & Capacity Planning 6.4 Just-in-Time & Lean Production Planning: Pull Systems, Kanban Sizing, Heijunka & SMED

7 Inventory Systems, Scheduling, Logistics & Transportation Modes

7.1 Deterministic Inventory Control: Classical EOQ, EPQ & Quantity Discounts 7.2 Stochastic Inventory Management: Safety Stock, Continuous / Periodic Review & Newsvendor 7.3 Production Sequencing, Dispatching Rules & Johnson's Algorithm 7.4 Supply Chain Logistics, Transportation Simplex & Distribution Network Design 7.5 Transportation Modes, Freight Economics & the Transshipment Model

8 Engineering Economics, Break-Even Analysis & Costing Systems

8.1 Equivalence, Cash Flow Analysis & Discounted Cash Flow Metrics (PW, FW, EAC) 8.2 Rate of Return Analysis (IRR, Incremental IRR) & Benefit-Cost (B/C) Ratios 8.3 Asset Depreciation (MACRS, Straight-Line), Tax Effects & Replacement Decisions 8.4 Break-Even Analysis, Technical Capability Assessment & Costing Systems (Traditional, ABC & Standard Costs)

9 Methods Engineering, Work Measurement, Learning Curves, Line Balancing & Process Planning

9.1 Methods Analysis, Process Charting & Motion Study (Therbligs, 5S) 9.2 Worker–Machine (Man–Machine) Charts, Multiple Activity Analysis & Synchronous Servicing 9.3 Time Study Fundamentals, Performance Rating & Allowance Factors 9.4 Predetermined Motion Time Systems (PMTS, MTM, MOST) & Work Sampling 9.5 Learning Curves: Unit and Cumulative-Average Models, Learning Rates & Applications 9.6 Assembly Line Balancing, Precedence Constraints & Efficiency Heuristics 9.7 Manufacturing & Service Process Planning: Operation Selection, Sequencing, Route Sheets, Fixturing & Mistake Proofing

10 Ergonomics, Human Capacity, Occupational Safety & DART Rates

10.1 Anthropometry, Percentile Design, Workstation Ergonomics & Link Analysis 10.2 Industrial Biomechanics, the NIOSH Lifting Equation & Lifting Aids 10.3 Limits of Human Capacity & Risk-Factor Tools: Work Physiology, Strength, Range of Motion, Vision, RULA & Slip Resistance 10.4 Environmental Ergonomics (Noise, Heat, Illumination), Human-Machine Systems & OSHA Safety 10.5 Safety Codes, Standards & Voluntary Guidelines (OSHA, ANSI, NIOSH, MIL-STD) and DART Rate Calculations

11 Statistical Process Control & Process Capability

11.1 Statistical Process Control for Variables (X-bar/R and X-bar/s Charts) 11.2 Statistical Process Control for Attributes (p, np, c, and u Charts) 11.3 Process Capability Indices (Cp, Cpk, Cpm, Pp, Ppk) & Sigma Levels

12 Design of Experiments, Acceptance Sampling & Quality Systems

12.1 Design of Experiments: 2^k Factorials, ANOVA & Taguchi Robust Design 12.2 Acceptance Sampling: OC Curves, AQL, LTPD, AOQL, Z1.4/MIL-STD-105E & Dodge-Romig 12.3 Quality Systems (Deming, TQM, ISO 9000, Benchmarking), Root Cause Analysis & DMAIC

13 Reliability Engineering & Maintenance Systems

13.1 Reliability Modeling, Failure Rates (lambda) & Life Distributions (Exponential, Weibull) 13.2 System Reliability Architecture: Series, Parallel, k-out-of-n & Redundancy 13.3 Equipment Maintenance Engineering, Availability (A), MTBF/MTTR & RCM

2.5 Markov Chains: Transition Matrices, Steady-State Probabilities & Absorbing States

Key Takeaways Explain This Topic AI

A Markov chain's next state depends only on its current state, and each row of the transition matrix P sums to 1.

The state distribution after n periods is the starting distribution times P raised to the n-th power.

Steady-state probabilities solve π = πP with the probabilities summing to 1; the mean time between visits to state i is 1 ÷ π_i.

For an absorbing chain, the fundamental matrix N = (I − Q)^-1 gives expected visits to transient states, and B = NR gives absorption probabilities.

Markov models support maintenance policy, quality yield, brand switching, and inventory decisions by attaching costs to long-run state probabilities.

Last updated: October 2026

2.5 Markov Chains

Markov chains model systems that move among a set of states at regular time steps, such as a machine that is good, degraded, or failed each shift. The specification lists Markov chains as a modeling technique, and they also support maintenance and reliability decisions.

1. The Markov Property and the Transition Matrix

A discrete-time Markov chain satisfies the Markov property : the probability of the next state depends only on the current state, not on how the system got there.

P ( X t + 1 = j ∣ X t = i , X t − 1 , … , X 0 ) = P ( X t + 1 = j ∣ X t = i ) = p i j P(X_{t+1} = j \mid X_t = i, X_{t-1}, \dots, X_0) = P(X_{t+1} = j \mid X_t = i) = p_{ij} P ( X t + 1 ​ = j ∣ X t ​ = i , X t − 1 ​ , … , X 0 ​ ) = P ( X t + 1 ​ = j ∣ X t ​ = i ) = p ij ​

The probabilities p i j p_{ij} p ij ​ form the transition matrix P \mathbf{P} P . Row i i i lists where the system goes from state i i i , so every row sums to 1 .

Machine-condition example. A packaging machine is inspected each shift and classified as Good (G), Degraded (D), or Failed (F). A failed machine is repaired during the next shift and returns to Good.

• From \ To G D F
• G 0.80 0.15 0.05
• D 0 0.70 0.30
• F 1.00 0 0
• 2. n-Step Probabilities

If π ( 0 ) \boldsymbol{\pi}^{(0)} π ( 0 ) is the row vector of starting probabilities, the distribution after n n n steps is:

π ( n ) = π ( n − 1 ) P = π ( 0 ) P n \boldsymbol{\pi}^{(n)} = \boldsymbol{\pi}^{(n-1)} \mathbf{P} = \boldsymbol{\pi}^{(0)} \mathbf{P}^n π ( n ) = π ( n − 1 ) P = π ( 0 ) P n

Starting from a newly repaired machine, π ( 0 ) = [ 1 , 0 , 0 ] \boldsymbol{\pi}^{(0)} = [1, 0, 0] π ( 0 ) = [ 1 , 0 , 0 ] :

After one shift: π ( 1 ) = [ 0.80 , 0.15 , 0.05 ] \boldsymbol{\pi}^{(1)} = [0.80, 0.15, 0.05] π ( 1 ) = [ 0.80 , 0.15 , 0.05 ] .

After two shifts:

P ( G ) = 0.80 ( 0.80 ) + 0.15 ( 0 ) + 0.05 ( 1.00 ) = 0.69 P(G) = 0.80(0.80) + 0.15(0) + 0.05(1.00) = 0.69 P ( G ) = 0.80 ( 0.80 ) + 0.15 ( 0 ) + 0.05 ( 1.00 ) = 0.69

P ( D ) = 0.80 ( 0.15 ) + 0.15 ( 0.70 ) + 0.05 ( 0 ) = 0.225 P(D) = 0.80(0.15) + 0.15(0.70) + 0.05(0) = 0.225 P ( D ) = 0.80 ( 0.15 ) + 0.15 ( 0.70 ) + 0.05 ( 0 ) = 0.225

P ( F ) = 0.80 ( 0.05 ) + 0.15 ( 0.30 ) + 0.05 ( 0 ) = 0.085 P(F) = 0.80(0.05) + 0.15(0.30) + 0.05(0) = 0.085 P ( F ) = 0.80 ( 0.05 ) + 0.15 ( 0.30 ) + 0.05 ( 0 ) = 0.085

The three values sum to 1, which is a quick check on the arithmetic. On the exam, multiply row vector by matrix one step at a time rather than computing P n \mathbf{P}^n P n .

3. Steady-State (Long-Run) Probabilities

For a chain in which every state can reach every other state and the chain is not periodic, the distribution settles to a unique steady state π \boldsymbol{\pi} π regardless of the starting state:

π = π P , ∑ i π i = 1 \boldsymbol{\pi} = \boldsymbol{\pi}\mathbf{P}, \qquad \sum_i \pi_i = 1 π = π P , ∑ i ​ π i ​ = 1

Write one balance equation per state, drop any one of them (they are redundant), and add the normalization equation.

Machine example:

From the D column: π D = 0.15 π G + 0.70 π D \pi_D = 0.15\pi_G + 0.70\pi_D π D ​ = 0.15 π G ​ + 0.70 π D ​ , so 0.30 π D = 0.15 π G 0.30\pi_D = 0.15\pi_G 0.30 π D ​ = 0.15 π G ​ and π D = 0.5 π G \pi_D = 0.5\pi_G π D ​ = 0.5 π G ​ .

From the F column: π F = 0.05 π G + 0.30 π D = 0.05 π G + 0.15 π G = 0.2 π G \pi_F = 0.05\pi_G + 0.30\pi_D = 0.05\pi_G + 0.15\pi_G = 0.2\pi_G π F ​ = 0.05 π G ​ + 0.30 π D ​ = 0.05 π G ​ + 0.15 π G ​ = 0.2 π G ​ .

Normalize: π G ( 1 + 0.5 + 0.2 ) = 1 \pi_G(1 + 0.5 + 0.2) = 1 π G ​ ( 1 + 0.5 + 0.2 ) = 1 , so π G = 0.588 \pi_G = 0.588 π G ​ = 0.588 , π D = 0.294 \pi_D = 0.294 π D ​ = 0.294 , and π F = 0.118 \pi_F = 0.118 π F ​ = 0.118 .

Check with the G column: 0.80 ( 0.588 ) + 1.00 ( 0.118 ) = 0.588 0.80(0.588) + 1.00(0.118) = 0.588 0.80 ( 0.588 ) + 1.00 ( 0.118 ) = 0.588 . ✓

Using the result.

The machine is failed about 11.8% of shifts, and the mean recurrence time of the failed state is 1 / π F = 8.5 1/\pi_F = 8.5 1/ π F ​ = 8.5 shifts between failures.

With costs of $1,000 per degraded shift (scrap) and $5,000 per failed shift (repair and lost output), the long-run expected cost is 0.294 ( 1,000 ) + 0.118 ( 5,000 ) ≈ $ 882 0.294(1{,}000) + 0.118(5{,}000) \approx \text{\textdollar}882 0.294 ( 1 , 000 ) + 0.118 ( 5 , 000 ) ≈ $ 882 per shift.

A preventive policy, such as repairing whenever the machine is found degraded, changes the matrix. Compare policies by recomputing π \boldsymbol{\pi} π and the expected cost.

4. Absorbing Markov Chains

A state is absorbing if the chain can never leave it ( p i i = 1 p_{ii} = 1 p ii ​ = 1 ). Arrange the matrix with transient states first:

P = [ Q R 0 I ] \mathbf{P} = \begin{bmatrix} \mathbf{Q} & \mathbf{R} \\ \mathbf{0} & \mathbf{I} \end{bmatrix} P = [ Q 0 ​ R I ​ ]

where Q \mathbf{Q} Q holds transitions among transient states and R \mathbf{R} R holds transitions from transient to absorbing states.

Fundamental matrix: N = ( I − Q ) − 1 \mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1} N = ( I − Q ) − 1 . Entry n i j n_{ij} n ij ​ is the expected number of visits to transient state j j j when starting in transient state i i i .

Expected steps before absorption: t = N 1 \mathbf{t} = \mathbf{N}\mathbf{1} t = N1 (row sums of N \mathbf{N} N ).

Absorption probabilities: B = N R \mathbf{B} = \mathbf{N}\mathbf{R} B = NR .

Rework loop example. Parts leave machining (state P) and go to Good stock 85% of the time, to Rework (state W) 10%, or to Scrap 5%. Reworked parts become Good 70% of the time, are scrapped 20%, and need rework again 10%.

Q = [ 0 0.10 0 0.10 ] , R = [ 0.85 0.05 0.70 0.20 ] , N = ( I − Q ) − 1 = [ 1 0.111 0 1.111 ] \mathbf{Q} = \begin{bmatrix} 0 & 0.10 \\ 0 & 0.10 \end{bmatrix}, \quad \mathbf{R} = \begin{bmatrix} 0.85 & 0.05 \\ 0.70 & 0.20 \end{bmatrix}, \quad \mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1} = \begin{bmatrix} 1 & 0.111 \\ 0 & 1.111 \end{bmatrix} Q = [ 0 0 ​ 0.10 0.10 ​ ] , R = [ 0.85 0.70 ​ 0.05 0.20 ​ ] , N = ( I − Q ) − 1 = [ 1 0 ​ 0.111 1.111 ​ ]

For a part starting in machining:

P ( Good ) = 1 ( 0.85 ) + 0.111 ( 0.70 ) = 0.928 P(\text{Good}) = 1(0.85) + 0.111(0.70) = 0.928 P ( Good ) = 1 ( 0.85 ) + 0.111 ( 0.70 ) = 0.928

P ( Scrap ) = 1 ( 0.05 ) + 0.111 ( 0.20 ) = 0.072 P(\text{Scrap}) = 1(0.05) + 0.111(0.20) = 0.072 P ( Scrap ) = 1 ( 0.05 ) + 0.111 ( 0.20 ) = 0.072

Expected rework operations per part = n P W = 0.111 = n_{PW} = 0.111 = n P W ​ = 0.111

For 10,000 parts started, expect about 9,278 good parts, 722 scrapped, and 1,111 rework operations. That rework load is what the capacity plan must cover.

5. Common Applications and Exam Traps

• Application States Decision supported
• Machine maintenance Condition levels Repair or replace policy
• Brand switching Product a customer buys Long-run market share
• Inventory Stock on hand at review Ordering policy cost
• Quality yield Process, rework, good, scrap Yield and rework capacity
• Accounts receivable Age of account, paid, bad debt Expected collections

Traps:

Columns, not rows, are used when writing balance equations for π j \pi_j π j ​ , because π j = ∑ i π i p i j \pi_j = \sum_i \pi_i p_{ij} π j ​ = ∑ i ​ π i ​ p ij ​ sums down column j j j .

Probabilities must be per time step; convert rates to per-shift or per-period probabilities first.

Steady-state results do not apply to absorbing chains, because those chains end in an absorbing state.