Study guide

2.1 DORA Metrics, Flow Metrics & Dashboard Design

Microsoft Certified: Azure DevOps Engineer Expert AZ-400

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Flow of Work, Traceability & Work Tracking

1.1 Flow of Work, GitHub Flow & Feedback Cycles 1.2 Work Tracking with Azure Boards & GitHub Projects 1.3 End-to-End DevOps Traceability & Quality Tracking

2 Chapter 2: DevOps Metrics, Dashboards & Team Collaboration

2.1 DORA Metrics, Flow Metrics & Dashboard Design 2.2 Lifecycle Metrics Across the Six DevOps Phases 2.3 Work Item Queries, WIQL Charts & Analytics 2.4 Project Documentation: Wikis, Mermaid Diagrams & Release Notes 2.5 Webhooks, Service Hooks & Microsoft Teams ChatOps

3 Chapter 3: Git Branching Strategies & Pull Request Workflows

3.1 Git Branching Models & Strategy Design 3.2 Pull Request Workflows & Merge Strategies 3.3 Branch Policies, Quality Gates & Compliance Restrictions

4 Chapter 4: Repository Management, Optimization & Git Data Operations

4.1 Git LFS Architecture & .gitattributes Tracking 4.2 LFS History Migration, git-fat & File Locking 4.3 Git Repository Scaling, Scalar & Monorepo Governance 4.4 Repository Permissions & Git Tag Governance 4.5 Git Data Recovery & Sensitive-Data Scrubbing

5 Chapter 5: Package Management & Dependency Versioning

5.1 Selecting Package Tools: Azure Artifacts vs. GitHub Packages 5.2 Feed Scoping, Upstream Sources & Pipeline Authentication 5.3 Package Views, Immutability & Quality Promotion 5.4 Dependency Versioning: SemVer, CalVer & Automated Versioning 5.5 Pipeline Artifact Versioning & Artifact Storage Strategy

6 Chapter 6: Pipeline Infrastructure, Agents & Integration

6.1 Selecting a CI/CD Solution: Azure Pipelines vs. GitHub Actions 6.2 GitHub-Azure Pipelines Integration & Hybrid Migration 6.3 Hosted vs. Self-Hosted Agents & Runners 6.4 VMSS Elastic Pools, Container Jobs & Agent Governance 6.5 Pipeline Triggers, Filters & Concurrency Management

7 Chapter 7: YAML Pipeline Architecture & Reusable Elements

7.1 YAML Pipeline Anatomy: Stages, Jobs, Steps & Workspaces 7.2 Job Execution Order: DAG Dependencies, Parallelism & Conditions 7.3 Reusable Elements: Templates, Parameters & Variables 7.4 Variable Groups, Secret Variables & Key Vault Linking 7.5 YAML Environments, Checks & Deployment Approvals

8 Chapter 8: Testing Strategy, Quality Gates & Code Coverage

8.1 Test Strategy Design: Local, Unit & Integration Testing 8.2 Load, Stress & Performance Testing in Pipelines 8.3 Test Execution Tasks, Agents & Results Integration 8.4 Code Coverage Analysis & SonarQube Integration 8.5 Quality Gates, Release Gates & Governance Checks

9 Chapter 9: Modern Deployment Strategies & Zero-Downtime Delivery

9.1 Progressive Delivery: Blue-Green, Canary & Ring Deployments 9.2 Minimizing Downtime: Deployment Slots & Rolling Updates 9.3 Feature Flags with Azure App Configuration & Feature Manager 9.4 Designing the High-Priority Hotfix Path 9.5 Ordered Dependency Deployments, Resiliency & Rollback

10 Chapter 10: Artifact Deployment & Database DevOps

10.1 Containerized Application Delivery: ACR, AKS & Container Apps 10.2 Binary, Web App, Function & Scripted Deployments 10.3 Database DevOps: State-Based & Migration-Based Strategies

11 Chapter 11: Infrastructure as Code (IaC) & Azure Deployment Environments

11.1 Infrastructure as Code Strategy: Bicep, ARM & Terraform 11.2 Desired State Configuration & Azure Machine Configuration 11.3 Azure Deployment Environments (ADE) & Developer Self-Service

12 Chapter 12: Pipeline Maintenance, Health & Modernization

12.1 Pipeline Health, Flaky Tests & Reliability Engineering 12.2 Pipeline Optimization: Caching, Slicing & Concurrency 12.3 Artifact & Build Retention Governance 12.4 Modernizing CI/CD: Classic to YAML Migration

13 Chapter 13: Authentication, Authorization & Identity Governance

13.1 Microsoft Entra Service Principals & Managed Identities 13.2 Service Connections, Workload Identity & Personal Access Tokens 13.3 Azure DevOps Projects, Teams & Permission Evaluation 13.4 Access Levels, GitHub Roles & Outside Collaborators

14 Chapter 14: Secrets Management, Sensitive Data & DevSecOps Scanning

14.1 Azure Key Vault, Secret Variables & Secretless CI/CD 14.2 Security Scanning Strategy, GHAS & Secret Scanning 14.3 CodeQL SAST in Pipelines and Containers 14.4 Software Supply Chain, Container Scanning & Cloud Defender

15 Chapter 15: Telemetry Collection, Azure Monitor & Alerting

15.1 Azure Monitor Architecture, AMA & Data Collection Rules 15.2 Application Insights & Specialized Azure Monitor Insights 15.3 Alerting Rules, Action Groups & GitHub Repository Insights

16 Chapter 16: Telemetry Analysis, Distributed Tracing & SRE Practices

16.1 Distributed Tracing, Application Map & Exception Triage 16.2 KQL Log Analytics, SRE Observability & Error Budgets

2.1 DORA Metrics, Flow Metrics & Dashboard Design

Key Takeaways Explain This Topic AI

The four DORA metrics (Deployment Frequency, Lead Time for Changes, Mean Time to Recovery/Restore Service, and Change Failure Rate) mathematically differentiate Elite from Low DevOps performers.

Lead Time for Changes measures the clock from code commit/merge to production deployment, whereas Customer Lead Time spans from initial backlog item creation to end-user delivery.

Cycle Time measures active development elapsed time from work item activation to resolution, and Little's Law proves that reducing Work in Progress (WIP) directly decreases cycle time.

In Cumulative Flow Diagrams (CFDs), a vertically widening band signals a downstream queue bottleneck, a widening horizontal gap reflects increasing cycle time, and flat horizontal bands denote blocked work.

Azure DevOps Dashboards provide team telemetry through native widgets including Cumulative Flow Diagram, Cycle Time, Lead Time, Velocity, and Burndown/Burnup charts.

Last updated: September 2026

2.1 DORA Metrics, Flow Metrics & Dashboard Design

Modern DevOps engineering requires moving beyond vanity metrics—such as raw lines of code written, total commit counts, or story point velocity—and focusing on outcome-based metrics that evaluate software delivery throughput, system stability, and flow efficiency. The DevOps Research and Assessment (DORA) metrics and Lean Flow Metrics provide the foundational framework tested extensively in the Microsoft Certified: DevOps Engineer Expert (AZ-400) examination.

The Four DORA Metrics

Established by the DORA research program (led by Dr. Nicole Forsgren, Jez Humble, and Gene Kim), these four metrics objectively evaluate an engineering organization's software delivery performance. DORA divides these metrics into two throughput (speed) metrics and two stability (quality) metrics.

1. Deployment Frequency (DF)

Definition: How frequently your organization successfully releases software to production or an app store.

Measurement Window: Measured per day, week, month, or year.

Operational Objective: Reduce batch sizes. High deployment frequency is the byproduct of continuous integration, automated deployment pipelines, and trunk-based development.

Exam Significance: Teams practicing continuous delivery aim for on-demand deployments (multiple times per day per developer or team), rather than scheduling high-risk, multi-week release trains.

2. Lead Time for Changes (LTTC)

Definition: The duration required for a code commit to successfully execute in production.

Measurement Window: The clock begins when an engineer commits code (or when a pull request is merged into the trunk) and stops when that code is actively running in the production environment.

Operational Objective: Eliminate queue times, automate verification gates, and streamline continuous integration/continuous deployment (CI/CD) pipelines.

Exam Distinction: Do not confuse Lead Time for Changes with overall Customer Lead Time . Lead Time for Changes measures engineering and pipeline execution velocity. Customer Lead Time measures the entire product lifecycle from initial backlog ideation to customer availability.

3. Mean Time to Recovery / Restore Service (MTTR)

Definition: The average time required to restore service when an unplanned outage, degradation, or production failure occurs.

Measurement Window: Starts when an incident is detected (via alerts or monitoring) and concludes when service availability and health are fully restored.

Operational Objective: Build resilient architectures, automate telemetry alerts, and adopt fast remediation strategies—such as deployment slot auto-swaps in Azure App Service, rolling rollbacks, or toggling off offending features using Azure App Configuration feature flags.

4. Change Failure Rate (CFR)

Definition: The percentage of deployments to production that result in degraded service, immediate outage, or require remediation (e.g., hotfixes, rollbacks, patches).

Formula: Change Failure Rate = ( Number of Failed Production Deployments Total Production Deployments ) × 100 \text{Change Failure Rate} = \left( \frac{\text{Number of Failed Production Deployments}}{\text{Total Production Deployments}} \right) \times 100 Change Failure Rate = ( Total Production Deployments Number of Failed Production Deployments ​ ) × 100

Operational Objective: Improve automated test coverage, implement static code analysis (SAST), enforce pull request branch policies, and utilize canary or blue-green deployment strategies.

DORA Performance Benchmarks

The AZ-400 syllabus tests your ability to categorize organizational maturity based on empirical DORA benchmarks:

  • Performance Tier Deployment Frequency Lead Time for Changes Mean Time to Recovery (MTTR) Change Failure Rate
  • Elite On-demand (multiple deploys per day) Less than 1 hour Less than 1 hour 0% – 15%
  • High Between once per day and once per week Between 1 day and 1 week Less than 1 day 0% – 15%
  • Medium Between once per week and once per month Between 1 week and 1 month Between 1 day and 1 week 16% – 30%
  • Low Fewer than once every 6 months More than 6 months More than 1 week 46% – 60%

Important

The Speed vs. Stability Fallacy: Traditional IT management assumed that deploying faster increased the risk of failure. DORA empirical research disproved this assumption. Elite performers excel at both throughput (on-demand deployments, sub-hour lead time) and stability (sub-hour recovery, low failure rate). High-performing teams do not trade speed for stability; automated testing and small batch sizes simultaneously enhance both.

Lean Flow Metrics in Azure Boards

While DORA evaluates the CI/CD and production delivery pipeline, Lean Flow Metrics evaluate work item progression across Azure Boards. They measure how work moves through an engineering team's value stream. +-----------------------------------------------------------------------------------------+

  • | CUSTOMER LEAD TIME |
  • | (From Customer Request / Work Item Creation to Final Production Release) |
  • +-----------------------------------------------------------------------------------------+
  • | |
  • v v
  • [ New Backlog Item ] ---> [ In Progress / Active ] ---> [ PR Merged ] ---> [ Deployed to Prod ]
  • | |
  • +--------- CYCLE TIME -------------+
  • (Active Development to Resolution)
  • |
  • +--- LEAD TIME FOR CHANGES ---+
  • (Commit / Merge to Production)
  • Core Flow Definitions

Customer Lead Time: Elapsed time from work item creation in Azure Boards until delivery to the customer in production.

Cycle Time: Elapsed time from when work actively begins on an item (work item state changes from New / Proposed to Active / In Progress ) until the item is completed ( Resolved or Closed ).

Throughput: The absolute count of completed work items delivered per unit of time (e.g., stories completed per sprint or bugs closed per week).

Work in Progress (WIP): The total number of work items currently active in the delivery process (neither backlog nor completed).

Little's Law and WIP Limits

Flow physics is governed by Little's Law : Average Cycle Time = Average Work in Progress (WIP) Average Throughput \text{Average Cycle Time} = \frac{\text{Average Work in Progress (WIP)}}{\text{Average Throughput}} Average Cycle Time = Average Throughput Average Work in Progress (WIP) ​

To decrease cycle time without hiring more engineers (increasing throughput capacity), a team must decrease Work in Progress. When teams take on too many concurrent tasks, context-switching overhead and queue waiting times skyrocket.

In Azure Boards, you enforce this by configuring Column WIP Limits on the Kanban board:

Open Azure Boards > Boards > Column Settings .

Assign a numeric WIP limit for active columns (e.g., Development , Code Review , QA Testing ).

When the number of work items in a column exceeds the limit, the column header displays a red visual warning indicator. Azure Boards does not hard-block item movement by default, but it visibly highlights the policy violation.

Cumulative Flow Diagrams (CFD)

A Cumulative Flow Diagram (CFD) is an area chart that displays the cumulative number of work items in each state/column of a Kanban board over time. The horizontal axis represents time (dates/sprints), and the vertical axis represents the cumulative count of work items.

Interpreting CFD Patterns

CFD Visual Pattern Root Cause / System Condition Operational Implication

Parallel, Even Bands Balanced flow. Work arrival rate equals departure rate. Predictable delivery, stable cycle times, healthy team capacity.

Vertically Widening Band Downstream bottleneck. Items are entering this column faster than the next column can consume them. Queue buildup. For example, if QA is widening while Dev is narrow, QA capacity or test automation is lagging.

Widening Horizontal Gap Increasing cycle time. Items are spending more total days traversing the system. Work is stagnating; WIP limits are either absent or routinely violated.

Flat Horizontal Line Blockage / zero throughput. No work items are moving out of this stage. Pipeline outage, external dependency failure, or missing deployment approvals.

Narrowing Band Starvation. Upstream stages are not producing enough work to feed downstream capacity. Backlog refinement deficiency or upstream blocking issue.

Tip

AZ-400 Exam Scenario Strategy: When presented with a CFD where the Verification or QA band is expanding vertically over a 30-day window, the correct exam answer is never to "increase developer velocity." The correct action is to impose or lower the WIP limit on the bottlenecked column and reallocate engineering capacity to assist with automated testing (swarming on the queue).

Azure DevOps Dashboard Widgets & Configuration

Azure DevOps provides configurable dashboards ( Overview > Dashboards ) that synthesize real-time telemetry from Azure Boards, Azure Repos, and Azure Pipelines. +-----------------------------------------------------------------------------------------+

  • | AZURE DEVOPS TEAM DASHBOARD |
  • +----------------------------+-----------------------------+------------------------------+
  • | CUMULATIVE FLOW (CFD) | CYCLE TIME WIDGET | VELOCITY WIDGET |
  • | [Area Chart: Bands over | [Scatter Plot & Rolling | [Bar Chart: Committed vs |
  • | Time across Columns] | Average: 50th/80th %ile] | Completed Story Points] |
  • +----------------------------+-----------------------------+------------------------------+
  • | BURNDOWN WIDGET | QUERY TILE WIDGET | PIPELINE STATUS TILE |
  • | [Sprint Remaining Hours | [Sev-1 Bugs: 0 (Green) | [Production CI/CD: Succeeded|
  • | vs Ideal Trend Line] | Conditional Rule Alert] | Duration: 8m 42s] |
  • +----------------------------+-----------------------------+------------------------------+
  • 1. Cumulative Flow Diagram Widget

Purpose: Monitors flow bottlenecks and overall work distribution.

Configuration: Configured per Team, Backlog level (Stories, Epics, Features), Swimlane, and Time window (e.g., past 30, 60, or 90 days). You can explicitly hide the Backlog (New) and Completed (Closed) states to focus purely on active WIP.

2. Cycle Time and Lead Time Widgets

Cycle Time Widget: Displays a scatter plot of individual work items and a rolling average line showing elapsed active days. Configurable to display 50th, 80th, and 95th percentile confidence bands.

Lead Time Widget: Displays the total duration from backlog item creation to completion.

3. Velocity Widget

Purpose: Evaluates sprint-over-sprint planning predictability.

Display: Bar chart showing Planned/Committed work (story points or item count at the start of the sprint) versus Completed work at sprint close across past iterations (default: past 6 sprints).

4. Sprint Burndown & Release Burnup Widgets

Sprint Burndown: Plots remaining estimated work (hours or story points) against an ideal linear burndown rate across the sprint iteration dates. Helps detect mid-sprint scope creep.

Release Burnup: Plots total completed scope against total planned scope across multiple sprints to forecast release completion dates.

5. Query Tile & Chart for Work Items Widgets

Query Tile: Displays a single numeric count based on a saved Work Item Query. Supports Conditional Rules (e.g., background turns green if open Critical Bugs = 0; turns red if open Critical Bugs > 0).

Chart for Work Items: Renders Pie, Bar, Column, or Pivot charts based on a flat work item query.

Common Exam Traps & Real-World Pitfalls

Confusing Lead Time with Cycle Time:

Trap: Assuming Lead Time starts when a developer picks up a task.

Fact: Lead time starts at creation/request. Cycle time starts when work begins.

Misidentifying the Scope of Lead Time for Changes:

Trap: Measuring Lead Time for Changes from customer request.

Fact: DORA Lead Time for Changes measures purely from commit/merge to production deployment.

Misreading CFD Vertical vs. Horizontal Dimensions:

Trap: Believing a vertically expanding band means the team is producing more output.

Fact: Vertical expansion indicates accumulation of inventory (WIP), meaning work is piling up and waiting for a downstream resource.

Loading diagram... Timeline Comparison: Lead Time for Changes vs Cycle Time vs Customer Lead Time