Study guide
16.2 KQL Log Analytics, SRE Observability & Error Budgets
Microsoft Certified: Azure DevOps Engineer Expert AZ-400
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Flow of Work, Traceability & Work Tracking
1.1 Flow of Work, GitHub Flow & Feedback Cycles 1.2 Work Tracking with Azure Boards & GitHub Projects 1.3 End-to-End DevOps Traceability & Quality Tracking
2 Chapter 2: DevOps Metrics, Dashboards & Team Collaboration
2.1 DORA Metrics, Flow Metrics & Dashboard Design 2.2 Lifecycle Metrics Across the Six DevOps Phases 2.3 Work Item Queries, WIQL Charts & Analytics 2.4 Project Documentation: Wikis, Mermaid Diagrams & Release Notes 2.5 Webhooks, Service Hooks & Microsoft Teams ChatOps
3 Chapter 3: Git Branching Strategies & Pull Request Workflows
3.1 Git Branching Models & Strategy Design 3.2 Pull Request Workflows & Merge Strategies 3.3 Branch Policies, Quality Gates & Compliance Restrictions
4 Chapter 4: Repository Management, Optimization & Git Data Operations
4.1 Git LFS Architecture & .gitattributes Tracking 4.2 LFS History Migration, git-fat & File Locking 4.3 Git Repository Scaling, Scalar & Monorepo Governance 4.4 Repository Permissions & Git Tag Governance 4.5 Git Data Recovery & Sensitive-Data Scrubbing
5 Chapter 5: Package Management & Dependency Versioning
5.1 Selecting Package Tools: Azure Artifacts vs. GitHub Packages 5.2 Feed Scoping, Upstream Sources & Pipeline Authentication 5.3 Package Views, Immutability & Quality Promotion 5.4 Dependency Versioning: SemVer, CalVer & Automated Versioning 5.5 Pipeline Artifact Versioning & Artifact Storage Strategy
6 Chapter 6: Pipeline Infrastructure, Agents & Integration
6.1 Selecting a CI/CD Solution: Azure Pipelines vs. GitHub Actions 6.2 GitHub-Azure Pipelines Integration & Hybrid Migration 6.3 Hosted vs. Self-Hosted Agents & Runners 6.4 VMSS Elastic Pools, Container Jobs & Agent Governance 6.5 Pipeline Triggers, Filters & Concurrency Management
7 Chapter 7: YAML Pipeline Architecture & Reusable Elements
7.1 YAML Pipeline Anatomy: Stages, Jobs, Steps & Workspaces 7.2 Job Execution Order: DAG Dependencies, Parallelism & Conditions 7.3 Reusable Elements: Templates, Parameters & Variables 7.4 Variable Groups, Secret Variables & Key Vault Linking 7.5 YAML Environments, Checks & Deployment Approvals
8 Chapter 8: Testing Strategy, Quality Gates & Code Coverage
8.1 Test Strategy Design: Local, Unit & Integration Testing 8.2 Load, Stress & Performance Testing in Pipelines 8.3 Test Execution Tasks, Agents & Results Integration 8.4 Code Coverage Analysis & SonarQube Integration 8.5 Quality Gates, Release Gates & Governance Checks
9 Chapter 9: Modern Deployment Strategies & Zero-Downtime Delivery
9.1 Progressive Delivery: Blue-Green, Canary & Ring Deployments 9.2 Minimizing Downtime: Deployment Slots & Rolling Updates 9.3 Feature Flags with Azure App Configuration & Feature Manager 9.4 Designing the High-Priority Hotfix Path 9.5 Ordered Dependency Deployments, Resiliency & Rollback
10 Chapter 10: Artifact Deployment & Database DevOps
10.1 Containerized Application Delivery: ACR, AKS & Container Apps 10.2 Binary, Web App, Function & Scripted Deployments 10.3 Database DevOps: State-Based & Migration-Based Strategies
11 Chapter 11: Infrastructure as Code (IaC) & Azure Deployment Environments
11.1 Infrastructure as Code Strategy: Bicep, ARM & Terraform 11.2 Desired State Configuration & Azure Machine Configuration 11.3 Azure Deployment Environments (ADE) & Developer Self-Service
12 Chapter 12: Pipeline Maintenance, Health & Modernization
12.1 Pipeline Health, Flaky Tests & Reliability Engineering 12.2 Pipeline Optimization: Caching, Slicing & Concurrency 12.3 Artifact & Build Retention Governance 12.4 Modernizing CI/CD: Classic to YAML Migration
13 Chapter 13: Authentication, Authorization & Identity Governance
13.1 Microsoft Entra Service Principals & Managed Identities 13.2 Service Connections, Workload Identity & Personal Access Tokens 13.3 Azure DevOps Projects, Teams & Permission Evaluation 13.4 Access Levels, GitHub Roles & Outside Collaborators
14 Chapter 14: Secrets Management, Sensitive Data & DevSecOps Scanning
14.1 Azure Key Vault, Secret Variables & Secretless CI/CD 14.2 Security Scanning Strategy, GHAS & Secret Scanning 14.3 CodeQL SAST in Pipelines and Containers 14.4 Software Supply Chain, Container Scanning & Cloud Defender
15 Chapter 15: Telemetry Collection, Azure Monitor & Alerting
15.1 Azure Monitor Architecture, AMA & Data Collection Rules 15.2 Application Insights & Specialized Azure Monitor Insights 15.3 Alerting Rules, Action Groups & GitHub Repository Insights
16 Chapter 16: Telemetry Analysis, Distributed Tracing & SRE Practices
16.1 Distributed Tracing, Application Map & Exception Triage 16.2 KQL Log Analytics, SRE Observability & Error Budgets
16.2 KQL Log Analytics, SRE Observability & Error Budgets
Key Takeaways Explain This Topic AI
Kusto Query Language (KQL) provides a high-performance, read-only analytics syntax for querying structured and semi-structured telemetry across Log Analytics tables including requests, dependencies, exceptions, and traces.
Production latency analysis must evaluate statistical percentiles (p50, p90, p95, p99) using percentile() rather than arithmetic averages, as averages disguise extreme tail-latency spikes that degrade end-user experience.
Site Reliability Engineering (SRE) balances speed and stability using the SLI/SLO/SLA hierarchy: Service Level Indicators measure real-time performance, Service Level Objectives define internal reliability targets, and Service Level Agreements enforce external contractual commitments with financial penalties.
The Error Budget (1 - SLO) defines the allowable failure threshold; tracking error budget burn rate enables automated CI/CD release gating where budget exhaustion triggers automated deployment freezes to prioritize reliability engineering.
Azure Chaos Studio validates system resilience by injecting controlled agent-based faults (CPU pressure, disk I/O, process termination) and service-direct faults (AKS Chaos Mesh pod kills, Cosmos DB failovers) into staging environments to verify automated recovery before production rollout.
Last updated: September 2026
16.2 KQL Log Analytics, SRE Observability & Error Budgets
Collecting telemetry is only the first step in building a reliable cloud ecosystem. To operate modern systems at scale, DevOps and Site Reliability Engineering (SRE) teams must analyze vast volumes of log records rapidly, identify subtle performance regressions before they cause outages, establish measurable reliability targets, and proactively test system resilience under failure conditions.
For the AZ-400 exam, candidates must master Kusto Query Language (KQL) to query Azure Log Analytics workspaces and Application Insights, apply SRE principles —specifically the interplay between Service Level Indicators (SLIs) , Service Level Objectives (SLOs) , Service Level Agreements (SLAs) , and Error Budgets —and integrate automated chaos experiments using Azure Chaos Studio into deployment verification pipelines.
1. Kusto Query Language (KQL) Architecture & Core Schema Tables
Log Analytics and Application Insights are powered by the Azure Data Explorer (ADX / Kusto) distributed database engine. KQL is a powerful, read-only, pipeline-based query language optimized for blazing-fast aggregations, text searches, time-series analysis, and pattern matching across petabytes of structured and semi-structured log records. [KQL Pipeline Syntax]
- requests <-- Source Table
- | where timestamp > ago(24h) <-- Filter rows by time
- | where success == false <-- Filter rows by condition
- | summarize FailureCount=count() <-- Aggregate data
- by bin(timestamp, 1h), cloud_RoleName <-- Group by time bucket and role
- | render timechart <-- Visualize results
- Application Insights Core Schema Tables
When Application Insights is backed by a Log Analytics workspace (workspace-based Application Insights), telemetry is organized into standardized tables:
requests : Inbound HTTP/gRPC transactions. Key columns: timestamp , id , name (URL path), duration (milliseconds), success (boolean), resultCode (HTTP status code), client_IP , cloud_RoleName , and operation_Id .
dependencies : Outbound calls to external components (SQL, Redis, HTTP endpoints, Service Bus). Key columns: target , type (e.g., "SQL", "HTTP"), name , duration , success , resultCode , data (e.g., SQL command text), and operation_Id .
exceptions : Caught and unhandled application exceptions. Key columns: type (class name), outerMessage , innermostMessage , details (structured stack trace), problemId , severityLevel , and operation_Id .
traces : Log statements emitted by application logging frameworks ( ILogger , log4j , winston ). Key columns: message , severityLevel (0=Verbose, 1=Info, 2=Warning, 3=Error, 4=Critical), and customDimensions .
customEvents : Business workflow and telemetry markers emitted via TelemetryClient.TrackEvent() . Key columns: name , customDimensions , and customMeasurements .
Perf & InsightsMetrics : System-level performance metrics (CPU, available memory, disk IOPS) collected from virtual machines and container nodes.
2. Essential KQL Operators & Syntax
KQL follows a pipe-delimited model where the output of one operator becomes the tabular input to the next operator. Mastering the core operators is essential for AZ-400 query construction:
Filtering and Projection
where : Filters rows matching a boolean condition. Efficient queries place where clauses filtering by timestamp as early as possible to minimize data scan volume: requests | where timestamp > ago(1h) and resultCode startswith "5"
project : Selects, renames, and reorders specific columns to include in the result set, discarding all others to optimize query memory: requests | project timestamp, name, duration, resultCode, operation_Id
project-away : Excludes specific columns from the result set while retaining all others.
extend : Computes and adds a new calculated column to the dataset without dropping existing columns: requests | extend DurationSeconds = duration / 1000.0
Aggregation and Grouping
summarize : The primary aggregation operator. Produces a table aggregating rows using functions like count() , dcount() (distinct count), avg() , sum() , min() , max() , and percentile() : requests | summarize TotalRequests=count(), FailedRequests=countif(success == false) by cloud_RoleName
bin() : Rounds datetime or numeric values down to an integer multiple of a specified bin size. Used to create uniform time buckets for trend analysis: requests | summarize RequestRate=count() by bin(timestamp, 5m)
percentile() & percentiles() : Calculates statistical percentiles (e.g., percentile(duration, 95) ). This is critical for SRE latency analysis, as arithmetic averages disguise catastrophic outliers.
Relational Joins and Unions
join : Merges rows from two tables by matching values on specified columns. Supports multiple join kinds ( inner , leftouter , fullouter ). In KQL, the $left and $right prefixes explicitly distinguish column origins: requests
- | where success == false
- | join kind=inner (exceptions) on $left.operation_Id == $right.operation_Id
- union : Combines rows from two or more tables with compatible schemas: union requests, dependencies
- | where duration > 5000
- Dynamic JSON Parsing
Telemetry often contains nested, semi-structured JSON payloads stored in customDimensions or details . KQL provides first-class JSON parsing operators:
parse_json() / todynamic() : Interprets a string as a dynamic JSON object, allowing direct dot-notation or index-based property extraction: traces
- | extend EventData = parse_json(customDimensions)
- | extend TenantId = tostring(EventData.TenantId)
- | where TenantId == "ContosoEnterprise"
- 3. Real-World Production KQL Queries for DevOps
The AZ-400 exam frequently presents operational scenarios requiring candidates to identify the correct KQL query for troubleshooting performance degradation, isolating errors, or auditing release stability.
Query 1: Top 10 Slowest Downstream Dependencies
Identifies which external SQL queries, microservices, or APIs are causing the greatest latency across the system, computing total call counts, failure rates, and 95th percentile latency: dependencies
- | where timestamp > ago(12h)
- | summarize
- CallCount = count(),
- FailureCount = countif(success == false),
- FailureRate = round(100.0 * countif(success == false) / count(), 2),
- AvgDurationMs = round(avg(duration), 1),
- P95DurationMs = round(percentile(duration, 95), 1)
- by target, type, name
- | where CallCount > 50
- | top 10 by P95DurationMs desc
- Query 2: Calculating 95th Percentile Request Duration Over Time
Evaluates user-perceived latency over a 7-day window binned into 1-hour intervals, comparing the 50th, 95th, and 99th percentiles to detect gradual performance regressions: requests
- | where timestamp > ago(7d)
- | where cloud_RoleName == "checkout-service"
- | summarize
- P50 = percentile(duration, 50),
- P95 = percentile(duration, 95),
- P99 = percentile(duration, 99)
- by bin(timestamp, 1h)
- | render timechart
- Query 3: Aggregating Exceptions by Cloud Role Instance and Type
Triages an unhandled crash wave across Kubernetes pods or App Service instances, pinpointing whether an error is isolated to a specific instance (e.g., bad deployment or corrupt local disk) or spread across the entire fleet: exceptions
- | where timestamp > ago(4h)
- | summarize
- OccurrenceCount = count(),
- AffectedUsers = dcount(user_Id),
- SampleMessage = any(outerMessage)
- by cloud_RoleName, cloud_RoleInstance, type, problemId
- | order by OccurrenceCount desc
- Query 4: Correlating Deployments with Error Spikes and Degradation
Correlates application failure rates with deployment release markers to verify whether a canary release introduced regressions: let Deployments = customEvents
- | where timestamp > ago(24h)
- | where name == "PipelineDeployment"
- | project DeploymentTime = timestamp, ReleaseId = tostring(customDimensions.ReleaseId);
- requests
- | where timestamp > ago(24h)
- | summarize
- Total = count(),
- Failed = countif(success == false),
- FailureRate = round(100.0 * countif(success == false) / count(), 2)
- by bin(timestamp, 15m)
- | render timechart
- 4. SRE Observability Principles: SLIs, SLOs, SLAs & Error Budgets
Site Reliability Engineering (SRE) bridges the traditional divide between developers (who want to ship features quickly) and operations teams (who want stability). SRE achieves this balance using quantifiable reliability metrics. ┌─────────────────────────────────────────────────────────────┐
- │ Service Level Agreement (SLA): 99.5% │
- │ (Contractual commitment to customers; financial penalties) │
- │ ┌─────────────────────────────────────────────────────┐ │
- │ │ Service Level Objective (SLO): 99.9% │ │
- │ │ (Internal engineering target; dictates Error Budget)│ │
- │ │ ┌─────────────────────────────────────────────┐ │ │
- │ │ │ Service Level Indicator (SLI): Actual Value │ │ │
- │ │ │ (Quantitative real-time metric, e.g. 99.94%)│ │ │
- │ │ └─────────────────────────────────────────────┘ │ │
- │ └─────────────────────────────────────────────────────┘ │
- └─────────────────────────────────────────────────────────────┘
- 1. Service Level Indicators (SLIs)
An SLI is a carefully defined, quantitative measurement of service performance delivered to users in real time.
- Availability SLI Formula : Availability SLI = (Successful Requests / Total Valid Requests) * 100%
- Latency SLI Formula : Latency SLI = (Requests completed in < 200 ms / Total Valid Requests) * 100%
- 2. Service Level Objectives (SLOs)
An SLO is the internal target for service reliability agreed upon by product managers, developers, and SREs. It specifies the acceptable threshold for an SLI over a defined measurement window (typically a rolling 30-day period).
Example : "99.9% of valid HTTP requests to /api/orders will return a 2xx status code and complete with a duration under 250 ms over any rolling 30-day period."
3. Service Level Agreements (SLAs)
An SLA is an explicit or implicit commercial contract between a service provider and external paying customers. It specifies what level of service is guaranteed and what remedies (such as billing credits or refunds) will be provided if the guarantee is breached.
Important
The Golden Rule of SRE : SLO must always be stricter than SLA! If your customer SLA is 99.5%, your internal engineering SLO should be at least 99.9%. This creates a safety buffer. If an incident burns through your SLO, your engineering team receives an alert and takes corrective action before you violate the customer SLA and incur financial penalties.
4. The Error Budget: Formula and Downtime Allowances
Reliability is not 100%. Aiming for 100% reliability is prohibitively expensive, slows down feature delivery, and provides diminishing returns to users whose own internet connections have lower availability.
The Error Budget is the exact mathematical inverse of the Service Level Objective: Error Budget = 100% - SLO%
For a standard rolling 30-day month (43,200 total minutes), the error budget dictates the maximum allowable downtime before the SLO is violated:
- SLO Target Error Budget (%) Allowed Downtime per 30-Day Month Allowed Downtime per Year
- 99.0% ("Two Nines") 1.0% 432 minutes (7.2 hours) 87.6 hours (3.65 days)
- 99.5% 0.5% 216 minutes (3.6 hours) 43.8 hours
- 99.9% ("Three Nines") 0.1% 43.2 minutes 8.76 hours
- 99.95% 0.05% 21.6 minutes 4.38 hours
- 99.99% ("Four Nines") 0.01% 4.32 minutes 52.6 minutes
- 99.999% ("Five Nines") 0.001% 25.9 seconds 5.26 minutes
- Error Budget Burn Rate & Multi-Window Alerting
The Burn Rate is the speed at which an application consumes its error budget relative to its SLO period:
Burn Rate = 1 : Consumes exactly 100% of the error budget over the entire 30-day window. (Steady state, zero alerts).
Burn Rate = 14.4 : Consumes 2% of the monthly error budget in exactly 1 hour. This indicates a severe outage requiring immediate on-call paging.
Burn Rate = 6 : Consumes 5% of the monthly error budget in 6 hours. Indicates a moderate regression requiring ticket creation and same-day investigation.
Modern Azure Monitor alert rules implement Multi-Window Multi-Burn-Rate alerting to alert promptly during rapid catastrophic failures while avoiding false alarms from brief, transient spikes.
Error Budget Policy & Release Gating (Governance)
An error budget is useless without an enforceable Error Budget Policy that governs software delivery: [Error Budget Status]
- │
- ┌────────────────────┴────────────────────┐
- ▼ ▼
- [Budget > 20% Remaining] [Budget Exhausted (<= 0%)]
- Fast feature deployments • AUTOMATED CI/CD RELEASE FREEZE
- Canary & progressive rollout • Feature deployments halted
- High velocity & innovation • 100% focus on reliability sprints
- Tech debt, bugs & resilience
Healthy Error Budget (> 20% remaining) : Teams deploy new features rapidly, experiment with canary releases, and take calculated deployment risks.
Exhausted Error Budget (<= 0% remaining) : The automated CI/CD pipeline triggers a release gate that blocks non-emergency feature deployments. The engineering team shifts 100% of development effort toward reliability sprints: fixing bugs, refactoring flaky dependencies, improving test coverage, and hardening infrastructure until the rolling error budget recovers.
5. Chaos Engineering with Azure Chaos Studio
Traditional testing validates that software functions correctly under optimal conditions. Chaos Engineering is the discipline of experimenting on a software system to build confidence in its capability to withstand turbulent and disruptive conditions in production.
Azure Chaos Studio is a fully managed chaos engineering and fault injection service in Microsoft Azure. It allows DevOps teams to deliberately simulate infrastructure failures, network partitions, and application crashes within controlled safety boundaries. [Azure Chaos Studio]
- │
- ├─► [Target: Azure Kubernetes Service] ──► Inject Fault: Chaos Mesh Pod Chaos (Kill Pods)
- ├─► [Target: Azure Virtual Machines] ──► Inject Fault: Chaos Agent CPU Pressure (95% CPU)
- ├─► [Target: Azure Cosmos DB] ──► Inject Fault: Service-Direct Region Failover
- └─► [Target: Virtual Network / NSG] ──► Inject Fault: Network Latency / Packet Drop
- Core Architecture: Targets, Capabilities, and Experiments
Targets and Capabilities :
Before injecting faults, an Azure resource must be onboarded as a Chaos Target .
Specific Capabilities (e.g., CPUPressure-1.0 , CosmosFailover-1.0 , PodChaos-2.1 ) must be enabled on the target, granting permission to execute that specific disruption.
Chaos Experiment Resource :
An ARM resource ( Microsoft.Chaos/experiments ) that orchestrates the execution of faults.
Organized into sequential Steps , parallel Branches , and specific fault Actions with defined durations (e.g., inject 80% CPU stress for 10 minutes).
Managed Identity Authentication : Chaos Studio uses a system-assigned or user-assigned managed identity. The experiment identity must be granted appropriate Azure Role-Based Access Control (RBAC) roles (e.g., Virtual Machine Contributor , Cosmos DB Operator ) on the target resources.
Agent-Based Faults vs. Service-Direct Faults
Dimension Agent-Based Faults Service-Direct Faults (Agentless)
Mechanism Requires installing the Azure Chaos Agent on the target operating system Executes directly against the Azure control plane or resource provider API without any guest agent
Supported Targets Azure Virtual Machines (Windows/Linux), Virtual Machine Scale Sets, Azure Arc-enabled hybrid servers Azure Cosmos DB, Azure Key Vault, Azure App Service, Network Security Groups, Azure Redis Cache, AKS
Types of Disruptions - Physical CPU stress / core pinning; - Virtual memory exhaustion; - Disk I/O fill and latency; - Physical process kill ( kill -9 ); - Network socket termination - Cosmos DB multi-region failover; - Redis cache reboot / node failover; - NSG security rule blocking (network partition); - AKS Pod Chaos / DNS failures (via native Chaos Mesh integration)
Exam Context Use when validating OS-level failure handling, memory leak watchdogs, or host service crash recovery Use when validating PaaS failover, zone redundancy, circuit breakers, and network partition resiliency
Integrating Chaos into CI/CD Release Validation Gates
Elite DevOps teams do not inject chaos randomly in production without validation. They incorporate chaos experiments directly into automated CI/CD release pipelines:
Deploy the new microservice build to a pre-production or staging environment.
Trigger synthetic user load using a load-testing tool (such as Azure Load Testing).
Trigger an Azure Chaos Studio experiment via the Azure CLI or REST API task in Azure Pipelines / GitHub Actions: az chaos experiment start --name "exp-aks-pod-kill" --resource-group "rg-staging"
Evaluate automated release quality gates:
Verify that AKS replica autoscaling or pod restarts maintain request availability.
Verify that application circuit breakers (e.g., Polly or Envoy) trip gracefully without cascading failures.
Verify that the 95th percentile latency SLI remains within SLO bounds during the active disruption.
If the system fails to recover gracefully, the pipeline automatically aborts the release and rolls back the deployment.
- 6. KQL Reference Table & SRE Matrix
- KQL Operator Reference
Operator Syntax Pattern DevOps & SRE Use Case
where requests | where timestamp > ago(1h) Filters rows based on predicate; must be placed early to optimize query scan efficiency.
summarize requests | summarize count() by bin(timestamp, 5m) Aggregates time-series data into metric buckets for charting and alerting.
percentile() requests | summarize percentile(duration, 95) Computes p95 tail latency, eliminating average skew for SRE latency SLIs.
extend requests | extend DurationSec = duration / 1000.0 Creates dynamic calculated fields without discarding raw columns.
project requests | project timestamp, name, duration Selects specific columns to return, reducing query payload and memory usage.
join requests | join kind=inner (exceptions) on $left.operation_Id == $right.operation_Id Correlates disparate tables (e.g., joining failed requests with exception stack traces).
parse_json() traces | extend Prop = parse_json(customDimensions).Key Extracts nested properties from semi-structured JSON telemetry columns.
top dependencies | top 10 by duration desc Selects the top N rows ordered by a metric; ideal for slowest dependencies or top error types.
render requests | summarize count() by bin(timestamp, 1h) | render timechart Visualizes query results directly as a line timechart, barchart, or piechart.
SLI vs. SLO vs. SLA Comparison Matrix
Dimension Service Level Indicator (SLI) Service Level Objective (SLO) Service Level Agreement (SLA)
Definition A quantitative measure of real-time service performance The internal reliability target agreed upon by engineering and business The external legal/commercial guarantee made to paying customers
Primary Audience Engineers, SREs, on-call responders Product managers, DevOps teams, development leads Customers, executive leadership, legal/finance departments
Formula / Example (Successful Requests / Total Requests) * 100 = 99.94% 99.9% availability over a rolling 30-day window 99.5% availability monthly; breach results in 10% billing credit
Consequence of Breach Fires burn-rate alerts to on-call engineers Triggers CI/CD deployment freeze; pivots development to reliability Financial penalties, customer credits, breach of contract liability
Flexibility Dynamic real-time data Adaptable internally as system matures Highly rigid; contractual renegotiation required
- 7. Realistic Exam Scenario & Common Traps
- Scenario: Regulating Deployment Velocity via Error Budgets
Organization : CloudPay Financial processes online transactions across an AKS cluster and Azure Cosmos DB. The platform maintains a customer SLA of 99.5% monthly availability.
Objective 1 : Establish internal SRE metrics. The team sets an internal SLO of 99.9% availability over a 30-day rolling window, creating a 0.1% monthly error budget (43.2 minutes of allowed downtime).
Objective 2 : Automate release gates. The team configures an Azure DevOps release pipeline that queries Log Analytics before deploying new microservice builds. If the 30-day error budget is exhausted (burn rate > 100% of budget), the pipeline automatically halts feature releases and routes work items into a technical debt sprint.
Objective 3 : Pre-production resiliency validation. Before promoting builds to production, the team runs an Azure Chaos Studio experiment that reboots the primary Azure Cache for Redis instance and injects 500ms network latency on downstream SQL dependencies to ensure the application's circuit breaker and distributed cache fallbacks function properly.
Common Exam Traps to Avoid
Trap: Using arithmetic average ( avg() ) instead of percentiles for latency SLIs. Averages hide severe tail-latency degradation experienced by the 95th or 99th percentile of users. Modern SRE questions mandate using percentile(duration, 95) or percentile(duration, 99) .
Trap: Setting the internal SLO equal to or lower than the external SLA. If the SLO equals the SLA (e.g., both 99.9%), any SLO breach immediately triggers financial customer penalties. SLOs must always be stricter than SLAs to provide an operational safety margin.
Trap: Believing Azure Chaos Studio requires an in-guest agent for all resources. Agent-based faults are only required for OS-level VM disruptions (CPU, physical memory, disk I/O). PaaS services (Cosmos DB failovers, Redis reboots, AKS pod termination via Chaos Mesh) use service-direct faults that require zero guest agents.
Trap: Continuing feature deployments when the error budget is depleted. The core tenet of SRE governance is that an exhausted error budget mandates halting non-critical feature deployments and redirecting engineering capacity to stability and reliability sprints.