Study guide
12.1 Cloud Native Observability (Logging, Metrics & Tracing)
Kubernetes and Cloud Native Associate KCNA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Introduction & Exam Overview
1.1 About the KCNA Exam & Certification Path 1.2 How to Use This Study Guide
2 Chapter 2: Kubernetes Fundamentals - Cluster Architecture & the API
2.1 Kubernetes Architecture & Control Plane Components 2.2 Worker Node Architecture & Node Components 2.3 Kubernetes API, Declarative Management & kubectl 2.4 Kubernetes Releases, API Versioning & Deprecation Policy
3 Chapter 3: Kubernetes Fundamentals - Workloads & Configuration
3.1 Core Workload Objects (Pods, Deployments & ReplicaSets) 3.2 Advanced Controllers (StatefulSets, DaemonSets, Jobs & CronJobs) 3.3 Namespaces, Labels, Selectors & Annotations 3.4 ConfigMaps, Secrets & Application Configuration 3.5 Pod & Container Lifecycle, Restart Policies & Graceful Termination
4 Chapter 4: Kubernetes Fundamentals - Scheduling & Autoscaling
4.1 Scheduling, Resource Management & Node Selection 4.2 Advanced Scheduling: Affinity, Topology Spread, Priority & Disruption Budgets 4.3 Autoscaling Workloads & Clusters (HPA, VPA, Cluster Autoscaler, KEDA)
5 Chapter 5: Kubernetes Fundamentals - Cluster Administration & Extensibility
5.1 Kubernetes Distributions, Installation Tools & Managed Services 5.2 Cluster Administration: Node Maintenance, Upgrades & etcd Backup 5.3 Extending Kubernetes: CRDs, Operators & the Controller Pattern
6 Chapter 6: Kubernetes Fundamentals - Containerization
6.1 Container Fundamentals, Runtimes & Registries 6.2 Container Image Anatomy: Layers, Tags, Digests & Pull Policy
7 Chapter 7: Container Orchestration - Networking & Service Discovery
7.1 Kubernetes Networking, Services & Ingress 7.2 Cluster DNS & Service Discovery with CoreDNS 7.3 NetworkPolicies, Zero-Trust Segmentation & the Gateway API
8 Chapter 8: Container Orchestration - Storage
8.1 Storage Concepts (PVs, PVCs, StorageClasses & CSI) 8.2 Volume Types, Snapshots & Stateful Storage Operations
9 Chapter 9: Container Orchestration - Security
9.1 Kubernetes Security & Access Control (RBAC & ServiceAccounts) 9.2 The 4C's of Cloud Native Security & Supply-Chain Trust 9.3 Service Mesh, mTLS & Zero-Trust Service Identity
10 Chapter 10: Container Orchestration - Troubleshooting
10.1 Container Troubleshooting & Health Probes 10.2 Cluster & Node-Level Troubleshooting
11 Chapter 11: Cloud Native Application Delivery
11.1 Cloud Native CI/CD Pipelines & Build Tooling 11.2 GitOps & Package Management (Helm, Kustomize, Argo CD & Flux) 11.3 Deployment Strategies & Release Management 11.4 Progressive Delivery Automation: Argo Rollouts, Flagger & Feature Flags 11.5 Application Debugging & Developer Workflows in Kubernetes 11.6 Platform Engineering & Developer Self-Service
12 Chapter 12: Cloud Native Architecture - Observability & Principles
12.1 Cloud Native Observability (Logging, Metrics & Tracing) 12.2 Cloud Native Principles & Architecture Patterns 12.3 Serverless, FaaS & Event-Driven Architecture
13 Chapter 13: Cloud Native Architecture - CNCF Ecosystem & Community
13.1 The CNCF Landscape, Governance & Project Lifecycle 13.2 Cloud Native Community, Collaboration & Contribution
12.1 Cloud Native Observability (Logging, Metrics & Tracing)
Key Takeaways Explain This Topic AI
The three pillars of cloud-native observability are Metrics (numeric aggregated time-series data), Logs (timestamped discrete event records), and Traces (end-to-end request propagation paths across microservices).
Prometheus is the CNCF graduated standard for time-series monitoring, utilizing a pull-based scraping model over HTTP, PromQL query language, exporter agents (Node Exporter, kube-state-metrics), and native Grafana dashboard integration.
In Kubernetes logging architecture, applications write log streams to standard output (stdout) and standard error (stderr), which the node container runtime captures and stores as local files under /var/log/pods.
Log aggregation agents like Fluentd, Fluent Bit, and Vector execute as node-level DaemonSets to tail local log files, enrich entries with Kubernetes metadata, and forward streams to storage engines like Grafana Loki or Elasticsearch.
Distributed tracing relies on OpenTelemetry standards and engines like Jaeger or Zipkin, passing Span Context metadata (TraceID and SpanID) across microservice HTTP/gRPC boundaries to visualize request latency and system bottlenecks.
Last updated: August 2026
12.1 Cloud Native Observability (Logging, Metrics & Tracing)
Quick Answer: Cloud-native observability provides actionable visibility into system state through three core pillars: Metrics (numeric time-series telemetry scraped by Prometheus for alerting and scaling), Logs (event text streams emitted to stdout/stderr and collected by DaemonSet agents like Fluent Bit), and Traces (distributed request paths tracked across microservices via OpenTelemetry and Jaeger). Together, these signals enable rapid detection, isolation, and root-cause analysis.
In traditional monolithic environments, monitoring typically involved checking server uptime and CPU utilization. However, in dynamic, highly distributed cloud-native architectures—where hundreds of ephemeral containers continuously scale, migrate, and fail across a cluster—traditional monitoring is insufficient. Observability measures how well a system's internal state can be inferred solely from its external outputs.
The Three Pillars of Cloud Native Observability
Observability relies on three distinct types of telemetry signals, often referred to as the "Three Pillars of Observability": ┌─────────────────────────────────────────────────────────────────────────────┐
- │ THE THREE PILLARS OF OBSERVABILITY │
- ├──────────────────────────┬──────────────────────────┬───────────────────────┤
- │ METRICS │ LOGS │ TRACES │
- ├──────────────────────────┼──────────────────────────┼───────────────────────┤
- │ • Numeric Time-Series │ • Timestamped Events │ • End-to-End Request │
- │ • Aggregated & Efficient │ • Rich Text & Context │ Journey Across Apps │
- │ • Ideal for Alerting & │ • Ideal for Debugging │ • Ideal for Latency │
- │ Real-time Dashboards │ Specific Errors │ & Bottleneck Analysis│
- │ • Tools: Prometheus │ • Tools: Fluent Bit, Loki│ • Tools: Jaeger, OTel │
- └──────────────────────────┴──────────────────────────┴───────────────────────┘
- Observability Telemetry Comparison Matrix
Telemetry Type Data Structure Storage Cost & Overhead Primary Use Case Representative CNCF Tools
Metrics Numeric key-value pairs indexed by time and labels Low overhead; highly compressible time-series data Real-time monitoring, auto-scaling (HPA), instant alerting Prometheus , Thanos, VictoriaMetrics
Logs Discrete, timestamped textual records or structured JSON High storage volume; requires indexing and retention policies Investigating specific errors, auditing, root-cause analysis Fluentd , Fluent Bit , Grafana Loki
Traces Directed Acyclic Graphs (DAGs) of timed Spans linked by TraceIDs Moderate to high overhead; relies on head/tail sampling Identifying latency bottlenecks and microservice dependencies OpenTelemetry , Jaeger , Zipkin
Prometheus Time-Series Monitoring Architecture
Prometheus is a CNCF graduated project and the industry standard for monitoring containerized Kubernetes workloads. ┌─────────────────┐ Scrapes HTTP ┌─────────────────────┐
- │ Node Exporter │◄────────────────────────┤ │
- └─────────────────┘ /metrics │ │
- │ │ Query PromQL
- ┌─────────────────┐ Scrapes HTTP │ Prometheus Server │◄───────────────────────┐
- │kube-state-metrics◄────────────────────────┤ (TSDB Datastore) │ │
- └─────────────────┘ /metrics │ ├────────────────┐ │
- │ │ │ │
- ┌─────────────────┐ Scrapes HTTP │ │ ▼ │
- │ App Pod (/metrics)◄───────────────────────┤ │ ┌────────────────┴┐
- └─────────────────┘ /metrics └──────────┬──────────┘ │Grafana Dashboard│
- │ └─────────────────┘
- │ Triggers Alerts
- ▼
- ┌─────────────────────┐
- │ Alertmanager │
- └─────────────────────┘
- Key Architectural Components
Pull-Based Scraping Model: Unlike traditional agent-push frameworks, the Prometheus server periodically initiates HTTP GET requests to target /metrics endpoints published by applications and cluster services. Target discovery occurs dynamically via the Kubernetes API.
Time-Series Database (TSDB): Prometheus stores metric streams as time-series data consisting of a metric name, timestamp, numeric value, and key-value label pairs (e.g., http_requests_total{method="POST", status="500"} ).
PromQL (Prometheus Query Language): A flexible functional query language used to compute real-time aggregations, rates of change, and quantiles (e.g., rate(http_requests_total[5m]) ).
Exporters: Auxiliary agents that translate existing non-Prometheus metrics into Prometheus format:
Node Exporter: Exposes host-level kernel and hardware metrics (CPU, memory, disk I/O).
kube-state-metrics: Listens to the Kubernetes API server and generates metrics about object state (e.g., Pod readiness, Deployment replica counts, resource quota usage).
Alertmanager: Handles alerts sent by Prometheus rules, deduplicating, grouping, and routing notifications to external channels like PagerDuty or Slack.
Grafana Integration: Grafana acts as the visualization layer, connecting to Prometheus to render rich dashboards.
Kubernetes Container Logging Architecture & Aggregation
Kubernetes does not provide a native centralized storage system for container logs. Instead, it defines a standard logging architecture based on Twelve-Factor application principles.
Node-Level Logging Pipeline
Standard Streams ( stdout / stderr ): Containerized applications write log messages directly to standard output ( stdout ) and standard error ( stderr ).
Container Runtime File Capture: The node container runtime (containerd or CRI-O) redirects these streams into log files on the host node filesystem, typically located at /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/0.log .
kubectl logs Command: Running kubectl logs <pod-name> queries the local node kubelet API, which streams text directly from these local node log files. ┌────────────────────────────────────────────────────────────────────────┐
- │ WORKER NODE │
- │ │
- │ ┌───────────────────┐ Writes stdout ┌─────────────────────────────┐ │
- │ │ Containerized App │─────────────────►│ Container Runtime Log File │ │
- │ └───────────────────┘ │ (/var/log/pods/...) │ │
- │ └──────────────┬──────────────┘ │
- │ │ Tails Log File │
- │ ┌─────────────────────────────────────────────────────▼──────────────┐ │
- │ │ Fluent Bit Logging Agent (DaemonSet Instance on Every Node) │ │
- │ └─────────────────────────────────────────────────────┬──────────────┘ │
- └───────────────────────────────────────────────────────┼────────────────┘
- │ Ships Logs
- ▼
- ┌────────────────────────────────┐
- │ Centralized Storage Engine │
- │ (Grafana Loki / Elasticsearch)│
- └────────────────────────────────┘
- Centralized Log Aggregation Patterns
To persist logs when Pods terminate or nodes rotate, log aggregation agents operate using two main architectural patterns:
Node-Level Agent (DaemonSet Pattern - Recommended): A lightweight log collector (such as Fluent Bit , Fluentd , or Vector) runs as a DaemonSet on every node. It tails all log files in /var/log/pods , enriches each log line with Kubernetes API metadata (namespace, pod name, container name, labels), and streams them to a centralized store like Grafana Loki or Elasticsearch .
Sidecar Container Pattern: A secondary container runs alongside the app container in the same Pod to read logs from a shared volume or custom log file when the app cannot write to stdout / stderr .
Distributed Tracing & OpenTelemetry
While metrics indicate that a system is failing, and logs detail why an error occurred, Distributed Tracing explains where latency or failure occurred across a network of interconnected microservices.
Core Distributed Tracing Concepts
Span: The fundamental building block of a trace. Represents a single named, timed contiguous unit of execution within a service (e.g., executing a database SQL query or calling an external HTTP API). A span contains start/end timestamps, tags, logs, and context metadata.
Trace: A collection of structured spans forming a Directed Acyclic Graph (DAG) that represents the complete end-to-end execution path of a single client request across multiple microservices.
Span Context Propagation: As a request travels between microservices over HTTP or gRPC, services pass tracing context metadata—primarily a unique TraceID and SpanID —via HTTP headers (standardized by the W3C Trace Context header traceparent ). [Client Request] ──► [API Gateway (Span A)] ──► [Auth Service (Span B)]
└──► [Order Service (Span C)] ──► [Database Query (Span D)]
- Total Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
- OpenTelemetry (OTel) & Tracing Backends
OpenTelemetry (OTel): A CNCF graduated project resulting from the merger of OpenTracing and OpenCensus. OTel provides a unified, vendor-agnostic set of APIs, SDKs, and the OTel Collector to generate, process, and export metrics, logs, and traces.
Jaeger & Zipkin: Open-source distributed tracing storage engines and UIs (both CNCF projects) that receive trace spans from OTel agents and render interactive visual timeline graphs for latency analysis.