Study guide
6.7 Container & Node Logging Architecture, Journalctl & Fluentd
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.7 Container & Node Logging Architecture, Journalctl & Fluentd
Key Takeaways Explain This Topic AI
Kubernetes container logging relies on applications writing directly to stdout and stderr streams; the Container Runtime Interface (CRI) captures these streams and stores them in /var/log/pods/<namespace> <name> <uid>/<container>/<retry>.log.
Log files on nodes are managed via symlinks in /var/log/containers/; kubelet and containerd enforce log rotation policies (e.g., containerLogMaxSize and containerLogMaxFiles) to prevent host disk exhaustion.
The 'kubectl logs' CLI provides versatile streaming options: '-f' (follow), '-p' (--previous for crashed instances), '--all-containers' (multi-container pods), '--timestamps', '--tail=N', and '-l' (label selector aggregation).
Host-level and systemd daemon logging requires 'journalctl' commands on node instances ('journalctl -u kubelet -e --no-pager', 'journalctl -u containerd').
Cluster-level logging architectures deploy node-level logging agents (DaemonSets like Fluent Bit, Fluentd, or Promtail) that scrape /var/log/pods and forward structured logs to centralized backends (Elasticsearch, Loki, Cloud Logging).
Last updated: August 2026
6.7 Container & Node Logging Architecture, Journalctl & Fluentd
In containerized environments, applications are ephemeral. When a container crashes, restarts, or is rescheduled to another node, its local in-memory state and ephemeral files are lost. To maintain observability and troubleshoot complex failures, Kubernetes establishes a standardized logging architecture .
Understanding how application output travels from standard streams to on-disk files, how kubectl logs streams data, how host daemons log to systemd-journald , and how cluster-wide log forwarders operate is vital for the CKA exam and production engineering.
1. Container Engine Logging Pipeline & Storage Layout
Kubernetes assumes that containerized applications write their operational logs directly to standard output ( stdout ) and standard error ( stderr ). +-----------------------------------------------------------------------------------------+
- | CONTAINER LOG PIPELINE ARCHITECTURE |
- | |
- | [APPLICATION IN CONTAINER] |
- | | (stdout / stderr) |
- | v |
- | [CONTAINER RUNTIME (containerd / CRI-O)] |
- | - Formats logs into CRI format (timestamp stream log-line) |
- | - Writes to physical disk on host node: |
- | /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<retry-count>.log |
- | | |
- | +---> Symlinked to: /var/log/containers/<pod-name>_<namespace>_<container>.log |
- | | |
- | +====================================+====================================+ |
- | | | | |
- | v v v |
- | [KUBECTL LOGS STREAM] [NODE-LEVEL AGENT] [HOST DAEMONS] |
- | - Kubelet serves log stream - Fluentd / Fluent Bit / Promtail - kubelet |
- | to kube-apiserver over - DaemonSet mounts /var/log/pods - containerd |
- | HTTPS port 10250 - Forwards to Elastic / Loki - journalctl |
- +-----------------------------------------------------------------------------------------+
Physical On-Disk Log Structure:
- Primary Log Path: /var/log/pods/<namespace>_<pod-name>_<pod-uuid>/<container-name>/0.log
- Convenience Symlink: /var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log
- CRI Log Line Format: 2026-08-23T08:15:30.123456789Z stdout F [INFO] Server listening on port 8080
- 2026-08-23T08:15:31.987654321Z stderr F [ERROR] Failed to connect to database
2026-08-23T... : ISO 8601 UTC timestamp.
stdout / stderr : Origin stream.
F (Full) / P (Partial): Indicates whether the log line was complete or truncated across multiple chunks.
2. Kubelet & Containerd Log Rotation Mechanics
To prevent rogue applications from generating massive logs that exhaust host disk space ( DiskPressure ), both the kubelet and containerd implement automatic log rotation.
Kubelet Configuration Parameters ( /var/lib/kubelet/config.yaml ):
containerLogMaxSize : Maximum file size before rotation (default: 10Mi ).
containerLogMaxFiles : Maximum number of rotated log archives retained per container (default: 5 ).
Kubelet asks the runtime to rotate logs according to configured size and file-count limits. Exact rotated filenames and whether archives are compressed are runtime-specific, so inspect /var/log/pods and the node runtime rather than assuming a 0.log.1.gz convention.
3. Mastering kubectl logs Syntax
kubectl logs queries the kubelet API on the target node via kube-apiserver : # 1. Basic log retrieval for a single-container pod
kubectl logs web-pod
- # 2. Specify container in a multi-container pod
- kubectl logs multi-pod -c app-container
- # 3. Stream logs across ALL containers in a multi-container pod simultaneously
- kubectl logs multi-pod --all-containers=true
- # 4. Stream logs from a previously crashed container instance
- kubectl logs web-pod --previous
- # 5. Follow live log output in real time (-f)
- kubectl logs -f web-pod
- # 6. View the last N lines (--tail)
- kubectl logs --tail=50 web-pod
- # 7. View logs generated in the last 15 minutes (--since)
- kubectl logs --since=15m web-pod
- # 8. Include RFC3339 timestamps in log output
- kubectl logs --timestamps web-pod
- # 9. Aggregate logs from all pods matching a label selector
- kubectl logs -l app=payment-service --tail=20
- 4. Host Systemd & OS Log Diagnostics ( journalctl )
Host-level systemd services (such as kubelet and containerd ) do not write to /var/log/pods . Their logs are managed by systemd-journald . # View kubelet logs starting from the end of the journal (-e) with line limit (-n)
journalctl -u kubelet -e --no-pager -n 100
- # Follow live kubelet logs
- journalctl -u kubelet -f
- # View containerd runtime logs
- journalctl -u containerd -e --no-pager -n 100
- # Filter logs by time window
- journalctl -u kubelet --since "2026-08-23 07:00:00" --until "2026-08-23 08:00:00"
- # Check kernel messages for hardware errors or OOM events
- dmesg -T | tail -n 50
- 5. Cluster-Level Logging Architectures
Kubernetes does not provide a native centralized storage solution for logs. Three primary architectural patterns exist for cluster-level logging: +-----------------------------------------------------------------------------------------+
- | CLUSTER LOGGING PATTERNS COMPARISON |
- | |
- | [PATTERN 1: NODE LOGGING AGENT DAEMONSET (Standard / Recommended)] |
- | - Fluent Bit / Fluentd runs as DaemonSet on every node. |
- | - Mounts host `/var/log/pods` read-only. |
- | - Parses CRI formatting, enriches with K8s metadata, forwards to Elastic / Loki. |
- | - Pros: Minimal resource overhead; zero changes to application pods. |
- | |
- | [PATTERN 2: SIDECAR LOGGING CONTAINER] |
- | - Sidecar container runs inside application pod. |
- | - Streams application log files (e.g., `/var/log/nginx/access.log`) to sidecar stdout |
- | or ships directly to central backend. |
- | - Pros: Useful for legacy applications that cannot log to stdout. |
- | - Cons: Doubled memory/CPU overhead per pod. |
- | |
- | [PATTERN 3: DIRECT APPLICATION LOGGING] |
- | - Application directly connects to logging backend (e.g., Logstash / Datadog API). |
- | - Cons: Tightly couples application code to logging infrastructure. |
- +-----------------------------------------------------------------------------------------+
Troubleshooting Missing Logs in Centralized Pipelines:
DaemonSet Pod Failed: Verify logging DaemonSet health: kubectl get pods -n logging .
Host Volume Mount Broken: Ensure Fluent Bit manifest mounts /var/log/pods and /var/log/containers .
Application Writing to Internal File: If kubectl logs produces nothing, the app may be logging to a private on-disk file instead of stdout / stderr .