Study guide
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium)
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium)
Key Takeaways Explain This Topic AI
Kubernetes mandates a fundamental IP-per-Pod networking model where every Pod receives a unique, routable IP address and all Pods can communicate with all other Pods across nodes without Network Address Translation (NAT).
The Container Network Interface (CNI) standard defines how the kubelet invokes external network binary plugins located in /opt/cni/bin using configuration files in /etc/cni/net.d/ ordered lexicographically.
Flannel provides a lightweight Layer 3 overlay network (typically VXLAN on UDP port 4789) or host-gw backend, but fundamentally lacks support for Kubernetes NetworkPolicies.
Calico uses Border Gateway Protocol (BGP) for unencapsulated L3 routing or IP-in-IP/VXLAN encapsulation, enforcing rich L3/L4 NetworkPolicies via its Felix node agent and Linux iptables or eBPF.
Cilium leverages Linux kernel eBPF (Extended Berkeley Packet Filter) to replace kube-proxy, provide high-performance L3/L4/L7 policy enforcement, and deliver deep packet observability via Hubble without iptables overhead.
Last updated: August 2026
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium)
Networking is one of the most heavily tested domains on the Certified Kubernetes Administrator (CKA) examination. Kubernetes departs fundamentally from legacy container networking models (such as Docker's default host-port mapping and host-private NAT bridges) by establishing a clean, uniform, cluster-wide flat network abstraction known as the IP-per-Pod model .
To implement this model across heterogeneous infrastructure (bare-metal servers, virtual machines, public cloud VPCs), Kubernetes decouples the container runtime from network implementation details via the Container Network Interface (CNI) specification. Understanding how CNI plugins allocate IP addresses, wire virtual network interfaces, route inter-node traffic, and enforce security boundaries is essential for any production Kubernetes administrator.
1. The Kubernetes Network Model Requirements
The Kubernetes network specification establishes four foundational invariants that every cluster implementation must satisfy:
All Pods can communicate with all other Pods on any node without Network Address Translation (NAT).
All agents on a node (such as the kubelet and system daemons) can communicate with all Pods running on that same node.
The IP address that a Pod sees as its own is the exact same IP address that every other Pod and host in the cluster sees it as (no masquerading or port mapping).
Pods configured with hostNetwork: true can communicate with all Pods across all nodes within the network namespace of the host. +-----------------------------------------------------------------------------------------+
- | KUBERNETES IP-PER-POD NETWORK MODEL |
- | |
- | WORKER NODE 01 (Host IP: 192.168.1.10) WORKER NODE 02 (Host IP: 192.168.1.20) |
- | Pod CIDR: 10.244.1.0/24 Pod CIDR: 10.244.2.0/24 |
- | +------------------------------------+ +------------------------------------+ |
- | | +--------------+ +-------------+ | | +--------------+ +-------------+ | |
- | | | Pod A | | Pod B | | | | Pod C | | Pod D | | |
- | | | 10.244.1.2 | | 10.244.1.3 | | | | 10.244.2.2 | | 10.244.2.3 | | |
- | | +-------+------+ +------+------+ | | +-------+------+ +------+------+ | |
- | | | (veth) | (veth) | | | (veth) | (veth) | |
- | | v v | | v v | |
- | | [ cni0 / CNI Data Path ] | | [ cni0 / CNI Data Path ] | |
- | +------------------+-----------------+ +------------------+-----------------+ |
- | | | |
- | +================== UNDERLAY ================+ |
- | (No NAT / Routable or Overlay) |
- +-----------------------------------------------------------------------------------------+
Architectural Advantages of IP-per-Pod:
No Port Collision: Two applications running on port 8080 (e.g., microservices) can run concurrently on the same worker node without requiring port remapping or dynamic host port allocation.
Clean Service Discovery: DNS records directly map service names to stable virtual IPs (VIPs) or individual Pod IPs.
Simplified Telemetry & Logging: Network traces, firewall logs, and packet captures retain genuine, un-NATed IP addresses.
2. The Container Network Interface (CNI) Architecture
The Container Network Interface (CNI) is a CNCF project that standardizes how container runtimes (like containerd and CRI-O ) interact with network plugins to configure container network namespaces. +-----------------------------------------------------------------------------------------+
- | CNI PLUGIN EXECUTION LIFECYCLE |
- | |
- | [1. Kubelet / CRI Runtime] |
- | | (Spawns Pod sandbox & creates Linux network namespace: /var/run/netns/<id>) |
- | v |
- | [2. Read CNI Configs] ---> Inspects /etc/cni/net.d/ (Sorted Lexicographically) |
- | | (e.g., 10-calico.conflist, 10-flannel.conflist) |
- | v |
- | [3. Invoke CNI Binary] ---> Calls executable in /opt/cni/bin/ (e.g., calico, flannel) |
- | | Environment: CNI_COMMAND=ADD, CNI_CONTAINERID=..., |
- | | CNI_NETNS=/var/run/netns/..., CNI_IFNAME=eth0 |
- | v |
- | [4. IPAM & Veth Wiring]---> Calls IPAM plugin (host-local / calico-ipam) for Pod IP |
- | | Creates virtual ethernet pair (vethxxxx <-> eth0) |
- | | Plumbs eth0 into Pod netns, sets default gateway & routes |
- | v |
- | [5. JSON Response] ---> Returns assigned IP, MAC, routes back to Kubelet in JSON |
- +-----------------------------------------------------------------------------------------+
CNI Directory Layout & Configuration:
Configuration Directory ( /etc/cni/net.d/ ): Contains JSON or .conflist configuration files. The container runtime reads these files in lexicographical order ; the first file in alphabetical order defines the primary CNI plugin.
Binary Directory ( /opt/cni/bin/ ): Contains standalone executable binaries invoked by the runtime, such as bridge , loopback , host-local , calico , flannel , cilium-cni , portmap , and macvlan .
Core CNI Operations:
ADD : Invoked when a Pod sandbox is created. Creates the virtual interface inside the Pod namespace, assigns an IP address from the IPAM pool, sets up default routes, and connects the host-side veth interface to the host network.
DEL : Invoked when a Pod is terminated. Deallocates the Pod IP address back to the IPAM pool, deletes the veth pair, and cleans up routing/iptables entries.
CHECK : Probes an existing container's network namespace to verify that expected interfaces, routes, and IP allocations remain valid.
VERSION : Queries the plugin binary for supported CNI specification versions.
Important
The NetworkPluginNotReady Node Condition: When a worker node boots or joins a cluster, the kubelet remains in NotReady status with the message container runtime network not ready: NetworkPluginNotReady until a valid CNI configuration file is placed in /etc/cni/net.d/ and the backing CNI plugin daemonset starts successfully.
3. Deep Architectural Comparison: Flannel vs. Calico vs. Cilium
Production Kubernetes environments deploy different CNI implementations depending on performance requirements, underlying cloud fabric capabilities, and security policy requirements.
Feature / Dimension Flannel Project Calico Cilium
Primary Architectural Paradigm Layer 3 Overlay (VXLAN) / Host-GW Layer 3 Routing (BGP) & Overlay (VXLAN/IPIP) eBPF-native Kernel Bypass
NetworkPolicy Support NO (Requires separate policy engine) YES (Rich L3/L4 Policies) YES (Rich L3/L4 and L7 HTTP/gRPC Policies)
Data Plane Technology Linux Kernel Bridge + VXLAN Linux iptables / IPVS / eBPF Linux eBPF (Bypasses iptables & conntrack)
Service / Kube-Proxy Replacement No Optional (Calico eBPF data plane) Yes (Full kube-proxy replacement via eBPF)
Inter-Node Routing Protocols UDP encapsulation (Port 4789) BGP (Port 179) via BIRD daemon Direct routing or Geneve/VXLAN overlay
Encryption Capabilities WireGuard (recent releases) WireGuard / IPsec Native WireGuard / IPsec
Observability & Telemetry Basic Linux interface stats Felix metrics / Prometheus Hubble (L3/L4/L7 flow logs, DNS tracing)
Operational Complexity Very Low (Ideal for simple labs) Moderate to High High (Requires modern Linux kernel ≥ 5.4 \ge 5.4 ≥ 5.4 )
1. Flannel Architecture
Flannel (developed originally by CoreOS) is designed for simplicity. It allocates a subnet (e.g., a /24 from the cluster /16 Pod CIDR) to each node via etcd or the Kubernetes API ( spec.podCIDR ).
VXLAN Backend (Default): Flannel creates a virtual interface ( flannel.1 ) on each node. Outgoing Pod packets destined for a remote node are encapsulated inside a UDP packet (port 4789 ) by the Linux kernel VXLAN driver, transmitted over the physical underlay, and decapsulated by the remote node's flannel.1 device.
Host-GW Backend: If all nodes reside on the same Layer 2 physical network broadcast domain, Flannel configures direct static routes in the host kernel routing table (using the destination node IP as the next-hop gateway), eliminating overlay encapsulation overhead entirely.
Caution
Flannel Does NOT Enforce NetworkPolicies: Flannel is strictly a packet transport plugin. If you deploy a Kubernetes NetworkPolicy resource on a cluster running standard Flannel CNI, all network traffic will continue to be allowed . The policies are silently ignored by the data plane unless an auxiliary policy controller (like Canal, which pairs Flannel routing with Calico policy enforcement) is installed.
2. Project Calico Architecture
Calico supports multiple data planes: unencapsulated Layer 3 routing with BGP as well as IP-in-IP or VXLAN overlays. The selected mode depends on underlay routing and operator configuration; do not assume every Calico installation advertises Pod routes directly.
Felix: The core Calico node agent running as a DaemonSet. Programs Linux kernel routing tables, iptables chains, and eBPF maps to enforce NetworkPolicies and route packets to local veth interfaces.
BIRD (BGP Daemon): Runs on every node to peer with other nodes (Node-to-Node Mesh) or with Top-of-Rack (ToR) physical switches (BGP Route Reflectors), advertising host /26 or /24 Pod IP blocks.
IPAM (calico-ipam): Allocates dynamic IP blocks ( IPPool ) with support for cross-subnet CIDR allocation and CIDR auto-detection.
Encapsulation Options (for routed networks): If intermediate physical routers drop unknown Pod IPs, Calico can be configured to use IP-in-IP (protocol 4) or VXLAN (UDP port 4789) encapsulation selectively across subnet boundaries.
3. Cilium Architecture
Cilium is an advanced, high-performance CNI plugin engineered entirely around eBPF (Extended Berkeley Packet Filter) .
Kernel Bypass: Traditional packet forwarding requires traversing lengthy Netfilter/iptables chains inside the Linux kernel. Cilium attaches eBPF bytecode programs directly to the network device drivers (using XDP - eXpress Data Path ) and Linux socket layer ( sock_ops ).
Kube-Proxy Replacement: Cilium handles Kubernetes Services directly in eBPF maps at socket creation time, replacing kube-proxy and eliminating iptables/IPVS rule table bloat.
L7 Network Security: Cilium can inspect and filter HTTP methods, URL paths, headers, and gRPC methods directly in the data path.
Hubble: Built-in network observability platform providing real-time graphical visibility into service dependencies, packet drops, DNS resolution latencies, and TLS handshakes.
4. MTU Tuning & Encapsulation Overhead
A frequent source of mysterious network degradation, intermittent connection timeouts, and SSL/TLS handshake freezes in Kubernetes clusters is MTU (Maximum Transmission Unit) misconfiguration . +-----------------------------------------------------------------------------------------+
- | PACKET ENCAPSULATION & MTU OVERHEAD |
- | |
- | Standard Ethernet Frame (MTU: 1500 Bytes) |
- | +---------------------------------------------------------------------------------+ |
- | | Standard IP Header (20B) | TCP Header (20B) | TCP Payload / Data (1460 Bytes) | |
- | +---------------------------------------------------------------------------------+ |
- | |
- | VXLAN Encapsulated Frame (Requires 50 Bytes of Overhead) |
- | +---------------------------------------------------------------------------------+ |
- | | Outer IP (20B) | UDP (8B) | VXLAN (8B) | Inner MAC (14B) | Inner IP | TCP | Data| |
- | +---------------------------------------------------------------------------------+ |
- | |<--------------------------- 50 Bytes Overhead ---------------->| |
- | Effective Inner Pod MTU Must Be: 1500 - 50 = 1450 Bytes |
- +-----------------------------------------------------------------------------------------+
Encapsulation Overhead Calculation:
Standard Physical Underlay MTU: Typically 1500 bytes (or 9000 bytes for Jumbo Frames).
VXLAN Overhead: 50 bytes (Outer IP: 20B + UDP: 8B + VXLAN: 8B + Inner Ethernet Frame: 14B). The Pod network interface MTU must be set to ≤ 1450 \le 1450 ≤ 1450 .
Geneve Overhead: 50 bytes. Pod MTU must be ≤ 1450 \le 1450 ≤ 1450 .
IP-in-IP Overhead: 20 bytes (Outer IP header). Pod MTU must be ≤ 1480 \le 1480 ≤ 1480 .
WireGuard Encryption Overhead: 60 to 80 bytes depending on underlay encapsulation.
Tip
Symptom of MTU Mismatch: Small packets (such as ping ICMP requests and TCP SYN packets) succeed because they fit within the MTU, but large data transfers (such as curl downloading a large JSON payload or TLS certificate exchange) hang indefinitely due to unhandled packet fragmentation.
5. CNI Inspection & Troubleshooting Runbook
When diagnosing networking issues during the CKA examination, follow this systematic command runbook: # 1. Inspect CNI configuration directory
- ls -la /etc/cni/net.d/
- cat /etc/cni/net.d/10-calico.conflist
- # 2. Verify CNI plugin binaries are installed
- ls -la /opt/cni/bin/
- # 3. Check CNI DaemonSet pods in kube-system
- kubectl get pods -n kube-system -o wide -l k8s-app=calico-node
- kubectl get pods -n kube-system -o wide -l app=flannel
- kubectl get pods -n kube-system -o wide -l k8s-app=cilium
- # 4. View CNI agent logs on a failing worker node
- kubectl logs -n kube-system daemonset/calico-node -c calico-node --tail=100
- # 5. Inspect host network interfaces and routing tables
- ip link show
- ip route show
- # 6. Verify Pod IP address assignment and veth binding
- kubectl get pods -o wide
- ip netns list