Study guide
6.9 Service, Endpoint & Ingress Connectivity Troubleshooting
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
6.9 Service, Endpoint & Ingress Connectivity Troubleshooting
Key Takeaways Explain This Topic AI
Kubernetes Service routing depends on accurate label selector matching; if a Service's 'spec.selector' does not match the target Pod's 'metadata.labels', no Endpoints or EndpointSlices will be populated (Endpoints will show '<none>').
A Pod will be omitted from Service Endpoints even if labels match if the Pod's readinessProbe is failing or if the Pod is not in the 'Running' phase.
Port mapping mismatches are a frequent source of connectivity failures: 'spec.ports[].port' is the external port exposed by the Service, while 'spec.ports[].targetPort' must match the container's active listening port.
Ingress routing failures occur due to missing IngressClasses, path routing mismatches (Prefix vs Exact vs ImplementationSpecific), rewrite annotation omissions, or backend Service name/port discrepancies.
Systematic network debugging utilizes interactive test pods ('kubectl run tmp-shell --rm -it --image=busybox:1.36') to execute 'nslookup', 'nc -zv', and 'curl' across every hop of the network path.
Last updated: August 2026
6.9 Service, Endpoint & Ingress Connectivity Troubleshooting
Networking in Kubernetes operates across multiple virtual layers: container virtual ethernet pairs ( veth ), CNI flat routing, virtual ClusterIP load-balancing via kube-proxy , cluster-internal DNS resolution via CoreDNS, and HTTP/HTTPS layer 7 ingress routing.
When a client or frontend microservice cannot communicate with a backend service, administrators must isolate the exact hop where traffic is dropped.
1. End-to-End Traffic Path & Hop-by-Hop Triage +-----------------------------------------------------------------------------------------+
- | END-TO-END SERVICE TRAFFIC PIPELINE |
- | |
- | [CLIENT POD] |
- | | |
- | v (1. Resolves DNS: backend-svc.default.svc.cluster.local -> 10.96.0.50) |
- | [CoreDNS] |
- | | |
- | v (2. Sends packet to ClusterIP 10.96.0.50:80) |
- | [KUBE-PROXY / IPTABLES / IPVS on Host Kernel] |
- | - Matches DNAT rule for 10.96.0.50:80 |
- | - Randomly selects backend Pod IP from Endpoints / EndpointSlices (e.g., 10.244.1.25) |
- | - Rewrites destination IP to 10.244.1.25:8080 (targetPort) |
- | | |
- | v (3. CNI Routes packet across node network overlay) |
- | [TARGET BACKEND POD (10.244.1.25)] |
- | - Listens on container port 8080 |
- | - Readiness probe must be PASSING (otherwise omitted from Endpoints) |
- +-----------------------------------------------------------------------------------------+
- 2. Service & Endpoints Diagnostic Checklist
When a ClusterIP or NodePort Service fails to route traffic, execute this sequential 5-step checklist: # Step 1: Verify the Service exists and has an assigned ClusterIP
kubectl get svc backend-svc
- # Step 2: Check if Endpoints and EndpointSlices exist for the service
- kubectl get endpoints backend-svc
- kubectl get endpointslices -l kubernetes.io/service-name=backend-svc
- Case A: Endpoints displays <none>
If kubectl get endpoints backend-svc shows ENDPOINTS: <none> , traffic cannot be routed because the Service has zero healthy backend pods. [TROUBLESHOOTING ZERO ENDPOINTS]
- 1. Check Service Selector: $ kubectl get svc backend-svc -o jsonpath='{.spec.selector}'
- Output: map[app:web-backend env:prod]
- 2. Check Pod Labels: $ kubectl get pods --show-labels
- Output: pod-1: app=web-backend, env=production <-- MISMATCH! (prod vs production)
- 3. Check Pod Health: $ kubectl get pods -l app=web-backend
- If Pods are Running but 0/1 Ready -> Readiness probe is failing!
Important
A pod is added to an Endpoints object only if :
All key-value pairs in service.spec.selector match the pod's metadata.labels .
The pod is in the Running phase.
The pod's readinessProbe is succeeding (or no readiness probe is configured).
Case B: Endpoints Populated, But Connection Refused / Times Out
If endpoints exist (e.g., 10.244.1.25:8080 ), but curling the ClusterIP fails: # Verify port mapping configuration in Service spec
kubectl get svc backend-svc -o yaml spec:
ports:
- name: http
- port: 80 # Port exposed on the Service ClusterIP
- targetPort: 8080 # Port where application container actually listens
- protocol: TCP
TargetPort Mismatch: If the application inside the container listens on port 3000 , but targetPort is set to 8080 , packets reach the pod but the Linux kernel returns TCP RST (Connection Refused).
Named Port Typo: If targetPort: http-web is used, ensure the container spec contains ports: [{name: http-web, containerPort: 8080}] .
3. Ingress Routing & Controller Diagnostics
An Ingress resource is merely a set of rules; traffic routing requires an active Ingress Controller (e.g., ingress-nginx ). +-----------------------------------------------------------------------------------------+
- | INGRESS ARCHITECTURAL FLOW |
- | |
- | External Client ---> [Ingress Controller (e.g., NGINX Pod / NodePort / LB)] |
- | | |
- | +---> Evaluates Host: app.example.com |
- | +---> Evaluates Path: /api (Prefix matching) |
- | | |
- | v (Routes directly to Pod IPs from Endpoints) |
- | [Backend Pods] |
- +-----------------------------------------------------------------------------------------+
High-Frequency Ingress Issues & Solutions:
Missing or Mismatched ingressClassName : spec:
ingressClassName: nginx # Must match an existing IngressClass: kubectl get ingressclass
Path Type Matching ( Prefix vs Exact ):
pathType: Exact matching /api will NOT match /api/v1/users .
Use pathType: Prefix with path: /api .
Rewrite Annotation Omission: If incoming traffic hits app.example.com/api/users , but the backend application expects requests at /users , include the NGINX rewrite annotation: metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: app.example.com
http:
paths:
- path: /api(/|$)(.*)
- pathType: ImplementationSpecific
backend:
service:
name: backend-svc
port:
number: 80
Inspect Ingress Controller Logs: kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=100
4. In-Cluster Connectivity Debugging Runbook
To prove whether a failure is DNS-related, Service-related, or container-related, run a temporary interactive test container: # 1. Launch a temporary debugging pod
kubectl run tmp-net --rm -it --image=busybox:1.36 --restart=Never -- /bin/sh
# --- Inside the test container shell ---
- # 2. Test DNS resolution of the service name
- nslookup backend-svc
- nslookup backend-svc.default.svc.cluster.local
- # 3. Test TCP port reachability on ClusterIP
- nc -zv 10.96.0.50 80
- # 4. Test HTTP payload response
- wget -qO- http://backend-svc:80/healthz
- # 5. Test direct Pod IP connectivity (Bypassing Service layer)
- # Retrieve Pod IP: kubectl get pods -l app=backend -o wide
- wget -qO- http://10.244.1.25:8080/healthz