Study guide
3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
Certified Kubernetes Administrator CKA
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: CKA Exam Environment & Cluster Architecture
1.1 CKA Exam Mechanics, Scoring & Remote Terminal Environment 1.2 Kubernetes Control Plane & Worker Node Components 1.3 etcd Architecture, Clustering & Raft Consensus 1.4 Initializing Clusters with Kubeadm 1.5 Highly Available (HA) Control Plane Topologies 1.6 Cluster Lifecycle & Step-by-Step Upgrades 1.7 etcd Snapshot Backup and Restore Procedures
2 Chapter 2: Cluster Extensions, Packaging & RBAC
2.1 Kubernetes TLS Certificates & PKI Infrastructure 2.2 Managing Kubeconfig Files, Users, & Contexts 2.3 Installing Cluster Components with Helm & Kustomize 2.4 Role-Based Access Control (RBAC): Roles & RoleBindings 2.5 ClusterRoles & ClusterRoleBindings 2.6 Custom Resource Definitions (CRDs) & Operators 2.7 Pod Security Standards (PSS) & Admission Controllers
3 Chapter 3: Workloads, Lifecycle & Pod Scheduling
3.1 Pod Lifecycle, Multi-Container Patterns & Init Containers 3.2 Deployments: Rolling Updates, Rollouts & Rollbacks 3.3 DaemonSets & StatefulSets Architecture 3.4 Batch Workloads: Jobs & CronJobs 3.5 Managing ConfigMaps & Sensitive Data with Secrets 3.6 Resource Requests, Limits & Namespace ResourceQuotas 3.7 NodeSelectors, Node Affinity & Anti-Affinity 3.8 Node Taints & Pod Tolerations 3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
4 Chapter 4: Services, Networking & Ingress Architecture
4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium) 4.2 Services Deep Dive: ClusterIP, NodePort, LoadBalancer & EndpointSlices 4.3 Kube-Proxy Service Dataplane: iptables, nftables, IPVS & eBPF 4.4 CoreDNS Configuration, Resolution & Custom Forwarding 4.5 NetworkPolicies: Ingress, Egress & Default-Deny Isolation 4.6 Ingress Architecture, Controllers & Path-Based Routing 4.7 Gateway API: GatewayClass, Gateway, HTTPRoute & Traffic Splitting 4.8 External Traffic Ingress: NodePort, Cloud LoadBalancers & MetalLB
5 Chapter 5: Storage Architecture & Persistent Volumes
5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes 5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle 5.3 StorageClasses, Dynamic Provisioning & VolumeBindingModes 5.4 Storage Access Modes, Reclaim Policies & Volume Expansion 5.5 Container Storage Interface (CSI) Architecture & Drivers 5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows
6 Chapter 6: Advanced Troubleshooting, Monitoring & Maintenance
6.1 Systematic Kubernetes Troubleshooting Methodology 6.2 Node Failure, Kubelet Diagnostics & Systemd Services 6.3 Control Plane Failures, Static Pods & Component Logs 6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation 6.5 Pod Startup Failures, CrashLoopBackOff & OOMKilled Diagnostics 6.6 Workload Scheduling Failures & Resource Constraint Resolution 6.7 Container & Node Logging Architecture, Journalctl & Fluentd 6.8 Metrics Server, Resource Utilization & Kubectl Top 6.9 Service, Endpoint & Ingress Connectivity Troubleshooting 6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors 6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
3.9 Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
Key Takeaways Explain This Topic AI
The Horizontal Pod Autoscaler (HPA) operates as a control loop in kube-controller-manager, scaling Pod replicas dynamically based on resource utilization, custom in-cluster metrics, or external metrics.
HPA replica calculation uses the exact formula: desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)], evaluated outside a ±10% tolerance band.
Scaling on resource metrics (CPU and Memory) requires metrics-server and mandatory resources.requests definitions on all targeted Pod containers; missing requests result in <unknown> metric errors.
The autoscaling/v2 API specification supports multi-metric scaling, custom metrics (custom.metrics.k8s.io), external metrics (external.metrics.k8s.io), and granular behavior stabilization policies.
The behavior block allows administrators to configure scaleUp and scaleDown policies, selectPolicy algorithms (Min, Max, Disabled), and stabilization windows (default 300s for scaleDown) to prevent thrashing.
Last updated: August 2026
Autoscaling: Horizontal Pod Autoscaler (HPA) & Custom Metrics
In modern cloud architectures, static workload sizing leads either to resource starvation during traffic spikes or wasted infrastructure spend during low-utilization periods. The Kubernetes Horizontal Pod Autoscaler (HPA) automates the horizontal scaling of Deployments, ReplicaSets, and StatefulSets by continuously monitoring compute utilization and application-level metrics. Mastering the autoscaling/v2 API specification , the mathematical scaling formula , metrics aggregation pipelines , and stabilization behavior tuning is a critical requirement for production cluster administration and the CKA exam.
1. HPA Architecture and Control Loop Mechanics
The HPA is implemented as a continuous control loop within kube-controller-manager . Periodically (governed by the --horizontal-pod-autoscaler-sync-period flag, default 15 seconds ), the controller manager queries the Kubernetes Metrics APIs to evaluate workload utilization against target thresholds. +---------------------------------------------------------------------------------------------------+
- | HPA CONTROL LOOP & METRIC PIPELINE |
- | |
- | +------------------------------+ |
- | | kube-controller-manager | |
- | | (HPA Sync Period: every 15s) | |
- | +------------------------------+ |
- | | |
- | +-----------------------------+-----------------------------+ |
- | | | | |
- | v v v |
- | +-------------------------+ +-------------------------+ +-------------------------+ |
- | | Resource Metrics API | | Custom Metrics API | | External Metrics API |
- | | (metrics.k8s.io) | | (custom.metrics.k8s.io) | | (external.metrics.k8s.io)|
- | +-------------------------+ +-------------------------+ +-------------------------+ |
- | | | | |
- | v v v |
- | +-------------------------+ +-------------------------+ +-------------------------+ |
- | | metrics-server | | Prometheus Adapter / | | CloudWatch / Datadog / | |
- | | (kubelet cAdvisor CPU/M)| | KEDA (HTTP RPS, Latency)| | SQS Queue Depth Adapter | |
- | +-------------------------+ +-------------------------+ +-------------------------+ |
- | | |
- | v |
- | +------------------------------+ |
- | | Scale Subresource (/scale) | |
- | | Deployment.spec.replicas | |
- | +------------------------------+ |
- +---------------------------------------------------------------------------------------------------+
- The Three Metric Pipeline APIs
Resource Metrics ( metrics.k8s.io ) :
Provides core container CPU and Memory utilization collected directly from the kubelet embedded cAdvisor engine.
Backed by the in-cluster metrics-server add-on.
Custom Metrics ( custom.metrics.k8s.io ) :
Exposes application-level metrics originating from within the cluster (e.g., HTTP requests per second, active WebSocket connections, queue latency).
Backed by custom metric adapters such as the Prometheus Adapter or KEDA (Kubernetes Event-driven Autoscaling) .
External Metrics ( external.metrics.k8s.io ) :
Exposes metrics originating outside the Kubernetes cluster boundary (e.g., AWS SQS queue length, GCP Pub/Sub backlog depth, Azure Service Bus message count).
Handled by cloud-provider metric adapters.
2. The Mathematical Scaling Algorithm
The HPA calculates the desired number of replicas using a deterministic mathematical formula based on the ratio between the currently observed metric value and the target metric value: desiredReplicas = ⌈ currentReplicas × ( currentMetricValue desiredMetricValue ) ⌉ \text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \left( \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right) \right\rceil desiredReplicas = ⌈ currentReplicas × ( desiredMetricValue currentMetricValue ) ⌉
Where ⌈ x ⌉ \lceil x \rceil ⌈ x ⌉ is the ceiling function (rounding up to the nearest integer).
The Tolerance Band (+/- 10%)
To prevent micro-oscillations and scale thrashing caused by minor metric fluctuations, the HPA controller applies a tolerance band (default: 0.1 0.1 0.1 or 10 % 10\% 10% , configurable via --horizontal-pod-autoscaler-tolerance ): ∣ 1 − currentMetricValue desiredMetricValue ∣ ≤ 0.10 ⟹ No Scaling Action \left| 1 - \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right| \le 0.10 \implies \text{No Scaling Action} 1 − desiredMetricValue currentMetricValue ≤ 0.10 ⟹ No Scaling Action
- Step-by-Step Numerical Example
- Current Replicas : 4 4 4
- Target CPU Utilization : 50 % 50\% 50%
- Current Average CPU Utilization : 82 % 82\% 82%
- Compute Metric Ratio : Ratio = 82 50 = 1.64 \text{Ratio} = \frac{82}{50} = 1.64 Ratio = 50 82 = 1.64
Evaluate Tolerance Band : ∣ 1 − 1.64 ∣ = 0.64 > 0.10 ( Exceeds tolerance, scaling required ) | 1 - 1.64 | = 0.64 > 0.10 \quad (\text{Exceeds tolerance, scaling required}) ∣1 − 1.64∣ = 0.64 > 0.10 ( Exceeds tolerance, scaling required )
Apply Formula : desiredReplicas = ⌈ 4 × 1.64 ⌉ = ⌈ 6.56 ⌉ = 7 \text{desiredReplicas} = \lceil 4 \times 1.64 \rceil = \lceil 6.56 \rceil = 7 desiredReplicas = ⌈ 4 × 1.64 ⌉ = ⌈ 6.56 ⌉ = 7
Result : HPA scales the Deployment from 4 4 4 to 7 7 7 replicas .
Metric Target Types
Utilization : Percentage of the Pod's requested resource (e.g., averageUtilization: 75 ). Requires resources.requests on all containers .
AverageValue : Direct average value across all running Pods (e.g., averageValue: 500m CPU or averageValue: 100 requests/sec).
Value : Absolute total value regardless of replica count (used exclusively in External and Object metrics).
Important
When target utilization ( type: Utilization ) is specified, the metric is calculated as: Utilization = Total Actual Usage Across All Pods Total Requested Value Across All Pods × 100 % \text{Utilization} = \frac{\text{Total Actual Usage Across All Pods}}{\text{Total Requested Value Across All Pods}} \times 100\% Utilization = Total Requested Value Across All Pods Total Actual Usage Across All Pods × 100% If the relevant resource request is missing, utilization is undefined for that Pod. The controller accounts conservatively for missing metrics; if it has no usable value for a configured metric, it cannot make a scaling decision from that metric and may show <unknown> .
3. The autoscaling/v2 API Specification
The autoscaling/v2 API supports multiple metrics and fine-grained behavior controls. autoscaling/v1 remains stable for basic CPU-based HPAs; the old autoscaling/v2beta2 API is deprecated/removed and should not be used. apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
- name: payment-processor-hpa
- namespace: production
spec:
scaleTargetRef:
- apiVersion: apps/v1
- kind: Deployment
- name: payment-processor
- minReplicas: 2
- maxReplicas: 20
metrics:
- # 1. Resource Metric: Average CPU Utilization
- type: Resource
resource:
name: cpu
target:
- type: Utilization
- averageUtilization: 70
- # 2. Resource Metric: Memory Average Value
- type: Resource
resource:
name: memory
target:
- type: AverageValue
- averageValue: 600Mi
- # 3. Custom In-Cluster Metric: HTTP Requests Per Second
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
- type: AverageValue
- averageValue: 1500
- # 4. External Cloud Metric: AWS SQS Queue Backlog
- type: External
external:
metric:
name: aws_sqs_queue_depth
selector:
matchLabels:
queue_name: payment_orders
target:
- type: Value
- value: 5000
- Multi-Metric Resolution Rule
When multiple metrics are configured in the metrics array, the HPA calculates the proposed desired replica count for each metric independently , and then selects the highest proposed replica count to guarantee system stability and headroom.
4. Advanced Scaling Behavior and Stabilization Windows
The behavior block provides granular control over the velocity of scaling up and scaling down, eliminating the classic "flapping" (rapid cycling between scale-up and scale-down) problem. spec:
behavior:
scaleUp:
- stabilizationWindowSeconds: 0
- selectPolicy: Max
policies:
- type: Percent
- value: 100 # Double the replicas
- periodSeconds: 15
- type: Pods
- value: 4 # Or add 4 pods
- periodSeconds: 15
scaleDown:
- stabilizationWindowSeconds: 300 # 5-minute cooldown window
- selectPolicy: Min
policies:
- type: Percent
- value: 10 # Remove at most 10% of replicas
- periodSeconds: 60
- type: Pods
- value: 1 # Or remove 1 pod per minute
- periodSeconds: 60
- Key Behavior Parameters
stabilizationWindowSeconds :
For scaleDown , defaults to 300 seconds (5 minutes) . The controller looks back over the last 5 minutes of calculated desired states and picks the maximum replica count observed during that window. This ensures traffic dips do not prematurely destroy capacity needed for subsequent spikes.
For scaleUp , defaults to 0 seconds , enabling instantaneous burst scale-up when traffic spikes occur.
selectPolicy :
Max : Chooses the policy that produces the highest change in replicas (fastest scale).
Min : Chooses the policy that produces the lowest change in replicas (most conservative scale).
Disabled : Completely disables scaling in that direction.
- 5. Imperative Management Commands # Imperatively create an HPA targeting a Deployment with 70% CPU target
- kubectl autoscale deployment frontend --cpu-percent=70 --min=2 --max=10
- # Inspect HPA state and metrics
- kubectl get hpa
- # View detailed events, status conditions, and evaluation metrics
- kubectl describe hpa frontend
- 6. HPA Status Conditions & Troubleshooting Guide
When running kubectl describe hpa <name> , inspect Conditions :
Condition Status Operational Meaning
AbleToScale True The HPA controller can access the target workload's /scale subresource and adjust replica counts.
ScalingActive True The HPA is actively receiving valid metric data from the API server. If False , metric values show <unknown> .
ScalingLimited True The desired scale is bounded by minReplicas or maxReplicas . +-----------------------------------------------------------------------------------------+
- | HPA TROUBLESHOOTING DECISION TREE |
- | |
- | [Symptom: HPA shows TARGETS: <unknown>/70%] |
- | | |
- | v |
- | 1. Is metrics-server running in kube-system? |
- | -> kubectl get pods -n kube-system -l k8s-app=metrics-server |
- | -> If missing: Deploy metrics-server manifest. |
- | | |
- | v (Running) |
- | 2. Can kubectl top nodes / pods fetch metrics? |
- | -> kubectl top pods |
- | -> If failing: Check metrics-server logs for TLS cert errors or |
- | add --kubelet-insecure-tls flag. |
- | | |
- | v (Working) |
- | 3. Do ALL containers in the target Deployment specify resources.requests.cpu? |
- | -> If requests are omitted, HPA CANNOT calculate percentage! |
- | -> Fix: Add resources.requests.cpu to container spec. |
- +-----------------------------------------------------------------------------------------+
- Common Root Causes of HPA Failures on CKA Exam
Missing resources.requests : If CPU requests are omitted, kubectl get hpa displays TARGETS: <unknown>/80% . HPA requires requests as the mathematical denominator.
Metrics-Server Flags : In test environments or bare-metal clusters with self-signed kubelet certificates, metrics-server crashes unless passed --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP .
Network Policies : Overly restrictive NetworkPolicies blocking ingress to worker node port 10250 prevent metrics-server from scraping cAdvisor .