Study guide
9.2 Endpoint Auto-Scaling & Hardware Acceleration
AWS Certified Machine Learning Engineer — Associate MLA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Introduction & Exam Strategy
1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS
2 Domain 1: Data Ingestion & Storage Architecture for ML
2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying
3 Domain 1: Feature Engineering & Data Transformation
3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store
4 Domain 1: Data Quality, Integrity & Ground Truth Labeling
4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus
5 Domain 2: Model Selection & Framework Architecture
5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock
6 Domain 2: SageMaker Model Training & Hyperparameter Optimization
6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)
7 Domain 2: Model Evaluation, Tracking & Model Registry
7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry
8 Domain 3: SageMaker Inference Options & Endpoint Architecture
8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)
9 Domain 3: Deployment Strategies & Infrastructure Scaling
9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration
10 Domain 3: MLOps Pipelines & Workflow Orchestration
10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining
11 Domain 4: Production Model Monitoring with SageMaker Model Monitor
11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch
12 Domain 4: Responsible AI, Bias Detection & Governance
12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance
13 Domain 4: ML Security, Network Isolation & Cost Optimization
13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance
9.2 Endpoint Auto-Scaling & Hardware Acceleration
Key Takeaways Explain This Topic AI
SageMaker real-time endpoints integrate with Application Auto Scaling to adjust instance capacity dynamically between MinCapacity and MaxCapacity per production variant.
Target Tracking Scaling is the recommended policy, automatically adjusting instance count to maintain predefined metrics such as SageMakerVariantInvocationsPerInstance calculated from load testing.
Step Scaling adjusts instance capacity based on graduated metric threshold violation bands with configurable scale-out and scale-in cooldown periods to prevent capacity oscillation (flapping).
Scheduled Scaling anticipates predictable, cyclical traffic patterns by executing time-based capacity adjustments at predefined cron intervals (e.g., scaling up before business hours).
AWS Inferentia (inf1/inf2) chips compiled via the AWS Neuron SDK deliver up to 50% lower cost-per-inference compared to GPU instances, while SageMaker Neo compiles models across hardware targets to reduce memory footprint and latency up to 2x without accuracy loss.
Last updated: August 2026
Endpoint Auto-Scaling & Hardware Acceleration
Operating production machine learning endpoints requires balancing strict latency Service Level Agreements (SLAs) with infrastructure cost efficiency. Under-provisioning compute resources leads to request timeouts, increased ModelLatency , and 5XX invocation errors during traffic surges. Conversely, over-provisioning static multi-instance GPU fleets during low-traffic windows results in massive idle cloud expenditure.
To achieve optimal price-performance, ML engineers must implement Application Auto Scaling policies tailored to workload dynamics and select the appropriate compute hardware acceleration (General Purpose CPUs, NVIDIA GPUs, AWS Inferentia, and AWS Trainium), supplemented by model optimization via SageMaker Neo and the AWS Neuron SDK .
1. SageMaker Endpoint Auto-Scaling Mechanics
SageMaker real-time endpoints integrate directly with Application Auto Scaling . Auto-scaling is configured independently at the Production Variant level, meaning different variants behind the same endpoint can have distinct scaling policies, instance bounds, and metric targets. +-----------------------------------------------------------------------------------------+
- | SAGEMAKER APPLICATION AUTO SCALING ARCHITECTURE |
- | |
- | Client Invocations |
- | | |
- | v |
- | [SageMaker Real-Time Endpoint Variant] |
- | (Current Fleet: 2 x ml.c6i.2xlarge Instances) |
- | | |
- | v |
- | [CloudWatch Metric: InvocationsPerInstance] |
- | | |
- | v |
- | +-----------------------------------------+ |
- | | Application Auto Scaling Controller | |
- | | Target: 800 Invocations / Min / Inst | |
- | +-----------------------------------------+ |
- | | |
- | +------------------------+------------------------+ |
- | | | |
- | v (Load > 800) v (Load < 800) |
- | [SCALE OUT ACTION] [SCALE IN ACTION] |
- | Adds +2 Instances Terminates Instances |
- | (Respects MaxCapacity = 10) (Respects MinCapacity = 2) |
- | (ScaleOutCooldown = 60s) (ScaleInCooldown = 300s) |
- +-----------------------------------------------------------------------------------------+
Core Auto-Scaling Components:
Scalable Target ( RegisterScalableTarget ): Registers the SageMaker resource identifier ( endpoint/<endpoint-name>/variant/<variant-name> ) and sets the operational instance boundaries:
MinCapacity : The minimum number of instances to maintain (e.g., 2 for Multi-AZ high availability).
MaxCapacity : The upper ceiling instance count to prevent runaway cloud billing during distributed denial-of-service (DDoS) or unexpected traffic anomalies.
Scaling Policies ( PutScalingPolicy ): Defines the mathematical logic and metrics that dictate when and how instances are provisioned or terminated.
The Three Auto-Scaling Policy Types +-----------------------------------------------------------------------------------------+
- | AUTO-SCALING POLICIES COMPARED |
- | |
- | 1. TARGET TRACKING SCALING (Recommended Default) |
- | - Dynamically adjusts capacity to keep metric at target value |
- | - Primary Metric: SageMakerVariantInvocationsPerInstance |
- | - Automatically creates CloudWatch alarms for scale-out and scale-in |
- | |
- | 2. STEP SCALING |
- | - Adjusts capacity based on graduated metric threshold violation steps |
- | - Example: If CPU > 70% add 1; if CPU > 85% add 3; if CPU > 95% add 5 |
- | - Requires manual configuration of ScaleOutCooldown and ScaleInCooldown |
- | |
- | 3. SCHEDULED SCALING |
- | - Time-based scaling executed via cron expressions for predictable traffic |
- | - Example: Scale to 10 instances at 08:00 UTC; Scale to 2 instances at 18:00 UTC |
- +-----------------------------------------------------------------------------------------+
- Deep Dive: Target Tracking Scaling & Metric Calculation
Target Tracking Scaling is the standard best practice for SageMaker endpoints. Rather than managing complex threshold rules, you define a target metric value, and Application Auto Scaling automatically calculates instance additions or removals using proportional control algorithms.
Primary Metric: SageMakerVariantInvocationsPerInstance : Represents the average number of invocations per minute processed by each individual instance in the variant fleet. InvocationsPerInstance = Total Invocations Across Variant Current Active Instance Count \text{InvocationsPerInstance} = \frac{\text{Total Invocations Across Variant}}{\text{Current Active Instance Count}} InvocationsPerInstance = Current Active Instance Count Total Invocations Across Variant
How to Determine the Target Metric Value:
Perform load testing on a single instance of your target instance type (e.g., ml.c6i.xlarge ) to find the maximum requests per minute ( R P S m a x × 60 RPS_{max} \times 60 RP S ma x × 60 ) the model handles before latency exceeds SLA.
Apply a safety buffer factor (typically 70% of peak capacity) to absorb sudden traffic spikes while new instances initialize: Target Value = ( Max Requests Per Minute per Instance ) × 0.70 \text{Target Value} = (\text{Max Requests Per Minute per Instance}) \times 0.70 Target Value = ( Max Requests Per Minute per Instance ) × 0.70 import boto3
as_client = boto3.client('application-autoscaling')
- # Step 1: Register Scalable Target
- as_client.register_scalable_target(
- ServiceNamespace='sagemaker',
- ResourceId='endpoint/recommendation-v1/variant/AllTraffic',
- ScalableDimension='sagemaker:variant:DesiredInstanceCount',
- MinCapacity=2,
- MaxCapacity=10
- )
- # Step 2: Configure Target Tracking Policy
- as_client.put_scaling_policy(
- PolicyName='InvocationsTargetTracking',
- ServiceNamespace='sagemaker',
- ResourceId='endpoint/recommendation-v1/variant/AllTraffic',
- ScalableDimension='sagemaker:variant:DesiredInstanceCount',
- PolicyType='TargetTrackingScaling',
- TargetTrackingScalingPolicyConfiguration={
- 'TargetValue': 750.0, # Maintain 750 invocations/min per instance
- 'PredefinedMetricSpecification': {
- 'PredefinedMetricType': 'SageMakerVariantInvocationsPerInstance'
- },
- 'ScaleOutCooldown': 60, # Scale out quickly (60 seconds)
- 'ScaleInCooldown': 300 # Scale in conservatively (5 minutes to prevent flapping)
- }
- )
Important
Scale-In vs. Scale-Out Cooldowns:
ScaleOutCooldown (e.g., 60s): Ensures that after adding instances, the policy waits for new instances to finish booting and start accepting traffic before evaluating if further scaling is needed.
ScaleInCooldown (e.g., 300s): Prevents premature termination of instances during temporary traffic dips, eliminating capacity oscillation (flapping) .
2. Compute Hardware Acceleration & Architecture Selection
Selecting the optimal hardware architecture for inference is critical for meeting latency SLAs while controlling infrastructure costs. +-----------------------------------------------------------------------------------------+
- | INFERENCE COMPUTE HARDWARE SELECTION SPECTRUM |
- | |
- | [General Purpose CPU] [NVIDIA GPU] [AWS Inferentia] |
- | ml.c6i / ml.c7g ml.g5 / ml.g6 ml.inf1 / ml.inf2 |
- | - Classical ML (XGBoost) - Computer Vision (CNNs) - Transformer NLP / LLMs |
- | - Low-concurrency DL - Large LLM Hosting - Sub-10ms DL Latency |
- | - Lowest base hourly rate - High matrix throughput - Up to 50% cost savings |
- +-----------------------------------------------------------------------------------------+
- Compute Family Breakdown
General Purpose CPU ( ml.c6i , ml.m6i , ml.c7g Graviton):
Powered by Intel Xeon Scalable or AWS Graviton3 processors.
Best For: Classical ML algorithms (XGBoost, Random Forests, Linear Models, Scikit-learn tabular pipelines), lightweight neural networks, and applications with modest latency requirements (>50ms).
Graviton3 ( ml.c7g ): Delivers up to 25% higher compute performance and up to 20% lower cost compared to comparable x86 c6i instances for CPU-based inference.
NVIDIA GPU Accelerators ( ml.g5 , ml.g6 , ml.p4d ):
ml.g5 Series: Powered by NVIDIA A10G Tensor Core GPUs (24 GB VRAM). Industry standard for deep learning vision models (YOLO, ResNet), speech recognition, and mid-sized NLP models.
ml.g6 Series: Powered by NVIDIA L4 GPUs with Ada Lovelace architecture, optimized for generative AI inference, video processing, and graphics.
ml.p4d / ml.p5 Series: Multi-GPU clusters (NVIDIA A100/H100) used for massive foundation models (70B+ LLMs) requiring distributed model parallelism (e.g., vLLM, TensorRT-LLM).
AWS Inferentia ( ml.inf1 , ml.inf2 ):
Purpose-built, custom ASIC silicon designed by AWS specifically for deep learning inference acceleration.
ml.inf1 (Inferentia1): Delivers high throughput and low cost for standard vision and NLP models (BERT, RoBERTa).
ml.inf2 (Inferentia2): Powered by NeuronCore-v2 , featuring direct high-speed NeuronLink-v2 interconnects between chips. Supports FP16, BF16, INT8, and FP8 data types. Capable of hosting large transformer models (e.g., Llama-3, Mistral, Stable Diffusion) with up to 50% lower cost-per-inference than comparable NVIDIA GPU instances.
AWS Trainium ( ml.trn1 , ml.trn1n ):
Purpose-built AWS silicon designed specifically for high-efficiency deep learning model training and fine-tuning, delivering up to 50% cost-to-train savings.
Hardware Selection Decision Matrix
Hardware Architecture Instance Family Underlying Silicon Ideal Workload Profile Cost/Performance Profile
x86 CPU ml.c6i , ml.m6i Intel Xeon Scalable Tabular models, XGBoost, Scikit-learn, low-traffic inference Standard baseline cost.
ARM CPU (Graviton) ml.c7g , ml.m7g AWS Graviton3 Tabular ML, pre/post-processing pipelines Up to 20% lower cost than x86 CPU.
NVIDIA GPU ml.g5 , ml.g6 NVIDIA A10G / L4 Computer Vision, CNNs, Audio processing, Transformer fine-tuning High throughput; higher hourly rate.
AWS Inferentia ml.inf1 , ml.inf2 Inferentia1 / Inferentia2 High-throughput Transformer NLP, LLM inference, real-time embeddings Lowest cost-per-inference (up to 50% savings over GPUs).
AWS Trainium ml.trn1 , ml.trn1n Trainium (NeuronCore-v2) Deep learning training, LLM pre-training, distributed training Up to 50% cost-to-train savings over GPUs.
3. Model Compilation & Hardware Optimization
Deploying unoptimized models directly to hardware results in bloated memory footprints and suboptimal inference execution. AWS provides two key compilation and optimization frameworks: +-----------------------------------------------------------------------------------------+
- | MODEL COMPILATION & OPTIMIZATION TOOLS |
- | |
- | +------------------------------------+ +------------------------------------+ |
- | | AWS NEURON SDK | | AMAZON SAGEMAKER NEO | |
- | | (Target: Inferentia & Trainium) | | (Target: CPU, GPU, Embedded/Edge)| |
- | | - Compiles PyTorch / HuggingFace | | - Framework-agnostic compiler | |
- | | - Ahead-of-Time (AOT) tracing | | - Converts to hardware binary | |
- | | - Packages with torch-neuronx | | - Up to 2x speedup, 10x smaller | |
- | +------------------------------------+ +------------------------------------+ |
- +-----------------------------------------------------------------------------------------+
- 1. AWS Neuron SDK
To run deep learning models on AWS Inferentia ( inf1 / inf2 ) or AWS Trainium ( trn1 ), models must be compiled using the AWS Neuron SDK ( torch-neuron for inf1, torch-neuronx for inf2/trn1):
Ahead-of-Time (AOT) Compilation: Traces the computation graph with sample input tensors and compiles graph operations into Neuron machine code.
Neuron Runtime: Executes compiled artifacts on physical NeuronCores with zero runtime compilation overhead. import torch
- import torch_neuronx
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
- # Load standard PyTorch model
- model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased')
- model.eval()
- # Generate dummy input for tracing
- dummy_input = torch.zeros([1, 128], dtype=torch.int64)
- # Ahead-of-time compilation for Inferentia2 (NeuronCore-v2)
- neuron_model = torch_neuronx.trace(model, (dummy_input, dummy_input))
- neuron_model.save('bert_neuron_compiled.pt')
- 2. Amazon SageMaker Neo
Amazon SageMaker Neo is an automated model compiler that optimizes machine learning models for execution on diverse target hardware platforms (Intel CPUs, ARM processors, NVIDIA GPUs, and edge devices) without losing prediction accuracy .
How SageMaker Neo Works:
Takes trained models from frameworks like PyTorch, TensorFlow, MXNet, ONNX, or XGBoost.
Analyzes the computation graph and performs hardware-specific kernel tuning, operator fusion, and memory layout optimization.
Emits a compiled binary paired with the lightweight Neo Runtime engine.
Benefits:
Reduces model memory footprint by up to 10x .
Accelerates inference execution speed by up to 2x .
Eliminates the need to install heavy framework dependencies (like the full PyTorch or TensorFlow libraries) in the inference container. import boto3
sagemaker_client = boto3.client('sagemaker')
- # Create a SageMaker Neo Compilation Job
- response = sagemaker_client.create_compilation_job(
- CompilationJobName='xgboost-neo-c6i-optimization',
- RoleArn='arn:aws:iam::123456789012:role/SageMakerExecutionRole',
- ModelPackageVersionArn='arn:aws:sagemaker:us-east-1:123456789012:model-package/xgboost-v1/1',
- InputConfig={
- 'S3Uri': 's3://ml-models/xgboost/model.tar.gz',
- 'DataInputConfig': '{"input": [1, 50]}',
- 'Framework': 'XGBOOST'
- },
- OutputConfig={
- 'S3OutputLocation': 's3://ml-models/xgboost-neo-compiled/',
- 'TargetPlatform': {
- 'Os': 'LINUX',
- 'Arch': 'X86_64'
- },
- 'TargetDevice': 'ml_c6i'
- },
- StoppingCondition={
- 'MaxRuntimeInSeconds': 900
- }
- )