8.2 High-Throughput Offline Inference with Batch Transform

AWS Certified Machine Learning Engineer — Associate MLA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & Exam Strategy

1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS

2 Domain 1: Data Ingestion & Storage Architecture for ML

2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying

3 Domain 1: Feature Engineering & Data Transformation

3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store

4 Domain 1: Data Quality, Integrity & Ground Truth Labeling

4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus

5 Domain 2: Model Selection & Framework Architecture

5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock

6 Domain 2: SageMaker Model Training & Hyperparameter Optimization

6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)

7 Domain 2: Model Evaluation, Tracking & Model Registry

7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry

8 Domain 3: SageMaker Inference Options & Endpoint Architecture

8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)

9 Domain 3: Deployment Strategies & Infrastructure Scaling

9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration

10 Domain 3: MLOps Pipelines & Workflow Orchestration

10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining

11 Domain 4: Production Model Monitoring with SageMaker Model Monitor

11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch

12 Domain 4: Responsible AI, Bias Detection & Governance

12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance

13 Domain 4: ML Security, Network Isolation & Cost Optimization

13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance

8.2 High-Throughput Offline Inference with Batch Transform

Key Takeaways Explain This Topic AI

SageMaker Batch Transform is designed for high-throughput offline inference across massive datasets in Amazon S3, provisioning ephemeral compute clusters that automatically terminate upon job completion to eliminate idle costs.

SplitType determines how input files are partitioned for container ingestion ( None sends whole files, Line splits CSV/JSONLines by newline, and RecordIO splits binary protobuf records).

BatchStrategy controls mini-batch packaging ( MultiRecord packs multiple records per HTTP request up to MaxPayloadInMB to maximize GPU/CPU utilization, whereas SingleRecord sends one record per request).

DataProcessing parameters ( InputFilter , OutputFilter , and JoinSource='Input' ) enable filtering input features and automatically joining original record identifiers with model prediction outputs without writing custom Glue/Spark ETL scripts.

Last updated: August 2026

High-Throughput Offline Inference with Batch Transform

Not all machine learning applications require persistent, low-latency online endpoints. When predictions are generated over large, static datasets on a recurring schedule (such as nightly customer churn scoring, weekly sales forecasting, monthly portfolio risk valuations, or processing millions of archived documents), deploying persistent real-time endpoints results in massive compute idle waste and unnecessary operational overhead.

Amazon SageMaker Batch Transform is a fully managed distributed processing service engineered specifically for high-throughput, offline inference over massive datasets stored in Amazon S3. Batch Transform provisions an ephemeral cluster of compute instances, distributes input data partitions across workers, feeds mini-batches to model containers over local HTTP endpoints, persists inference results to Amazon S3, and immediately terminates all compute infrastructure upon completion.

1. Batch Transform Architectural Workflow +--------------------------------------------------------------------------------------------------+

• | BATCH TRANSFORM EXECUTION LIFECYCLE |
• | |
• | [Amazon S3 Input Bucket] |
• | - s3://input/data_part_01.csv (10 million rows) |
• | - s3://input/data_part_02.csv |
• | | |
• | v (Data Partitioning & SplitType='Line') |
• | +------------------------------------------------------------------------------------------+ |
• | | SAGEMAKER MANAGED TRANSFORM CLUSTER | |
• | | | |
• | | [Instance 1 (ml.c5.4xlarge)] [Instance 2 (ml.c5.4xlarge)] | |
• | | +------------------------------------+ +------------------------------------+ | |
• | | | Container HTTP Server (Port 8080) | | Container HTTP Server (Port 8080) | | |
• | | | - InputFilter: $[1:] (Drops ID) | | - InputFilter: $[1:] (Drops ID) | | |
• | | | - MultiRecord: 6 MB mini-batches | | - MultiRecord: 6 MB mini-batches | | |
• | | | - Model scoring: /invocations | | - Model scoring: /invocations | | |
• | | | - JoinSource: 'Input' (Joins ID) | | - JoinSource: 'Input' (Joins ID) | | |
• | | +------------------------------------+ +------------------------------------+ | |
• | +------------------------------------------------------------------------------------------+ |
• | | |
• | v (AssembleWith='Line' & S3 Upload) |
• | [Amazon S3 Output Bucket] |
• | - s3://output/data_part_01.csv.out (ID + Prediction columns joined, ordered matching input) |
• | - s3://output/data_part_02.csv.out |
• | | |
• | v |
• | [Cluster Automatically Terminates - Zero Idle Billing Incurred] |
• +--------------------------------------------------------------------------------------------------+
• 2. Core Configuration Parameters

Fine-tuning Batch Transform parameters is essential to maximize container saturation, prevent out-of-memory (OOM) crashes, and ensure deterministic record ordering. +--------------------------------------------------------------------------------------------------+

• | BATCH TRANSFORM KEY PARAMETERS MATRIX |
• | |
• | Parameter Allowed Values Functional Purpose |
• | --------- -------------- ------------------ |
• | SplitType None, Line, RecordIO How input files are divided into |
• | individual inference records. |
• | |
• | BatchStrategy MultiRecord, SingleRecord Whether to pack multiple records |
• | into a single HTTP POST request. |
• | |
• | MaxPayloadInMB Integer (1 to 100 MB, def: 6) Maximum payload size sent to the |
• | container in each HTTP request. |
• | |
• | MaxConcurrentTransforms Integer (>= 0, def: 0 auto) Maximum parallel HTTP requests sent |
• | to a single worker container. |
• | |
• | AssembleWith None, Line How prediction outputs are stitched |
• | together in the final S3 object. |
• +--------------------------------------------------------------------------------------------------+
• 2.1 SplitType

SplitType defines how SageMaker splits input S3 files before feeding them to the container:

None (Default): The entire file is treated as a single undivided input payload. Required for file formats that cannot be split by line (e.g., raw JPEG/PNG image files, DICOM medical files, or whole uncompressed Parquet/ZIP archives).

Line : The input file is split by newline ( \n ) delimiters. Mandatory for tabular CSV, text files, and JSONLines ( application/jsonlines ). Enables SageMaker to distribute subsets of rows across concurrent workers.

RecordIO : Splits binary data formatted in MXNet RecordIO-protobuf format.

2.2 BatchStrategy

BatchStrategy determines how individual records are batched into HTTP requests sent to the model container's /invocations endpoint:

MultiRecord : SageMaker packs as many consecutive records as possible into a single HTTP POST payload until reaching MaxPayloadInMB . This dramatically increases inference throughput by saturating CPU/GPU tensor vectorization and minimizing HTTP round-trip overhead. Highly recommended for tabular models (XGBoost, Linear Learner).

SingleRecord : SageMaker sends exactly one record per HTTP POST invocation. Essential when processing heavy deep learning inputs (such as high-resolution images or variable-length audio clips) or when the serving container does not support vectorized mini-batch parsing.

2.3 Sizing and Concurrency Tuning: MaxPayloadInMB & MaxConcurrentTransforms

MaxPayloadInMB : Sets the maximum size (in megabytes, up to 100 MB) of each mini-batch payload. Increasing MaxPayloadInMB improves throughput for tabular data. However, if set too high for memory-intensive models, the container may run out of memory (OOM) and return HTTP 500 or 413 Request Entity Too Large errors.

MaxConcurrentTransforms : The maximum number of concurrent HTTP requests that SageMaker can send simultaneously to each worker container instance. Setting this parameter to 1 or 2 is critical for memory-constrained GPU workloads to avoid VRAM exhaustion.

2.4 AssembleWith

AssembleWith defines how model outputs from individual batch requests are concatenated into the final S3 output file:

None (Default): Writes the raw binary output chunks into S3 without delimiter formatting.

Line : Appends a newline ( \n ) character after each prediction record, ensuring the output file is a clean, row-by-row CSV or JSONLines file whose line numbers map one-to-one with the input file.

3. Zero-ETL Data Association with DataProcessing

In standard ML pipelines, input datasets contain non-predictive metadata columns (such as customer_id , account_number , or timestamp ) alongside feature columns. However, trained model containers (such as SageMaker XGBoost or Scikit-learn) expect pure feature arrays and crash if unexpected identifier strings are passed to /invocations .

Historically, engineers had to write separate AWS Glue ETL jobs to strip IDs before inference and join them back after inference. SageMaker Batch Transform DataProcessing natively solves this with zero extra ETL code. +--------------------------------------------------------------------------------------------------+

• | DATAPROCESSING FILTERING PIPELINE |
• | |
• | Input CSV Row: [ "CUST_98214", 34, 1.2, 0.45, 0 ] (ID is column 0, features are columns 1-4) |
• | | |
• | v |
• | [1. InputFilter: '$[1:]'] ---> Strips CUST_98214 ---> Passes [ 34, 1.2, 0.45, 0 ] to container |
• | | |
• | v |
• | [Model Container /invocations] |
• | Generates Prediction: [ 0.874 ] (Churn Prob) |
• | | |
• | +-----------------------------------+ |
• | | |
• | v |
• | [2. OutputFilter: '$'] ---> Extracts prediction output [ 0.874 ] |
• | | |
• | v |
• | [3. JoinSource: 'Input'] ---> Joins original input with prediction |
• | | |
• | v |
• | Final S3 Output: [ "CUST_98214", 34, 1.2, 0.45, 0, 0.874 ] |
• +--------------------------------------------------------------------------------------------------+

The Three DataProcessing Parameters:

InputFilter : Uses JSONPath syntax (or CSV column slicing) to filter data before sending it to the model container.

Example for CSV: $[1:] strips the first column (e.g., customer_id ) and forwards columns 1 through N (the numerical features) to the model.

Example for JSONLines: $.features extracts only the nested feature array.

OutputFilter : Uses JSONPath to select specific fields from the container's inference output to be included in the output file.

Example: $.prediction or $[0] filters out auxiliary metadata from container responses, retaining only raw scores.

JoinSource : Specifies whether to join the original input data with the model prediction output.

None (Default): Writes only the model predictions to S3.

Input : Concatenates the original input record with the model prediction. When combined with AssembleWith='Line' , the final S3 output file preserves complete end-to-end lineage (identifying record + predictions) without requiring post-processing database joins.

• 4. Python SDK Implementation Example import sagemaker
• from sagemaker.transformer import Transformer

• session = sagemaker.Session()
• role = sagemaker.get_execution_role()

• # Configure the Batch Transformer
• transformer = Transformer(
• model_name="xgboost-churn-production-v2",
• instance_count=4, # 4 distributed worker instances
• instance_type="ml.m5.2xlarge",
• strategy="MultiRecord", # Pack multiple records per HTTP POST
• max_payload=6, # 6 MB per mini-batch
• max_concurrent_transforms=4, # 4 parallel HTTP requests per worker
• assemble_with="Line", # Delimit output predictions with newlines
• output_path="s3://customer-scoring-prod/batch-output/2026-08-16/",
• sagemaker_session=session
• )

• # Execute the Transform Job with DataProcessing (Zero-ETL Join)
• transformer.transform(
• data="s3://customer-scoring-prod/raw-features/",
• data_type="S3Prefix",
• content_type="text/csv",
• split_type="Line", # Split input CSV files by newline
• input_filter="$[1:]", # Strip Column 0 (CustomerID) before inference
• join_source="Input", # Join original input row with prediction output
• output_filter="$[0, -1]" # Retain only CustomerID ($[0]) and ChurnScore ($[-1])
• )

• # Wait for transform completion
• transformer.wait()

5. Batch Transform vs. Real-Time vs. Asynchronous Inference Decision Guide +--------------------------------------------------------------------------------------------------+

• | OFFLINE VS ONLINE INFERENCE DECISION MATRIX |
• | |
• | Criterion Batch Transform Online Endpoints (RT / Async) |
• | --------- --------------- ----------------------------- |
• | Latency Expectation Hours / Offline batch Sub-second up to 1 hour |
• | Input Source Large S3 dataset files Individual HTTP requests / S3 URIs |
• | Compute Lifecycle Ephemeral (job auto-stops) Persistent or Scale-to-Zero |
• | Data Pre/Post Processing Built-in DataProcessing Requires Serial Pipeline / Lambda |
• | Pricing Billed only for job run Hourly instance or per-request |
• | Best Workloads Daily Churn, Weekly Demand API Backends, Mobile Apps, Real-Time |
• +--------------------------------------------------------------------------------------------------+

Tip

Exam Trap Alert: If an exam question asks how to run offline batch inference on CSV data where the first column contains a user_id that the model rejects, do NOT choose creating a separate AWS Glue ETL job to drop the column. The most architecturally efficient, serverless, and AWS-native solution is configuring DataProcessing with InputFilter='$[1:]' and JoinSource='Input' directly on the SageMaker Batch Transform job.

Loading diagram... Batch Transform DataProcessing Architecture