3.1 Distributed Feature Engineering with AWS Glue & DataBrew

AWS Certified Machine Learning Engineer — Associate MLA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & Exam Strategy

1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS

2 Domain 1: Data Ingestion & Storage Architecture for ML

2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying

3 Domain 1: Feature Engineering & Data Transformation

3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store

4 Domain 1: Data Quality, Integrity & Ground Truth Labeling

4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus

5 Domain 2: Model Selection & Framework Architecture

5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock

6 Domain 2: SageMaker Model Training & Hyperparameter Optimization

6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)

7 Domain 2: Model Evaluation, Tracking & Model Registry

7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry

8 Domain 3: SageMaker Inference Options & Endpoint Architecture

8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)

9 Domain 3: Deployment Strategies & Infrastructure Scaling

9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration

10 Domain 3: MLOps Pipelines & Workflow Orchestration

10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining

11 Domain 4: Production Model Monitoring with SageMaker Model Monitor

11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch

12 Domain 4: Responsible AI, Bias Detection & Governance

12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance

13 Domain 4: ML Security, Network Isolation & Cost Optimization

13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance

3.1 Distributed Feature Engineering with AWS Glue & DataBrew

Key Takeaways Explain This Topic AI

AWS Glue DynamicFrames extend Apache Spark DataFrames by natively handling semi-structured, nested, and heterogeneous schemas without requiring upfront schema definitions.

Key DynamicFrame transformations include ResolveChoice to resolve type ambiguities, Relationalize to flatten nested JSON structures into relational tables, and Unbox to unpack embedded stringified JSON.

Glue Job Bookmarks maintain state across scheduled ETL executions by tracking processed object metadata (timestamps and offsets), preventing duplicate ingestion in incremental ML pipelines.

AWS Glue DataBrew provides over 250 visual, code-free transformations and automated data profiling, with direct recipe export to AWS Glue ETL or SageMaker Pipelines.

For Glue ETL scaling, standard worker type G.1X provides 1 DPU (4 vCPUs, 16 GB RAM) suitable for memory-intensive jobs, while G.2X provides 2 DPUs for compute-heavy ML feature transformations; Glue Auto Scaling dynamically provisions DPUs based on workload.

Last updated: August 2026

Distributed Feature Engineering with AWS Glue & DataBrew

Feature engineering at enterprise scale requires distributed data processing frameworks capable of handling terabytes to petabytes of tabular, semi-structured, and streaming data. For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master how to design scalable, cost-efficient, and maintainable data preparation pipelines using AWS Glue ETL , DynamicFrames , Job Bookmarks , and AWS Glue DataBrew . +-----------------------------------------------------------------------------------------+

• | DISTRIBUTED FEATURE ENGINEERING ON AWS |
• | |
• | [Raw Data Sources] [Processing Engines] [Feature Outputs] |
• | - S3 Raw Lake ---> - AWS Glue ETL (PySpark / Ray) --> - S3 Cleaned / Curated |
• | - RDS / DynamoDB - DynamicFrames Transformations - SageMaker Training S3 |
• | - Kinesis Streams - AWS Glue DataBrew (Visual) - SageMaker Feature Store |
• | - Redshift / JDBC - PySpark ML Pipelines - Glue Data Catalog |
• +-----------------------------------------------------------------------------------------+
• 1. AWS Glue ETL Architecture & Job Types

AWS Glue is a serverless data integration service that discovers, prepares, moves, and integrates data from multiple sources for machine learning and analytics. Glue manages the underlying compute infrastructure, provisioning Spark or Ray environments on demand.

Comparison of AWS Glue Job Types

Job Type Underlying Engine Primary Use Case in ML Compute Sizing & Scaling Cost & Startup Time

Spark (PySpark / Scala) Apache Spark Large-scale distributed feature transformation, heavy ETL on multi-GB/TB/PB datasets Worker types: G.1X , G.2X , G.4X , G.8X . Auto-scaling supported Higher cost per hour; fast startup with Glue 4.0 (~10-20 seconds)

Python Shell Single-node Python (3.9+) Lightweight preprocessing, small reference table lookups, API polling, Scikit-learn transformations on small datasets (<10 GB) Allocated as 0.0625 DPU (1 vCPU, 256 MB) or 1 DPU (4 vCPU, 16 GB) Lowest cost; near-instant startup (<5 seconds)

Ray Distributed Ray Core Distributed Python-native ML workloads, hyperparameter optimization, distributed Pandas/Modin workflows without Spark overhead Worker type: Z.2X (2 DPU = 8 vCPUs, 64 GB RAM). Auto-scaling supported Medium-high cost; optimized for Python ML libraries

Streaming ETL Spark Structured Streaming Real-time continuous feature extraction from Kinesis Data Streams or Apache Kafka (Amazon MSK) Worker types: G.025X , G.1X , G.2X . Runs continuously with micro-batches Billed per DPU-hour continuously while running

Tip

Exam Sizing Rule: If an exam question describes a lightweight feature extraction script (e.g., transforming a 500 MB CSV reference lookup or running basic NumPy/Pandas operations) that runs periodically, selecting a Python Shell job (at 0.0625 or 1 DPU) is significantly more cost-effective than provisioning a distributed PySpark cluster.

2. DynamicFrames & Specialized Glue Transformations

In standard Apache Spark, a DataFrame requires an explicit schema where every record in a column conforms to the same data type. However, raw ML data ingested from IoT devices, web event streams, or NoSQL databases often contains schema drift , nested fields, and heterogeneous data types (such as integers mixed with strings in the same column).

AWS Glue introduces the DynamicFrame , an extension of Spark DataFrames where each record is self-describing via a DynamicRecord . DynamicFrames allow schema flexibility without failing on unexpected types. +-----------------------------------------------------------------------------------------+

• | DYNAMICFRAME TRANSFORMATION FLOW |
• | |
• | +-----------------------+ ResolveChoice +-----------------------+ |
• | | DynamicFrame | --------------------> | Clean DynamicFrame | |
• | | - colA: int / string | (cast:int / make_cols)| - colA: int | |
• | +-----------------------+ +-----------------------+ |
• | | | |
• | | Relationalize | toDF() |
• | v v |
• | +-----------------------+ +-----------------------+ |
• | | Flattened Tables | | PySpark DataFrame | |
• | | - Root Table | | (Spark MLlib Pipeline)| |
• | | - Nested Child Tables | +-----------------------+ |
• | +-----------------------+ |
• +-----------------------------------------------------------------------------------------+
• Essential DynamicFrame Methods for ML Feature Engineering
• 1. ResolveChoice

When a column contains mixed types (e.g., "102" and 102 ), ResolveChoice determines how Glue resolves the conflict:

cast:target_type : Casts all values to a specified type (e.g., cast:int or cast:double ). Invalid values become null .

make_cols : Splits the ambiguous column into separate typed columns (e.g., col_int and col_string ).

make_struct : Preserves both types inside a nested structure containing all observed variants.

project:target_type : Keeps only records of the specified type and drops conflicting types. # Resolving type conflicts in a customer transaction dynamic frame

• resolved_df = dynamic_frame.resolveChoice(
• specs=[('transaction_amount', 'cast:double'), ('account_id', 'cast:string')],
• choice='match_catalog',
• database='ml_feature_store',
• table_name='customer_features'
• )
• 2. Relationalize

Extracts deeply nested JSON structures or arrays into a flattened root DynamicFrame and linked auxiliary child DynamicFrames, generating foreign keys to preserve relationships. # Flattening nested customer profile documents into relational tables

• flattened_collection = dynamic_frame.relationalize(
• root_table_name="customer_root",
• staging_path="s3://ml-lake-staging/relationalize/"
• )
• # Access the flattened root table
• customer_root_df = flattened_collection.select('customer_root')
• 3. Unbox

Unpacks stringified JSON embedded within a column into a structured DynamicFrame column type. # Unpacking JSON string payload into a struct

• unboxed_df = dynamic_frame.unbox(path="payload_json_str", target_type="struct")
• 4. Converting between DynamicFrame and PySpark DataFrame

To utilize PySpark's extensive pyspark.ml feature engineering library (such as vector assemblers or one-hot encoders), you must convert between DynamicFrames and DataFrames: # Convert DynamicFrame to Spark DataFrame

spark_df = dynamic_frame.toDF()

# Perform PySpark ML transformations...

# (e.g., StringIndexer, VectorAssembler)

• # Convert back to DynamicFrame for optimized S3/Glue Catalog write
• from awsglue.dynamicframe import DynamicFrame
• final_dynamic_frame = DynamicFrame.fromDF(transformed_spark_df, glueContext, "final_features")
• 3. Incremental Processing with AWS Glue Job Bookmarks

When training ML models continuously on scheduled intervals (e.g., daily or hourly retraining), processing the entire historical dataset repeatedly is computationally wasteful and expensive. AWS Glue Job Bookmarks maintain state information across job runs to ensure that only new or updated data is ingested. +-----------------------------------------------------------------------------------------+

• | GLUE JOB BOOKMARK STATE TRACKING |
• | |
• | Run #1 (08:00): [File 1 (07:30)] [File 2 (07:45)] ---> Processed & Bookmarked |
• | |
• | Run #2 (09:00): [File 1] (Skipped) [File 2] (Skipped) |
• | [File 3 (08:15)] [File 4 (08:50)] ---> Only New Files Processed |
• +-----------------------------------------------------------------------------------------+
• Job Bookmark Operational States

Enable: Glue records the state (S3 object last-modified timestamps, JDBC transaction IDs, or partitions). Subsequent runs process only records added since the previous run.

Disable: Glue ignores prior states and processes all source data from the beginning on every run.

Pause: Glue reads the state from a prior run to avoid reprocessing old data, but does not update the bookmark at the conclusion of the current run (ideal for testing pipeline scripts against a fixed new batch). import sys

• from awsglue.utils import getResolvedOptions
• from awsglue.context import GlueContext
• from pyspark.context import SparkContext
• from awsglue.job import Job

• args = getResolvedOptions(sys.argv, ['JOB_NAME'])
• sc = SparkContext()
• glueContext = GlueContext(sc)
• job = Job(glueContext)

• # Initialize job with bookmark support
• job.init(args['JOB_NAME'], args)

• # Read incrementally using the bookmark state
• source_dynf = glueContext.create_dynamic_frame.from_catalog(
• database="telemetry_db",
• table_name="device_logs",
• transformation_ctx="device_logs_ctx" # transformation_ctx is required for bookmarks
• )

# ... Feature engineering transformations ...

• # Commit state to update the bookmark
• job.commit()

Important

Bookmark Requirement: For Job Bookmarks to track state properly in Glue scripts, you must supply a unique transformation_ctx string in source/sink method calls and invoke job.commit() at the end of the script.

4. Visual Feature Preparation with AWS Glue DataBrew

AWS Glue DataBrew is a visual data preparation tool that enables data scientists and ML engineers to clean, normalize, and featurize data with an interactive, point-and-click interface featuring over 250 pre-built transformations without writing code. +-----------------------------------------------------------------------------------------+

• | GLUE DATABREW WORKFLOW |
• | |
• | [Data Sources] [Interactive Profiling] [Production Execution] |
• | - S3 (CSV, Parquet, JSON) - 250+ Visual Transforms - DataBrew Recipe Jobs |
• | - Redshift / RDS - Automated Data Profile - Export to S3 / Lake |
• | - Lake Formation - Correlation Matrix - Trigger via EventBridge |
• | - AppFlow Connectors - Missing Value Imputation - SageMaker Pipeline Step |
• +-----------------------------------------------------------------------------------------+
• Key Capabilities of Glue DataBrew for ML Engineers

Automated Data Profiling: Generates rich statistical distributions, identifies missing value rates, detects numerical outliers, surfaces duplicate rows, and computes feature correlation matrices across numeric features.

Visual Transformation Recipes: Transformations are saved as reusable, version-controlled Recipes (JSON/YAML action sequences) containing steps such as:

One-hot encoding and label/ordinal encoding.

Mathematical scalers: Z-score standard scaling, Min-Max normalization, logarithmic transforms.

Text cleaning: Tokenization, stop-word removal, regex pattern replacement, whitespace stripping.

Datetime decomposition: Extracting day of week, hour, quarter, is_weekend indicators.

Lineage & Versioning: Tracks end-to-end transformation lineage from raw source to curated destination.

Production Recipe Execution: DataBrew recipes can be applied across full multi-terabyte datasets by running DataBrew Recipe Jobs , which can be orchestrated as steps within SageMaker Pipelines or triggered by Amazon EventBridge .

5. Distributed Feature Engineering with PySpark ML

When writing custom PySpark feature transformations at scale within AWS Glue ETL, ML engineers utilize pyspark.ml.feature estimators and transformers. from pyspark.ml import Pipeline

• from pyspark.ml.feature import (
• StringIndexer,
• OneHotEncoder,
• VectorAssembler,
• StandardScaler,
• Imputer
• )

• # 1. Impute missing numerical values with median
• imputer = Imputer(
• inputCols=["age", "annual_income", "credit_score"],
• outputCols=["age_imputed", "income_imputed", "credit_imputed"]
• ).setStrategy("median")

• # 2. Convert categorical string labels to category indices
• string_indexer = StringIndexer(
• inputCols=["employment_type", "education_level"],
• outputCols=["employment_index", "education_index"],
• handleInvalid="keep" # Handle unseen categories gracefully in test data
• )

• # 3. One-hot encode indexed categorical columns
• one_hot_encoder = OneHotEncoder(
• inputCols=["employment_index", "education_index"],
• outputCols=["employment_vec", "education_vec"]
• )

• # 4. Assemble numeric features into a single vector for scaling
• numeric_assembler = VectorAssembler(
• inputCols=["age_imputed", "income_imputed", "credit_imputed"],
• outputCol="numeric_features_raw"
• )

• # 5. Standard scale numeric features to zero mean and unit variance
• scaler = StandardScaler(
• inputCol="numeric_features_raw",
• outputCol="numeric_features_scaled",
• withMean=True,
• withStd=True
• )

• # 6. Assemble final master feature vector for ML model training
• final_assembler = VectorAssembler(
• inputCols=["numeric_features_scaled", "employment_vec", "education_vec"],
• outputCol="features"
• )

• # Combine all transformation stages into a single deterministic Pipeline
• feature_pipeline = Pipeline(stages=[
• imputer,
• string_indexer,
• one_hot_encoder,
• numeric_assembler,
• scaler,
• final_assembler
• ])

• # Fit on training data and transform
• pipeline_model = feature_pipeline.fit(spark_df)
• transformed_dataset = pipeline_model.transform(spark_df)

Warning

StringIndexer Trap: In production streaming or batch test sets, new unseen category strings will cause StringIndexer to throw an error by default ( handleInvalid="error" ). Always specify handleInvalid="keep" or handleInvalid="skip" in ML feature engineering pipelines to prevent pipeline crashes on new categorical values.

6. Performance Tuning & DPU Sizing for AWS Glue ETL

In AWS Glue, compute capacity is measured in Data Processing Units (DPUs) . 1 DPU provides 4 vCPUs and 16 GB of memory .

Glue Worker Types Breakdown +-----------------------------------------------------------------------------------------+

• | GLUE WORKER TYPES MATRIX |
• | |
• | Worker Type DPU / Worker vCPU RAM Executors / Worker Best For |
• | ----------- ------------ ---- ------ ------------------ --------------- |
• | G.1X 1 DPU 4 16 GB 1 executor Memory-intensive |
• | G.2X 2 DPU 8 32 GB 1 large executor Compute-heavy ML |
• | G.4X 4 DPU 16 64 GB 1 massive executor Extreme memory |
• | G.8X 8 DPU 32 128 GB 1 ultra executor Terabyte joins |
• | G.025X 0.25 DPU 2 4 GB Micro-executor Streaming logs |
• +-----------------------------------------------------------------------------------------+
• Glue Performance Optimization Strategies

Glue Auto Scaling: Set --enable-auto-scaling=true . Glue automatically increases or decreases worker nodes based on real-time Spark execution stage metrics, preventing over-provisioning during simple read/write steps and scaling up during heavy transformations.

Mitigating Skew with Partitioning: Uneven partition sizes lead to "straggler tasks" where one Spark executor processes significantly more data than others. Use df.repartition(num_partitions, "partition_col") to distribute workloads evenly, or coalesce() to reduce partition count without full shuffling when saving output.

Broadcast Joins: When joining a large multi-million row transaction table with a small categorical lookup table (< 100 MB), force a broadcast join using from pyspark.sql.functions import broadcast; large_df.join(broadcast(small_df), "id") . This eliminates expensive distributed shuffle operations across the cluster.

DynamicFrame Pushdown Predicates: Filter partitions directly at the catalog level before reading files from S3 to minimize I/O overhead: glueContext.create_dynamic_frame.from_catalog(

• database="ml_lake",
• table_name="transactions",
• push_down_predicate="year == '2026' and month == '08'"
• )