Study guide

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations

AWS Certified Data Engineer - Associate DEA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Streaming Data Ingestion

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State

2 Chapter 2: Batch Data Ingestion & Migration

2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion

3 Chapter 3: ETL Transformation with AWS Glue

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage

4 Chapter 4: Big Data Processing & Workflow Orchestration

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD

5 Chapter 5: Object Storage & Data Lake Architecture

5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations

6 Chapter 6: Data Warehousing with Amazon Redshift

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum

7 Chapter 7: NoSQL, Relational & Specialty Data Stores

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing

8 Chapter 8: Pipeline Operations, Monitoring & Observability

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail

9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning

9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew

10 Chapter 10: High Availability, Backup & Cost Optimization

10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads

11 Chapter 11: Data Security, Encryption & Identity Management

11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints

12 Chapter 12: Data Governance, Privacy & Compliance

12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations

Key Takeaways Explain This Topic AI

AWS Glue runs serverless Apache Spark workloads using Data Processing Units (DPUs), where 1 DPU equals 4 vCPUs and 16 GB memory (or 8 vCPUs and 32 GB RAM on G.2X workers).

DynamicFrames extend PySpark DataFrames by handling semi-structured data with semi-rigid or evolving schemas without requiring pre-defined schema enforcement or failing on type mismatches.

Key DynamicFrame transformations like ApplyMapping, ResolveChoice, and Relationalize handle schema restructuring, type conflict resolution, and nested JSON unnesting natively.

Converting between DynamicFrames and PySpark DataFrames via toDF() and fromDF() enables leveraging advanced PySpark SQL window functions, UDFs, and complex joins.

Glue Auto Scaling can adjust workers during supported Spark jobs, while partition-aware writes, modern Glue runtimes, and the S3-optimized Parquet committer reduce idle capacity and avoid rename-heavy output patterns.

Last updated: August 2026

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations

AWS Glue is a fully managed, serverless data integration service that automates the extraction, transformation, and loading (ETL) of data across enterprise analytics storage environments. At the core of Glue's ETL engine is an optimized, serverless Apache Spark runtime configured with proprietary AWS extensions—most notably DynamicFrames —designed to handle massive, semi-structured, and schema-variant datasets.

Understanding how Glue provisions compute capacity, how DynamicFrames differ from native PySpark DataFrames, and how to apply specialized transformations is critical for passing the AWS Certified Data Engineer - Associate (DEA-C01) exam and building resilient data pipelines.

AWS Glue Jobs Execution Engine & Worker Types

AWS Glue decouples data engineers from cluster management by automatically provisioning, configuring, and scaling Apache Spark resources. Glue measures compute capacity in Data Processing Units (DPUs) . A single DPU provides a standardized allocation of vCPU and memory compute resources.

Glue Worker Specifications & Sizing Matrix

When configuring a Glue ETL Spark job, data engineers select specific worker types based on memory intensity, compute demands, and shuffle performance requirements:

Worker Type DPUs per Worker vCPU Allocation Memory (RAM) Disk per Worker (total / approximately free) Strong Fit

  • G.1X 1 DPU 4 vCPUs 16 GB 94 GB / 44 GB Standard transformations and moderate shuffles
  • G.2X 2 DPUs 8 vCPUs 32 GB 138 GB / 78 GB More memory and spill space per executor
  • G.4X 4 DPUs 16 vCPUs 64 GB 256 GB / 230 GB Large joins and memory-intensive aggregation
  • G.8X 8 DPUs 32 vCPUs 128 GB 512 GB / 485 GB Highest-memory general-purpose Spark worker
  • R.1X 1 DPU 4 vCPUs 32 GB 94 GB / 44 GB Memory-optimized Spark work with twice the G.1X memory per DPU

Exam Tip: Start from measured workload needs. Moving from G.1X to G.2X doubles memory per worker and increases local disk, which can relieve executor pressure, but it does not guarantee an OOM fix. Inspect driver versus executor failure, data skew, shuffle spill, partition sizing, and accidental collection before or alongside scaling.

Execution Classes & Infrastructure Optimizations

Standard Execution Class : Ideal for time-sensitive, production workloads that require fast job startup times and guaranteed compute availability.

Flex Execution Class : Utilizes non-critical, spare AWS compute capacity for non-urgent ETL jobs (such as overnight batch runs or testing). Flex uses a lower-priced execution class for eligible non-urgent jobs, though startup time can vary with available capacity; compare current Regional rates.

Glue Auto Scaling ( --enable-auto-scaling=true ) : Dynamically adds and removes workers based on real-time Spark stage demands. Unlike fixed-size Spark clusters, Glue Auto Scaling monitors executor metrics and releases idle workers during file listing or single-driver write phases, preventing DPU billing waste.

S3-optimized output : Current Glue runtimes enable the EMRFS S3-optimized committer for supported Parquet writes by default. Use supported runtime options, partition-aware writes, and compaction; do not invent a connection_options flag.

DynamicFrames vs. PySpark DataFrames

While native PySpark DataFrames require a rigid, predefined schema upfront, AWS Glue DynamicFrames maintain an explicit, self-describing dynamic schema that evaluates dataset structures on a row-by-row basis.

Structural & Operational Comparison

  • Feature / Metric DynamicFrame PySpark DataFrame
  • Schema Paradigm Dynamic (Self-describing row-by-row) Static (Predefined upfront before execution)

Handling Missing Data Retains original structures without null coercion Coerces missing fields into explicitly typed nulls

Schema Inconsistencies Wraps conflicting types in a Choice type container Throws runtime analysis exceptions or corrupts data

Transformation APIs Glue-native methods ( ApplyMapping , ResolveChoice ) Standard Spark SQL, PySpark functions, Window functions

Integration Target Optimized for AWS Glue Data Catalog & S3 sources Optimized for general Spark clusters and in-memory SQL

Key Scenarios for Using DynamicFrames vs PySpark DataFrames

Use DynamicFrames when : Ingesting raw JSON, semi-structured logs, or API payloads where fields appear conditionally, nested structures vary across records, or column data types mutate over time. They are convenient for Glue-native transforms and catalog-driven schema mapping, while bookmark support depends on source, API, and transformation context rather than a blanket DynamicFrame requirement.

Use PySpark DataFrames when : Performing complex analytical window functions ( ROW_NUMBER() OVER (...) ), multi-table relational JOIN operations, complex statistical UDFs, or interfacing with third-party PySpark libraries.

Converting Between DynamicFrames and DataFrames

Data engineers frequently convert between the two abstractions in a single Glue script to capitalize on the strengths of each API: # Convert Glue DynamicFrame to PySpark DataFrame

pyspark_df = dynamic_frame.toDF()

  • # Perform complex PySpark transformations (e.g., Window function)
  • from pyspark.sql.window import Window
  • from pyspark.sql.functions import row_number, col
  • windowSpec = Window.partitionBy("customer_id").orderBy(col("transaction_timestamp").desc())
  • transformed_df = pyspark_df.withColumn("rank", row_number().over(windowSpec)).filter("rank = 1")
  • # Convert back to DynamicFrame for optimized AWS Glue catalog writing
  • transformed_dynamic_frame = DynamicFrame.fromDF(transformed_df, glueContext, "transformed_dynamic_frame")
  • Native DynamicFrame Transformation Methods

AWS Glue provides specialized transformation methods tailored for semi-structured data manipulation:

1. ApplyMapping

Renames columns, alters data types, and drops unwanted fields in a single declarative step: mapped_dyf = ApplyMapping.apply(

  • frame=input_dyf,
  • mappings=[
  • ("cust_id", "string", "customer_id", "bigint"),
  • ("tx_amt", "double", "transaction_amount", "decimal(12,2)"),
  • ("raw_payload.event_time", "string", "event_timestamp", "timestamp")
  • ],
  • transformation_ctx="mapped_dyf"
  • )
  • 2. ResolveChoice

When a single column contains multiple conflicting data types across different records (e.g., user_id stored as int in early records and string in later records), Glue wraps the column in a Choice type. ResolveChoice resolves these conflicts using one of four strategies:

make_cols : Creates separate columns for each data type (e.g., user_id_int , user_id_string ).

cast:type : Casts all occurrences to a target type (e.g., cast:string ).

project:type : Retains only records matching a specific type and drops mismatched values.

make_struct : Combines conflicting types into a nested struct holding both values. resolved_dyf = ResolveChoice.apply(

  • frame=input_dyf,
  • choice="make_cols",
  • transformation_ctx="resolved_dyf"
  • )
  • 3. Relationalize

Flattens deeply nested JSON structures containing embedded arrays into a collection of relational DynamicFrames linked by primary and foreign key relationships ( root_id ), facilitating direct ingestion into relational data warehouses like Amazon Redshift. # Relationalize produces a DynamicFrameCollection

  • dfc = input_dyf.relationalize(
  • root_name="orders",
  • staging_path="s3://my-company-glue-staging/relationalize/"
  • )
  • # Extract root table and child array table
  • orders_root = dfc.select("orders")
  • orders_items = dfc.select("orders_items")
  • 4. Unnest

Unnests nested struct columns into top-level columns without splitting arrays into separate tables: unnested_dyf = input_dyf.unnest(transformation_ctx="unnested_dyf")

End-to-End AWS Glue PySpark Script Example

The following PySpark ETL script illustrates reading from the Glue Data Catalog, mapping schema fields, converting to a DataFrame for aggregation, converting back to a DynamicFrame, and writing optimized Parquet files to S3: import sys

  • from awsglue.transforms import *
  • from awsglue.utils import getResolvedOptions
  • from pyspark.context import SparkContext
  • from awsglue.context import GlueContext
  • from awsglue.job import Job
  • from awsglue.dynamicframe import DynamicFrame
  • from pyspark.sql.functions import col, sum as _sum
  • # Initialize Glue job arguments and context
  • args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  • sc = SparkContext()
  • glueContext = GlueContext(sc)
  • spark = glueContext.spark_session
  • job = Job(glueContext)
  • job.init(args['JOB_NAME'], args)
  • # 1. Read from Glue Data Catalog as DynamicFrame
  • raw_dyf = glueContext.create_dynamic_frame.from_catalog(
  • database="sales_db",
  • table_name="raw_transactions",
  • transformation_ctx="raw_dyf"
  • )
  • # 2. Apply Schema Mapping & Type Conversions
  • mapped_dyf = ApplyMapping.apply(
  • frame=raw_dyf,
  • mappings=[
  • ("store_id", "string", "store_id", "string"),
  • ("tx_date", "string", "transaction_date", "string"),
  • ("amount", "double", "amount", "double")
  • ],
  • transformation_ctx="mapped_dyf"
  • )
  • # 3. Convert to PySpark DF for Aggregation
  • df = mapped_dyf.toDF()
  • aggregated_df = df.groupBy("store_id", "transaction_date") \
  • .agg(_sum("amount").alias("total_daily_sales"))
  • # 4. Convert back to DynamicFrame
  • final_dyf = DynamicFrame.fromDF(aggregated_df, glueContext, "final_dyf")
  • # 5. Write to S3 in Partitioned Parquet format using S3 Direct Writer
  • glueContext.write_dynamic_frame.from_options(
  • frame=final_dyf,
  • connection_type="s3",
  • connection_options={
  • "path": "s3://company-analytics-lake/curated/daily_sales/",
  • "partitionKeys": ["transaction_date"]
  • },
  • format="parquet",
  • transformation_ctx="write_dyf"
  • )

job.commit()

Loading diagram... AWS Glue Job Data Processing Pipeline