Study guide

3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing

AWS Certified Data Engineer - Associate DEA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Streaming Data Ingestion

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State

2 Chapter 2: Batch Data Ingestion & Migration

2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion

3 Chapter 3: ETL Transformation with AWS Glue

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage

4 Chapter 4: Big Data Processing & Workflow Orchestration

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD

5 Chapter 5: Object Storage & Data Lake Architecture

5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations

6 Chapter 6: Data Warehousing with Amazon Redshift

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum

7 Chapter 7: NoSQL, Relational & Specialty Data Stores

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing

8 Chapter 8: Pipeline Operations, Monitoring & Observability

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail

9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning

9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew

10 Chapter 10: High Availability, Backup & Cost Optimization

10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads

11 Chapter 11: Data Security, Encryption & Identity Management

11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints

12 Chapter 12: Data Governance, Privacy & Compliance

12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance

3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing

Key Takeaways Explain This Topic AI

AWS Glue Job Bookmarks track state across job runs, processing only new or modified objects from S3 buckets or incremental primary keys from JDBC data stores.

Job Bookmarks require append-only or newly added files for S3 sources; updating existing S3 object contents without altering keys or timestamps bypasses state detection.

Pushdown predicates (push_down_predicate) filter S3 partitions at the Glue Data Catalog metadata level before Spark reads or lists files, reducing S3 API costs and job execution time.

The Small File Problem in Glue Spark jobs causes high S3 ListObjects/GET request latency and driver memory overhead, mitigated by groupFiles='inGroup' and groupSize settings.

Resetting Glue Job Bookmarks via CLI (aws glue reset-job-bookmark) enables full historical re-processing of source datasets when backfilling data.

Last updated: August 2026

3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing

Processing massive, continually growing enterprise datasets requires efficient incremental data processing patterns. Re-reading entire S3 data lakes or relational database tables during every batch execution wastes compute resources and increases pipeline latency.

AWS Glue provides two core mechanisms to achieve high-efficiency incremental processing: Glue Job Bookmarks for automated state tracking across job runs, and Pushdown Predicates for metadata-level partition pruning.

AWS Glue Job Bookmarks: State Management & Incremental Ingestion

AWS Glue Job Bookmarks track state information across consecutive runs of an ETL job. By persisting bookmark state, Glue ensures that subsequent job runs process only data ingested since the previous execution.

How Job Bookmarks Handle S3 vs. JDBC Data Sources

Amazon S3 Sources : Glue tracks processed S3 object keys, file sizes, and last modified timestamps. When a job runs, Glue compares S3 bucket contents against the stored bookmark state, passing only newly added or modified files to the job.

Crucial Limitation: S3 Bookmarks rely on append-only patterns or new file additions. If an existing S3 object's content is modified in-place without changing its object key or file modification metadata in a recognizable manner, the bookmark may not re-process the file.

JDBC Data Sources : For relational databases, Glue tracks a sequential column (such as an auto-incrementing primary key ID or a last_updated timestamp). Data engineers specify the bookmark key in connection options ( jobBookmarkKeys=["updated_at"] , jobBookmarkKeysSortOrder="asc" ).

Job Bookmark Configuration Options ( --job-bookmark-option )

Configuration Option Operational Behavior Real-World & Exam Scenario

job-bookmark-enable Maintains and updates state across job runs; processes only new/incremental data. Default for production pipelines ingesting incremental batch logs or CDC output.

job-bookmark-disable Ignores existing state and does not update state upon completion; processes all data in source. Used during testing or when full historical re-processing is intentionally required.

job-bookmark-pause Reads data starting from the last committed state, but does not commit new state upon completion. Useful for debugging failed runs without advancing state markers.

Resetting & Rewinding Bookmarks for Data Backfills

If a pipeline encounters a downstream corruption event and historical data must be reprocessed, data engineers can rewind or reset the job bookmark state using the AWS CLI: # Reset Glue Job Bookmark to reprocess all historical data from inception

aws glue reset-job-bookmark --job-name daily-sales-etl-job

Exam Tip: To ensure Job Bookmarks function correctly in custom PySpark scripts, you must pass the transformation_ctx parameter to create_dynamic_frame.from_catalog or create_dynamic_frame.from_options . If transformation_ctx is omitted, Glue cannot attach state markers to the reader source, causing the job to process the entire dataset every run.

Partition Processing & Pushdown Predicates

Partitioning S3 data lakes into hierarchical directory prefixes (e.g., s3://bucket/table/year=2026/month=08/day=13/ ) dramatically improves query efficiency. However, how Spark reads these partitions impacts cost and performance.

Pushdown Predicates vs. In-Memory Spark Filtering

In-Memory Spark Filtering ( df.filter(col("year") == 2026) ) : Spark lists all S3 objects across all partition directories, reads every file into executor memory, and then discards non-matching rows. This results in millions of expensive S3 ListObjectsV2 and GET API calls and high DPU utilization.

Catalog Pushdown Predicates ( push_down_predicate ) : Glue evaluates the predicate condition directly against the Glue Data Catalog metadata before Spark reads or lists files from S3 storage. Spark receives only the S3 object keys matching the target partition, bypassing non-matching S3 directories entirely.

Performance & Cost Comparison Matrix

  • Execution Aspect In-Memory Spark Filter Glue Pushdown Predicate ( push_down_predicate )
  • Evaluation Site Spark Executor Compute Memory AWS Glue Data Catalog Metadata Level
  • S3 API Overhead High (Lists and reads all objects in bucket) Minimal (Lists ONLY matched partition prefixes)
  • Data Transfer & I/O High (Transfers full dataset over network) Low (Transfers only filtered partition files)

DPU Execution Time Slow (Proportional to total historical size) Fast (Proportional ONLY to targeted partition size)

  • Pushdown Predicate Implementation Example # Pushdown predicate evaluated at Catalog metadata level
  • partitioned_dyf = glueContext.create_dynamic_frame.from_catalog(
  • database="analytics_db",
  • table_name="web_logs",
  • push_down_predicate="year == '2026' and month == '08' and day == '13'",
  • transformation_ctx="partitioned_dyf"
  • )
  • Solving the Small File Problem in AWS Glue

High-frequency ingestion pipelines (such as streaming Kinesis Firehose delivery or hourly micro-batches) often produce thousands of small files (e.g., < 5 MB) in S3. In Apache Spark, each small file creates an individual task partition, leading to severe performance bottlenecks:

Excessive S3 ListObjectsV2 and GET request latency.

Spark Driver out-of-memory errors caused by managing millions of file metadata objects.

High CPU overhead spent managing task lifecycle overhead rather than data processing.

Grouping Small Files in DynamicFrames

AWS Glue provides built-in file grouping parameters ( groupFiles and groupSize ) within DynamicFrame reader options. Grouping automatically coalesces small S3 files into larger, memory-efficient in-memory partitions before Spark task execution: # Group small S3 files into target 128 MB partitions

  • grouped_dyf = glueContext.create_dynamic_frame.from_options(
  • connection_type="s3",
  • connection_options={
  • "paths": ["s3://my-lake-bucket/raw_logs/"],
  • "groupFiles": "inGroup",
  • "groupSize": "134217728" # 128 MB in bytes (128 * 1024 * 1024)
  • },
  • format="json",
  • transformation_ctx="grouped_dyf"
  • )
  • Output File Compaction Strategy

When writing processed data back to S3, use coalesce(N) or repartition(N) on PySpark DataFrames to ensure output Parquet files are sized optimally (typically 128 MB to 512 MB per file ): # Coalesce output partition files to avoid small file proliferation in S3

  • compacted_df = transformed_df.coalesce(4)
  • compacted_dyf = DynamicFrame.fromDF(compacted_df, glueContext, "compacted_dyf")
  • End-to-End Incremental Glue PySpark Script

The following complete Glue script combines Job Bookmarks, Pushdown Predicates, Small File Grouping, and Partitioned S3 Writing: import sys

  • from awsglue.utils import getResolvedOptions
  • from pyspark.context import SparkContext
  • from awsglue.context import GlueContext
  • from awsglue.dynamicframe import DynamicFrame
  • from awsglue.job import Job
  • # 1. Parse Job Arguments
  • args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  • sc = SparkContext()
  • glueContext = GlueContext(sc)
  • job = Job(glueContext)
  • job.init(args['JOB_NAME'], args)
  • # 2. Ingest Incremental Data with Pushdown Predicate & File Grouping
  • incremental_dyf = glueContext.create_dynamic_frame.from_catalog(
  • database="telemetry_db",
  • table_name="device_events",
  • push_down_predicate="year == '2026' and month == '08'",
  • additional_options={
  • "groupFiles": "inGroup",
  • "groupSize": "134217728" # 128 MB grouping
  • },

transformation_ctx="incremental_dyf" # Required for Job Bookmark tracking!

)

  • # 3. Simple Transformation
  • transformed_df = incremental_dyf.toDF().dropDuplicates(["event_id"])
  • output_dyf = DynamicFrame.fromDF(transformed_df.coalesce(10), glueContext, "output_dyf")
  • # 4. Write Incremental Partitioned Parquet Data
  • glueContext.write_dynamic_frame.from_options(
  • frame=output_dyf,
  • connection_type="s3",
  • connection_options={
  • "path": "s3://company-analytics-lake/curated/device_events/",
  • "partitionKeys": ["year", "month"]
  • },
  • format="parquet",
  • transformation_ctx="write_output"
  • )
  • # 5. Commit Job Bookmark State
  • job.commit()

Loading diagram... Incremental Glue ETL Architecture with Bookmarks & Pushdown