Study guide
3.2 Glue Crawlers, Data Catalog & Schema Evolution
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
3.2 Glue Crawlers, Data Catalog & Schema Evolution
Key Takeaways Explain This Topic AI
The AWS Glue Data Catalog serves as a centralized, Hive-metastore-compatible metadata repository used by Amazon Athena, Amazon EMR, Amazon Redshift Spectrum, and AWS Lake Formation.
AWS Glue Crawlers inspect data stores (S3, JDBC, DynamoDB) using ordered classifiers to infer schemas, file formats, compression, and S3 partition structures automatically.
Classifiers execute in a strict priority order: custom Grok/JSON/XML classifiers run first, followed by built-in classifiers (Avro, Parquet, ORC, JSON, CSV).
Crawler configuration controls schema evolution by determining whether catalog table definitions are updated, ignored, or deprecated when source schema drift occurs.
Updating the Glue Data Catalog dynamically directly from Glue ETL jobs (enableUpdateCatalog=True) provides zero-latency metadata updates without requiring asynchronous crawler runs.
Last updated: August 2026
3.2 Glue Crawlers, Data Catalog & Schema Evolution
In modern data lakes built on Amazon S3, metadata management is the bridge connecting raw file storage to analytical query engines. The AWS Glue Data Catalog provides a centralized, Apache Hive-metastore-compatible repository that stores table definitions, column types, physical data locations, and partition key structures.
Query engines like Amazon Athena , Amazon Redshift Spectrum , Amazon EMR , and AWS Lake Formation rely on the Glue Data Catalog as their single source of truth for table schemas and security policies.
AWS Glue Crawlers Architecture & Classifier Precedence
AWS Glue Crawlers are automated processes that connect to source data stores, evaluate data samples using classifiers, infer format and schema attributes, and populate or update table definitions in the Glue Data Catalog.
Classifier Evaluation Order
When a crawler scans a data store, it tests the data against classifiers in a strict, deterministic sequence:
Custom Classifiers (User-Defined) : Evaluated first in the order specified in the crawler configuration. Custom classifiers include:
Grok Classifiers : Use Grok patterns to parse unstructured or semi-structured log files (e.g., Apache access logs, syslog).
JSON Path Classifiers : Define explicit JSONPath expressions to parse complex JSON documents.
XML Classifiers : Define row tags to parse XML schemas.
Custom CSV Classifiers : Specify custom column headers, delimiters, and quote characters.
Built-In Classifiers : Evaluated only if no custom classifier matches. Built-in classifiers automatically recognize standard formats including Apache Parquet , Apache Avro , ORC , JSON , BSON , XML , CSV , and compressed variants (Gzip, Snappy, Bzip2).
Important Exam Concept: If a custom classifier and a built-in classifier both match a dataset, the custom classifier takes precedence . If you create a custom Grok classifier for JSON files, Glue will use your Grok classifier instead of the built-in JSON classifier.
Configuring Crawlers for Schema Evolution & Object Deletion
As upstream data sources evolve, schemas inevitably change—columns are added, renamed, or dropped, and historical data partitions accumulate. AWS Glue Crawlers provide fine-grained controls to manage schema drift and data lifecycle events.
Schema Change Behavior Matrix
When a crawler detects a mismatch between the physical storage schema and the Data Catalog table definition, its behavior is governed by the Schema Change Policy :
Configuration Option Behavior When New Columns / Fields Are Detected Real-World & Exam Scenario
Update the table definition in the data catalog (Default) Adds newly discovered columns and updates data types in the Catalog table definition. Best for append-only data lakes where upstream producers regularly add new attributes.
Ignore the change and don't update the data catalog Retains existing catalog schema; new physical columns are ignored by query engines. Used when downstream analytics require strict schema lock and schema changes must be manually vetted.
Log the change Writes schema change details to CloudWatch Logs without modifying the Catalog table. Useful for auditing schema drift in strict governance environments.
Object Deletion & Deprecation Policy
When files or partitions are deleted from S3, crawler behavior depends on the Object Deletion Policy :
Mark the table as deprecated in the data catalog : Adds a property DEPRECATED to the catalog table. Recommended when historical metadata must be preserved for auditing.
Delete tables and partitions from the data catalog : Removes metadata entries for missing files/partitions immediately. Essential for maintaining clean catalogs when data retention policies delete old S3 prefixes.
Ignore the change : Leaves catalog definitions untouched even if underlying S3 objects no longer exist.
Catalog Partition Syncing Strategies: Crawlers vs ETL Direct Updates
In S3 data lakes, data is structured into Hive-style partition paths (e.g., s3://bucket/table/year=2026/month=08/day=13/ ). For query engines like Athena to leverage partition pruning, partition metadata must be registered in the Data Catalog.
Data engineers have three primary mechanisms to synchronize partition metadata:
Comparison of Partition Syncing Methods
Sync Mechanism Execution Mechanism Latency Overhead & Cost Best Use Case
Glue Crawler Asynchronous batch job (Scheduled / EventBridge) High (Runs periodically after batch completion) Medium (Incurs DPU crawler execution charges) Ingesting third-party or multi-source data lakes with unknown partition structures.
MSCK REPAIR TABLE Executed in Athena or Hive ( MSCK REPAIR TABLE table_name ) Manual / Triggered High (Scans full S3 file path tree; slow on large buckets) Ad-hoc repair of partition metadata after manual S3 data uploads.
ETL Direct Catalog Sync ( enableUpdateCatalog ) Inline within Glue ETL job ( write_dynamic_frame ) Zero (Real-time update during job write) Low (Zero additional compute cost; updates catalog directly) Recommended for production Glue ETL pipelines writing structured data to S3.
Direct Catalog Update Code Pattern
Instead of running a separate Glue Crawler after an ETL job completes, configure the Glue ETL job to update the catalog schema and partitions dynamically during execution: # Direct Catalog Update within Glue Job Write Option
- glueContext.write_dynamic_frame.from_catalog(
- frame=transformed_dyf,
- database="analytics_db",
- table_name="processed_events",
- additional_options={
- "enableUpdateCatalog": True,
- "updateBehavior": "UPDATE_IN_DATABASE",
- "partitionKeys": ["year", "month", "day"]
- },
- transformation_ctx="write_catalog_dyf"
- )
- Handling Schema Evolution in Amazon Athena Queries
When schema changes occur in S3 data lakes, downstream query engines like Amazon Athena handle evolution based on the underlying file format:
Parquet & ORC (Schema by Name vs. Schema by Read Index) :
Athena reads Parquet columns by name by default ( parquet.column.index.access=false ). Name mapping tolerates column reordering and adding or removing columns more safely than ordinal mapping. Set the property to true only when the files and table intentionally rely on column position.
Athena reads ORC columns by index by default ( orc.column.index.access=true ); set it to false when the dataset should map ORC columns by name.
JSON & CSV :
Schema evolution is handled by appending new columns to the end of the table schema in the Data Catalog. Missing fields in historical files automatically evaluate to NULL during Athena queries.
Boto3 Glue Crawler Automation Script
Data engineers automate crawler execution and status monitoring using the AWS SDK for Python ( boto3 ): import boto3
import time
glue_client = boto3.client('glue', region_name='us-east-1')
def trigger_and_wait_for_crawler(crawler_name):
- # Start Crawler
- response = glue_client.start_crawler(Name=crawler_name)
- print(f"Started crawler: {crawler_name}")
# Poll Crawler Status
while True:
- crawler = glue_client.get_crawler(Name=crawler_name)['Crawler']
- state = crawler['State']
- print(f"Crawler state: {state}")
if state == 'READY':
- last_crawl = crawler.get('LastCrawl', {})
- status = last_crawl.get('Status')
- print(f"Crawl completed with status: {status}")
- break
elif state == 'RUNNING' or state == 'STOPPING':
time.sleep(15)
else:
raise Exception(f"Unexpected crawler state: {state}")
- # Example invocation
- # trigger_and_wait_for_crawler("s3-raw-sales-crawler")