2.3 Data Cataloging, Governance, and Querying

AWS Certified Machine Learning Engineer — Associate MLA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Introduction & Exam Strategy

1.1 AWS MLA-C01 Blueprint, Format & Strategy 1.2 The Machine Learning Engineering Lifecycle on AWS

2 Domain 1: Data Ingestion & Storage Architecture for ML

2.1 Ingesting Batch and Streaming Data for ML 2.2 High-Performance Storage for Training and Inference 2.3 Data Cataloging, Governance, and Querying

3 Domain 1: Feature Engineering & Data Transformation

3.1 Distributed Feature Engineering with AWS Glue & DataBrew 3.2 Interactive Data Preparation with SageMaker Data Wrangler 3.3 Feature Management with SageMaker Feature Store

4 Domain 1: Data Quality, Integrity & Ground Truth Labeling

4.1 Data Validation, Schema Drift, and Quality Rules 4.2 Handling Missing Values, Outliers & Class Imbalance 4.3 Data Labeling with SageMaker Ground Truth & Ground Truth Plus

5 Domain 2: Model Selection & Framework Architecture

5.1 SageMaker Built-in Algorithms & Selection Criteria 5.2 Custom Models with Script Mode and BYOC 5.3 Foundation Models, JumpStart, and Amazon Bedrock

6 Domain 2: SageMaker Model Training & Hyperparameter Optimization

6.1 SageMaker Training Jobs & Cost Optimization 6.2 Distributed Training Strategies & Architectures 6.3 Hyperparameter Optimization with Automatic Model Tuning (AMT)

7 Domain 2: Model Evaluation, Tracking & Model Registry

7.1 ML Evaluation Metrics & Validation Strategies 7.2 Experiment Tracking with SageMaker Experiments 7.3 Model Governance with SageMaker Model Registry

8 Domain 3: SageMaker Inference Options & Endpoint Architecture

8.1 Real-Time, Serverless, and Asynchronous Inference 8.2 High-Throughput Offline Inference with Batch Transform 8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)

9 Domain 3: Deployment Strategies & Infrastructure Scaling

9.1 Production Deployment Patterns & Traffic Shifting 9.2 Endpoint Auto-Scaling & Hardware Acceleration

10 Domain 3: MLOps Pipelines & Workflow Orchestration

10.1 SageMaker Pipelines Core Architecture 10.2 Advanced Pipeline Patterns & Integration 10.3 End-to-End MLOps CI/CD & Automated Retraining

11 Domain 4: Production Model Monitoring with SageMaker Model Monitor

11.1 Data Quality & Model Quality Monitoring 11.2 Model Bias & Model Explainability Drift 11.3 Operational Observability with Amazon CloudWatch

12 Domain 4: Responsible AI, Bias Detection & Governance

12.1 Pre-Training and Post-Training Bias with SageMaker Clarify 12.2 Model Documentation & Auditing with Model Cards & Dashboard 12.3 Safety, Guardrails & Generative AI Governance

13 Domain 4: ML Security, Network Isolation & Cost Optimization

13.1 IAM Roles and Least Privilege for ML 13.2 Network Isolation, VPC Endpoints & Data Encryption 13.3 ML Cost Optimization & Resource Governance

2.3 Data Cataloging, Governance, and Querying

Key Takeaways Explain This Topic AI

The AWS Glue Data Catalog provides a centralized, Hive-compatible metadata repository that stores schemas, table definitions, and partition keys for data lakes across AWS analytical and ML services.

AWS Glue Crawlers automatically inspect S3 data, infer schemas and file formats, detect partition directory hierarchies, and update Data Catalog tables according to configured schema drift handling policies.

AWS Lake Formation provides centralized, granular access control including Tag-Based Access Control (LF-TBAC), row-level filtering, column-level masking, and cell-level security without duplicating physical data files in S3.

Amazon Athena provides serverless, interactive SQL data exploration and transformation directly on S3 data lakes, utilizing Partition Projection to query high-cardinality datasets without hitting Glue Data Catalog API limits.

Athena CREATE TABLE AS SELECT (CTAS) statements allow ML engineers to convert, filter, and partition raw tabular datasets into optimized Apache Parquet files directly in S3 for high-speed SageMaker training consumption.

Last updated: August 2026

Data Cataloging, Governance, and Querying

Before data can be fed into feature engineering pipelines or SageMaker model training jobs, it must be discovered, cataloged, governed, and explored. In enterprise environments, data lakes contain sensitive information (Personally Identifiable Information [PII], financial records) distributed across hundreds of tables. ML engineers must implement robust governance frameworks to control data access at granular levels, track schema evolution, and query raw data in-place without copying or moving petabytes of files.

1. AWS Glue Data Catalog: Centralized ML Metadata Management

The AWS Glue Data Catalog is a managed, Apache Hive Metastore-compatible metadata repository. It stores structural metadata—such as database names, table definitions, column data types, physical S3 URI locations, and partition key structures—without storing the underlying data records. +-----------------------------------------------------------------------------+

• | AWS GLUE DATA CATALOG INTEGRATION |
• | |
• | [Underlying Storage] |
• | └── Amazon S3 Data Lake (Parquet / ORC / JSON / CSV) |
• | ^ |
• | | Metadata Reference (Schemas, Partitions, SerDes) |
• | v |
• | [AWS GLUE DATA CATALOG] <--- (Maintained by Crawlers / ETL / APIs) |
• | | |
• | +---> Amazon Athena (Ad-hoc SQL Exploration & CTAS) |
• | +---> Amazon SageMaker (Data Wrangler / Feature Store / Studio) |
• | +---> Amazon EMR & AWS Glue ETL (Distributed Spark Jobs) |
• | +---> Amazon Redshift Spectrum (Data Warehouse Queries) |
• +-----------------------------------------------------------------------------+

Key Concepts of the Glue Data Catalog:

Databases & Tables: A database is a logical grouping of table definitions. A table contains schema definitions (column names, data types such as int , bigint , string , struct , array ), the S3 physical path, and the Serializer/Deserializer ( SerDe ) used to parse the file format.

Partitions: Each partition maps a specific partition key value (e.g., year=2026/month=08/ ) to an S3 folder location. When query engines query a partitioned Glue table, they use partition metadata to skip non-matching S3 prefixes.

Schema Versioning: The Glue Data Catalog tracks schema evolution across time. When new columns are added or data types change, a new schema version is created, allowing historical rollback and backward compatibility validation.

2. AWS Glue Crawlers and Schema Drift Management

AWS Glue Crawlers connect to source data stores (Amazon S3, Amazon RDS, DynamoDB), determine the data format using built-in or custom classifiers, infer schemas, and automatically register or update table definitions in the Glue Data Catalog. +-----------------------------------------------------------------------------+

• | GLUE CRAWLER WORKFLOW & LOGIC |
• | |
• | [Amazon S3 Buckets] ---> [AWS Glue Crawler] |
• | | |
• | +---> 1. Run Classifiers (Grok/JSON/Parquet|
• | +---> 2. Detect Partition Keys (S3 Paths) |
• | +---> 3. Compare with Catalog Schema |
• | | |
• | v |
• | [Schema Drift Action] -------------------------------------------------+ |
• | ├── "Update the table definition in the data catalog" (Default) | |
• | ├── "Add new columns only" | |
• | └── "Ignore the change and don't update the table" | |
• +-----------------------------------------------------------------------------+

Classifier Hierarchy:

Built-in Classifiers: Glue includes native classifiers for common formats: JSON, CSV, TSV, Parquet, ORC, Avro, XML, and web server log formats.

Custom Classifiers: When data arrives in proprietary delimited formats or custom log structures, you define a custom Grok pattern , JSONPath , or XML path classifier. Custom classifiers are evaluated first in priority order before built-in classifiers.

Managing Schema Drift & Deleted Objects:

When upstream producer systems change data schemas (e.g., adding a device_battery_level column or removing deprecated fields), the Glue Crawler behavior is governed by configuration settings:

When schema changes are detected:

Update the table definition in the data catalog: Automatically adds new columns or changes data types.

Add new columns only: Protects existing column definitions from accidental modification while adopting new features.

Ignore the change: Leaves the table definition intact.

How crawler should handle deleted objects in the data store:

Mark the table as deprecated in the data catalog:

Delete tables and partitions from the data catalog:

Ignore the change (Keep existing metadata):

Tip

Event-Driven Crawling: Rather than running expensive scheduled crawlers that scan entire S3 buckets on a cron schedule, configure Amazon S3 Event Notifications with Amazon EventBridge to invoke an AWS Lambda function that triggers the Glue Crawler only when new partitions or files land in S3 ( s3:ObjectCreated:* ).

3. Fine-Grained Data Governance with AWS Lake Formation

Securing machine learning datasets using standard IAM policies and S3 bucket policies becomes unmanageable as organizations scale to hundreds of datasets and thousands of columns. AWS Lake Formation sits on top of the AWS Glue Data Catalog and provides centralized, fine-grained access control. +-----------------------------------------------------------------------------+

• | AWS LAKE FORMATION FINE-GRAINED ACCESS CONTROL |
• | |
• | [Raw S3 Table: `customer_analytics` (Contains PII, Financials, Targets)] |
• | | |
• | v |
• | [AWS LAKE FORMATION PERMISSION ENGINE] |
• | ├── Tag-Based Access Control (LF-TBAC): `Confidentiality=Restricted` |
• | ├── Column-Level Security: Mask `ssn` and `credit_card` columns |
• | └── Row-Level Security: Filter `WHERE country = 'US' AND active = true` |
• | | |
• | +------------------------+------------------------+ |
• | | | |
• | v v |
• | [Role A: ML Feature Engineer] [Role B: Data Auditor] |
• | - Sees all sanitized features - Sees metadata only |
• | - PII columns masked with hashes - No raw data access |
• | - Filtered to authorized geographic rows - Full audit log views |
• +-----------------------------------------------------------------------------+

Lake Formation Security Capabilities:

Lake Formation Tag-Based Access Control (LF-TBAC):

Defines access permissions based on metadata tags assigned to databases, tables, or columns (e.g., Department=Marketing , DataClassification=Public , PII=True ).

When new tables or columns are added with matching tags, IAM roles automatically inherit appropriate access permissions without manually updating individual IAM or S3 bucket policies.

Column-Level Access Control & Data Masking:

Restricts specific IAM roles or ML user identities from accessing sensitive feature columns (e.g., social security numbers, bank account numbers, patient medical identifiers).

Supports Dynamic Data Masking :

Redact (Nullify): Replaces sensitive column values with nulls.

Custom Masking / Hashing: Masks strings with asterisks or cryptographic hashes.

Row-Level Filtering:

Restricts data access based on SQL WHERE filter clauses applied at query time.

Example: An ML training role for a European model can be granted access with a row filter WHERE region = 'EU' , ensuring compliance with data residency regulations without physically duplicating data into separate S3 regional buckets.

Cell-Level Security:

Combines row-level filtering and column-level restrictions simultaneously to grant access to an exact rectangular subsection of a table.

Cross-Account Data Sharing:

Shares Data Catalog tables and underlying S3 data directly with external AWS accounts using AWS Resource Access Manager (RAM) and Lake Formation cross-account permissions without copying data.

4. Ad-Hoc Data Exploration & Preparation with Amazon Athena

Amazon Athena is an interactive, serverless query service that allows ML engineers to analyze data directly in Amazon S3 using standard SQL (Presto/Trino engines). It requires zero infrastructure setup and charges only for the data scanned by queries. +-----------------------------------------------------------------------------+

• | AMAZON ATHENA IN ML DATA PREPARATION |
• | |
• | 1. Exploratory Data Analysis (EDA): |
• | [Athena SQL Query] ---> Scans S3 In-Place ---> Returns Distribution Stats |
• | |
• | 2. Partition Projection (High-Cardinality Optimization): |
• | [Athena Query with Projection] ---> Skips Glue Catalog Partition Lookups |
• | ---> Computes S3 paths dynamically in SQL |
• | |
• | 3. Feature Extraction & Dataset Splitting via CTAS: |
• | [Athena CTAS Query] ---> Reads Raw JSON/CSV |
• | ---> Transforms & Filters |
• | ---> Writes Snappy Parquet to S3 Training Folder |
• +-----------------------------------------------------------------------------+
• Partition Projection for High-Cardinality Datasets

The Challenge: Datasets with millions of partitions (e.g., IoT telemetry partitioned by device_id , year , month , day , hour ) cause severe performance degradation. Every query forces Athena to make thousands of GetPartitions API calls to the Glue Data Catalog, resulting in query latency and AWS Glue rate-limiting errors.

The Solution (Partition Projection):

Partition values and S3 directory locations are calculated directly from predefined table configuration properties (e.g., date ranges, numeric ranges, enums).

Athena computes the exact S3 prefix paths in memory without making metadata API calls to the Glue Data Catalog.

Dramatically accelerates query execution times and eliminates partition registration overhead. -- Example: Athena Table Definition with Partition Projection

• CREATE EXTERNAL TABLE telemetry_projected (
• sensor_id STRING,
• temperature DOUBLE,
• vibration DOUBLE,
• pressure DOUBLE
• )
• PARTITIONED BY (
• device_type STRING,
• log_date STRING
• )
• STORED AS PARQUET
• LOCATION 's3://iot-sensor-lake/telemetry/'
• TBLPROPERTIES (
• 'projection.enabled' = 'true',
• 'projection.device_type.type' = 'enum',
• 'projection.device_type.values' = 'turbine,pump,compressor',
• 'projection.log_date.type' = 'date',
• 'projection.log_date.range' = '2026/01/01,NOW',
• 'projection.log_date.format' = 'yyyy/MM/dd',
• 'storage.location.template' = 's3://iot-sensor-lake/telemetry/device_type=${device_type}/date=${log_date}'
• );
• CTAS (CREATE TABLE AS SELECT) for ML Training Preparation

ML engineers use Athena CTAS queries to extract, clean, encode, and export raw training datasets into partitioned, compressed Apache Parquet formats ready for SageMaker training consumption: -- Create optimized training split in S3 directly from Athena

• CREATE TABLE sagemaker_churn_train
• WITH (
• format = 'PARQUET',
• parquet_compression = 'SNAPPY',
• external_location = 's3://ml-feature-store-prod/training_splits/train/',
• partitioned_by = ARRAY['churn_label']
• ) AS
• SELECT
• account_id,
• tenure_months,
• total_spend,
• support_tickets,
• avg_session_length,
• CASE WHEN churned = 1 THEN 'CHURN' ELSE 'RETAIN' END AS churn_label
• FROM raw_customer_events
• WHERE event_date >= '2026-01-01'
• AND total_spend IS NOT NULL;
• Athena Workgroups for Cost Governance

Athena Workgroups allow ML teams to isolate query execution environments, configure separate Amazon CloudWatch query metrics, mandate query result encryption, and set Per-Query and Per-Workgroup Data Scan Limits to prevent runaway costs from unpartitioned table scans.

Important

Core Governance Exam Takeaway: When an enterprise ML scenario requires granting data scientists access to specific columns and authorized customer records while masking PII (like SSNs) without duplicating or redacting underlying S3 files, the correct AWS solution is AWS Lake Formation with column-level masking and row-level filtering .