Study guide
3.5 Technical and Business Catalogs, Connections & Lineage
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
3.5 Technical and Business Catalogs, Connections & Lineage
Key Takeaways Explain This Topic AI
The AWS Glue Data Catalog is a technical metastore for databases, tables, schemas, partitions, and connections; crawlers infer metadata but do not prove business meaning or quality.
Amazon SageMaker Catalog adds searchable business names, glossary terms, ownership, metadata forms, publication, and subscription workflows over technical assets.
Partition synchronization can use crawlers, ETL catalog updates, APIs, or Athena partition projection; choose based on scale, latency, and whether partitions are enumerable.
Lineage connects datasets, jobs, and job runs so engineers can perform root-cause and impact analysis; current SageMaker lineage is OpenLineage-compatible and can capture Glue and EMR Spark flows.
Last updated: August 2026
3.5 Technical and Business Catalogs, Connections & Lineage
A catalog answers two different questions: How can an engine read this data? and What does this data mean, who owns it, and may I use it? The AWS Glue Data Catalog primarily answers the technical question. Amazon SageMaker Catalog in SageMaker Unified Studio adds business discovery and governed collaboration.
AWS Glue Data Catalog as technical metadata
The Glue Data Catalog stores catalogs, databases, tables, columns, partition keys, table properties, and connections. Athena, Redshift Spectrum, EMR, Glue, and compatible engines can use it as a metastore. A table definition points to data; the catalog does not contain the S3 dataset itself.
A crawler connects to a source, applies custom classifiers before built-in classifiers, infers schemas, and creates or updates tables and partitions. Treat inference cautiously. Mixed files can cause columns to become strings or separate tables, and a crawler cannot decide whether a customer identifier is legally sensitive or whether a metric is approved.
Connections store reusable source properties and, for VPC sources, network configuration. The job still needs routes, security groups, DNS, credentials, and data permissions. Test a connection from the environment that will use it rather than assuming a successful catalog entry proves reachability.
Synchronizing partitions
Choose one method deliberately:
- Method Strong use Tradeoff
- Crawler Discover unknown layouts and schema changes Scans cost time and can infer unwanted changes
- ETL catalog update or Glue API Writer knows the exact new partition Application owns metadata correctness
- MSCK REPAIR TABLE Discover Hive-style S3 partitions Slow at very large partition counts
Athena partition projection Predictable date or integer partitions Calculated partitions are not stored in the catalog
Do not confuse an S3 prefix with a registered partition. A query engine prunes catalog partitions only when the table metadata and query predicate align. For high-cardinality layouts, partition projection can avoid millions of Glue API lookups.
Schema evolution and conversion
Adding nullable columns is usually safer than renaming or changing types. Define producer compatibility rules, preserve the raw version, and test old and new readers. Glue table updates describe a schema; they do not rewrite existing files.
For heterogeneous relational migrations, current AWS guidance recommends DMS Schema Conversion to assess and convert supported schema and code objects. DMS data movement and schema conversion are separate concerns, and unsupported code still needs manual remediation.
Amazon SageMaker Catalog
A technical name such as daily_sales_agg_v3 is difficult for another team to trust. In SageMaker Unified Studio, a project can inventory an asset, add a business name, description, owner, README, glossary terms, and structured metadata forms, then publish the latest asset version to the SageMaker Catalog. Publication makes the asset discoverable beyond the project; updating the inventory later does not automatically publish the new version.
Consumers search the catalog and request a subscription . When approved, access is granted to the requesting project's execution role, commonly through Lake Formation for governed tables. Approval is not merely a UI label—it must produce data-plane access for the execution identity. Revocation must remove that grant.
Technical and business catalogs complement each other:
Glue supplies schema, location, partitions, and engine integration.
SageMaker Catalog supplies shared language, curation, ownership, discovery, and subscription workflow.
Lake Formation enforces governed table access.
Data lineage
Lineage should connect a source dataset to a job, a job run, and its output dataset. It supports two investigations:
Root cause: Which upstream input or job produced a bad column?
Impact: Which downstream tables, dashboards, or models depend on the field I will change?
SageMaker Unified Studio lineage is compatible with OpenLineage events and can visualize datasets, jobs, and job runs. Current integrations can capture lineage from Glue and EMR Spark jobs, with column-level detail when supported. The graph is evidence only if run identities, source versions, and output versions are accurate.
Do not equate CloudTrail with data lineage. CloudTrail records API activity; it can show that a job or catalog API was called, but it does not by itself express the semantic mapping from source column to transformed output column.
Governance checklist
For each production asset, record a stable identifier, technical location, schema, owner, business definition, sensitivity, freshness expectation, quality status, lineage, and access policy. Separate draft inventory from published approved assets, and automate checks for missing owners or stale metadata. A catalog with thousands of uncurated tables is an inventory, not a trusted data product marketplace.