Study guide
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation
Key Takeaways Explain This Topic AI
AWS Lake Formation provides centralized, fine-grained access control (FGAC) down to column, row, and cell levels across AWS Glue Data Catalog tables without modifying underlying Amazon S3 object files.
Tag-Based Access Control (LF-TBAC) simplifies security management at scale by attaching LF-tags to databases, tables, and columns, enabling role-based access without explicit resource ARN grants.
Lake Formation credential vending issues temporary, restricted IAM credentials to query engines (Amazon Athena, Amazon EMR, Amazon Redshift Spectrum), eliminating direct S3 bucket permissions for end users.
Cross-account dataset sharing is achieved natively via AWS Resource Access Manager (RAM) integrations, allowing external accounts to query shared resources via Resource Links without copying data.
Data Cell Filters unify column projection and row-level SQL predicate expressions into a single security policy object for precise cell-level security.
Last updated: August 2026
Centralized Data Lake Governance & Access Control with AWS Lake Formation
Traditional data lake security models rely on a combination of Amazon S3 bucket policies , IAM policies , and KMS key permissions . As data lakes scale to thousands of tables and millions of objects, managing prefix-level access across multiple analytics tools becomes operationally unmaintainable. IAM policies hit maximum size limits, and bucket policies lack the ability to enforce column- or row-level permissions natively without duplicating datasets.
AWS Lake Formation solves these governance challenges by providing a centralized authorization layer on top of the AWS Glue Data Catalog and Amazon S3. Instead of defining broad permissions at the S3 object prefix level, Lake Formation enforces fine-grained access control (FGAC) at the database, table, column, row, and cell levels.
Lake Formation Architecture & Registration
To manage an Amazon S3 path with AWS Lake Formation, the location must first be registered with Lake Formation. When an S3 path is registered, Lake Formation registers the underlying IAM role ( AWSServiceRoleForLakeFormationDataAccess ) that possesses read/write privileges to that S3 location.
Data Location Registration & Initial Policy Migration
Register Data Location : Administrators register S3 storage locations (e.g., s3://my-company-datalake/analytics/ ) with Lake Formation via RegisterDataLocation .
Revoke Default Permissions : By default, new AWS Glue catalog objects grant ALL permissions to IAMAllowedPrincipals . To transition to Lake Formation governance, these default grants must be revoked so that IAM policies no longer bypass Lake Formation evaluations.
Data Location Permissions : Principals must be granted DATA_LOCATION_ACCESS permissions on the registered S3 path to create databases or tables pointing to those locations. # Register an S3 bucket path with Lake Formation using AWS CLI
aws lakeformation register-data-location --resource-arn arn:aws:s3:::company-data-lake-prod/analytics/ --use-service-linked-role
Credential Vending Mechanics
Lake Formation does not modify physical S3 objects or re-encrypt files on disk. Instead, it acts as a credential vending service for integrated query engines such as Amazon Athena , Amazon Redshift Spectrum , Amazon EMR (with Lake Formation integration) , and AWS Glue ETL .
How Credential Vending Works
A user submits a SQL query through an integrated engine (e.g., Athena).
The query engine requests data access authorization from AWS Lake Formation on behalf of the user's IAM principal.
Lake Formation checks its permission database (evaluating named resource grants or LF-tags, column exclusions, and row filters).
If authorized, Lake Formation vends short-lived credentials for the registered S3 location.
The integrated query engine fetches the objects and enforces the authorized table, column, row, or cell filters. The credentials themselves are not byte-range credentials that independently encode row and column filters.
Because end users and IAM roles never receive direct s3:GetObject permissions on the raw S3 bucket, they cannot bypass business rules or access unapproved columns.
Fine-Grained Access Control (FGAC) & Data Cell Filters
AWS Lake Formation enables three levels of fine-grained access control:
Column-Level Security : Specifies explicit column inclusion or exclusion lists. For example, financial analysts can query customer_id and lifetime_value , but the credit_card_number column is omitted from query planning.
Row-Level Security : Uses SQL predicate expressions to restrict which rows a principal can view (e.g., region = 'us-east-1' or department = 'marketing' ).
Cell-Level Security : Combines column projection and row-level SQL expressions into a single policy entity called a Data Cell Filter .
Creating and Applying a Data Cell Filter
A Data Cell Filter is defined on a target Glue Data Catalog table and granted to specific IAM users, roles, or SAML groups. {
- "DataCellFilter": {
- "TableCatalogId": "123456789012",
- "DatabaseName": "financial_db",
- "TableName": "customer_transactions",
- "Name": "us_analyst_transactions_filter",
- "ColumnNames": ["transaction_id", "customer_id", "amount", "transaction_date"],
- "RowFilter": {
- "FilterExpression": "country_code = 'US' AND is_fraudulent = false"
- }
- }
- }
When a Data Analyst executes SELECT * FROM financial_db.customer_transactions in Athena, Lake Formation applies us_analyst_transactions_filter . The engine executes the query as if non-permitted columns ( ssn , tax_id ) and non-matching rows ( country_code != 'US' ) do not exist in the database.
Tag-Based Access Control (LF-TBAC)
Managing access via Named Resources (explicit grants on DatabaseA.TableB ) becomes cumbersome when managing hundreds of databases and thousands of tables. Tag-Based Access Control (LF-TBAC) provides a scalable governance model using policy abstractions.
LF-TBAC Concepts & Inheritance
Define LF-Tags : Administrators construct key-value tag taxonomies in Lake Formation (e.g., Environment: [Dev, Staging, Prod] , Confidentiality: [Public, Restricted, Secret] , Department: [Finance, HR, Engineering] ).
Assign LF-Tags to Catalog Resources : Tags are attached to Glue Databases, Tables, or individual Columns. Tags follow strict inheritance rules: Table tags inherit from Database tags, and Column tags inherit from Table tags unless overridden.
Grant Permissions on LF-Tag Expressions : Security policies grant privileges ( SELECT , ALTER , DROP ) to IAM principals based on logical tag expressions rather than explicit ARNs. # Grant SELECT permission to DataScienceRole for any table tagged Confidentiality=Restricted
- aws lakeformation grant-permissions \
- + --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/DataScienceRole \
- + --permissions SELECT \
+ --resource '{"LFTagPolicy":{"CatalogId":"123456789012","ResourceType":"TABLE","Expression":[{"TagKey":"Confidentiality","TagValues":["Restricted","Public"]}]}}'
When a new table is cataloged and tagged Confidentiality=Restricted , DataScienceRole instantly gains query access without requiring an administrator to execute manual grant commands.
Secure Cross-Account Data Sharing
Enterprise data architectures frequently separate data producers (e.g., Central Data Platform Account) from data consumers (e.g., Business Unit Accounts). AWS Lake Formation enables cross-account data sharing natively without requiring S3 data replication or complex cross-account IAM role switching.
Cross-Account Sharing Workflow
AWS RAM Integration : Lake Formation uses AWS Resource Access Manager (RAM) under the hood to manage catalog resource shares across AWS accounts or within an AWS Organization.
Producer Account Grant : The Data Lake Admin in Account A grants SELECT permissions on a catalog table or LF-Tag to the AWS Account ID of Account B.
Consumer Account Acceptance & Resource Link : Account B receives the RAM invitation (automatically accepted within an Organization). The Data Lake Admin in Account B creates a Resource Link —a local catalog alias—in Account B's Glue Data Catalog pointing to the shared table in Account A.
Query Execution : Analysts in Account B query the Resource Link via Athena. Lake Formation vends short-lived S3 access credentials to Account B's query engine, fetching data directly from Account A's S3 bucket while enforcing Account A's row/column permissions.
Lake Formation vs. Traditional IAM & S3 Access Control
- Feature Traditional S3 + IAM Policies AWS Lake Formation Governance
- Access Granularity S3 object prefix level (all-or-nothing file access) Column, Row, and Cell-level filtering
- Governance Model Decentralized IAM policies & bucket policies Centralized Data Lake Administrator console
- Policy Scalability Low (bounded by IAM policy size limits) High (Tag-Based Access Control - LF-TBAC)
- Credential Mechanism Direct IAM permissions on s3:GetObject Scoped temporary credential vending
- Cross-Account Access S3 Bucket Policies + IAM Cross-Account Roles AWS RAM integration + Glue Resource Links
Query Engine Integration Manual configuration per application Native integration with Athena, EMR, Redshift, Glue