Study guide
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions
AWS Certified Data Engineer - Associate DEA-C01
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Streaming Data Ingestion
1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State
2 Chapter 2: Batch Data Ingestion & Migration
2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion
3 Chapter 3: ETL Transformation with AWS Glue
3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage
4 Chapter 4: Big Data Processing & Workflow Orchestration
4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD
5 Chapter 5: Object Storage & Data Lake Architecture
5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations
6 Chapter 6: Data Warehousing with Amazon Redshift
6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum
7 Chapter 7: NoSQL, Relational & Specialty Data Stores
7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing
8 Chapter 8: Pipeline Operations, Monitoring & Observability
8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail
9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning
9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew
10 Chapter 10: High Availability, Backup & Cost Optimization
10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads
11 Chapter 11: Data Security, Encryption & Identity Management
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints
12 Chapter 12: Data Governance, Privacy & Compliance
12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance
11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions
Key Takeaways Explain This Topic AI
IAM policy evaluation defaults to implicit deny, requiring an explicit allow while any explicit deny overrides all allows regardless of evaluation order or policy location.
Attribute-Based Access Control (ABAC) uses aws:PrincipalTag/${TagKey} and aws:ResourceTag/${TagKey} condition keys to dynamically scale permissions without updating IAM policies for new users or resources. S3 fine-grained access control uses condition keys such as s3:prefix and s3:delimiter to constrain user operations to specific virtual folders within a bucket.
Fine-grained access control (FGAC) for DynamoDB relies on dynamodb:LeadingKeys for row-level (partition key) security and dynamodb:Attributes with dynamodb:Select for column-level filtering.
Cross-account role assumption requires a trust policy that authorizes the calling principal; third-party access should also require a unique sts:ExternalId to mitigate the confused-deputy problem.
IAM controls AWS API access and authentication paths, while Amazon Redshift users, groups, and database roles control privileges inside a database; COPY or UNLOAD can require both database privileges and an IAM role attached to the warehouse.
Last updated: August 2026
Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions
Security is a zero-compromise domain in data engineering. Building robust data pipelines on AWS requires strict enforcement of the Principle of Least Privilege . AWS Identity and Access Management (IAM) provides the foundation for authentication and authorization across all data processing, storage, and analytics services. For the AWS Certified Data Engineer - Associate (DEA-C01) exam, you must master IAM policy structure, permission evaluation logic, Attribute-Based Access Control (ABAC), and service-specific Fine-Grained Access Control (FGAC) techniques.
IAM Policy Architecture & Evaluation Logic
Every access request made to an AWS API is evaluated by the IAM policy engine. To grant or restrict access, IAM evaluates several policy types attached to or surrounding an IAM principal:
Identity-Based Policies : Attached directly to IAM Users, Groups, or Roles (Managed or Inline).
Resource-Based Policies : Attached directly to AWS resources (e.g., S3 Bucket Policies, KMS Key Policies, Lambda Function Policies, Glue Resource Policies).
Permissions Boundaries : Advanced features that set the maximum allowable permissions an identity-based policy can grant.
Service Control Policies (SCPs) : Governance boundaries applied at the AWS Organizations OU or Account level.
Session Policies : Passed programmatically during temporary session creation ( sts:AssumeRole ).
The IAM Policy Evaluation Flow
Understanding policy evaluation order is a high-frequency exam topic:
Default State : By default, all requests are implicitly denied ( Implicit Deny ).
Explicit Deny Overrides Everything : If any policy applicable to the request contains an Explicit Deny , the final decision is immediately DENIED , regardless of how many Explicit Allow statements exist.
Explicit Allow and Boundaries : An identity- or resource-based allow must apply, and guardrails such as permissions boundaries, session policies, SCPs, and resource control policies must also allow the request where their evaluation model requires an intersection. A resource-based policy can affect this evaluation differently for a user, role, or role session, so use the documented matrix for edge cases. Request Received --> Explicit deny anywhere?
|-- YES --> [ DENIED ]
+-- NO --> Applicable identity/resource allow?
|-- NO --> [ DENIED (Implicit) ]
+-- YES --> Required boundary, session, SCP/RCP intersections permit it?
- |-- YES --> [ ALLOWED ]
- +-- NO --> [ DENIED ]
- Fine-Grained Access Control (FGAC) with IAM Conditions
While basic IAM policies grant broad permissions on actions (e.g., s3:GetObject ), data engineering workflows require fine-grained constraints based on runtime context. IAM Condition blocks restrict permissions using key-value comparisons.
Essential Global Condition Keys for Data Pipelines
Condition Key Purpose & Data Engineering Use Case
aws:PrincipalTag/${TagKey} Matches tags on the requesting IAM role/user for Attribute-Based Access Control (ABAC).
aws:ResourceTag/${TagKey} Matches tags attached to the target resource (e.g., S3 bucket, DynamoDB table, Glue job).
aws:SourceVpc / aws:SourceVpce Restricts API calls to originate from a specific VPC or VPC Endpoint ID.
aws:SourceIp Restricts access to corporate IP CIDR blocks (Note: Does not work for traffic passing through VPC Endpoints).
aws:MultiFactorAuthPresent Enforces MFA requirement for administrative operations like deleting S3 data or KMS keys.
Attribute-Based Access Control (ABAC)
Instead of creating separate IAM policies for every team or project, ABAC allows you to scale security dynamically using tags. For instance, an analytics team member tagged Department=Finance can access supported AWS resources whose authorization documentation exposes compatible resource-tag condition keys. For S3 general purpose buckets, first enable S3 ABAC in the account and Region before relying on bucket tags for authorization. The following policy demonstrates the pattern after enablement; do not assume every database object supports IAM resource tags: {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "ABACS3BucketRead",
- "Effect": "Allow",
- "Action": ["s3:GetObject", "s3:ListBucket"],
- "Resource": [
- "arn:aws:s3:::corporate-data-lake",
- "arn:aws:s3:::corporate-data-lake/*"
- ],
- "Condition": {
- "StringEquals": {
- "aws:ResourceTag/Department": "${aws:PrincipalTag/Department}"
- }
- }
- }
- ]
- }
- Service-Specific FGAC Implementation Patterns
- 1. Amazon S3 Virtual Folder Security ( s3:prefix & s3:delimiter )
Data lakes partition data into S3 prefixes (e.g., s3://my-lake/raw/finance/ vs s3://my-lake/raw/hr/ ). To restrict a team's read access strictly to their folder, condition keys MUST regulate both s3:GetObject and s3:ListBucket : {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "AllowListBucketPrefix",
- "Effect": "Allow",
- "Action": "s3:ListBucket",
- "Resource": "arn:aws:s3:::my-company-data-lake",
- "Condition": {
- "StringLike": {
- "s3:prefix": ["raw/finance/*", "raw/finance"]
- }
- }
- },
- {
- "Sid": "AllowGetObjectInsidePrefix",
- "Effect": "Allow",
- "Action": "s3:GetObject",
- "Resource": "arn:aws:s3:::my-company-data-lake/raw/finance/*"
- }
- ]
- }
- 2. Amazon DynamoDB Row-Level & Column-Level Security
For transactional or operational data stores in DynamoDB, fine-grained access control protects sensitive user data without duplicating tables:
Row-Level Security ( dynamodb:LeadingKeys ) : Restricts a user or application to querying/updating items where the Partition Key equals their specific identity (e.g., UserId ).
Column-Level Security ( dynamodb:Attributes ) : Specifies exact attribute names that can be read or written, combined with dynamodb:Select to prevent retrieving forbidden fields. {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "DynamoDBRowAndColumnFGAC",
- "Effect": "Allow",
- "Action": ["dynamodb:GetItem", "dynamodb:Query"],
- "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/UserProfileStore",
- "Condition": {
- "ForAllValues:StringEquals": {
- "dynamodb:LeadingKeys": ["${aws:PrincipalTag/ApplicationId}"],
- "dynamodb:Attributes": ["UserId", "Email", "Preferences"]
- },
- "StringEqualsIfExists": {
- "dynamodb:Select": "SPECIFIC_ATTRIBUTES"
- }
- }
- }
- ]
- }
- IAM Roles for Data Pipelines & Cross-Account Access
Data processing services (AWS Glue, Amazon EMR, AWS Lambda, Amazon Redshift) do not use static IAM User credentials. Instead, they assume service Execution Roles via AWS Security Token Service (STS) to obtain temporary security credentials.
Cross-Account Access & The Confused Deputy Problem
When a third-party vendor or a separate production AWS account needs to access your data lake, cross-account IAM role assumption must be established using a Trust Policy (Resource Policy on the target Role).
To prevent the Confused Deputy Problem (where an attacker tricks a third-party service into accessing another customer's resources), require a third-party-generated, unique-per-customer sts:ExternalId in the trust policy. An external ID is an authorization discriminator, not a secret , so do not treat it as a password: {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "CrossAccountRoleTrustPolicy",
- "Effect": "Allow",
- "Principal": {
- "AWS": "arn:aws:iam::987654321098:root"
- },
- "Action": "sts:AssumeRole",
- "Condition": {
- "StringEquals": {
- "sts:ExternalId": "vendor-generated-customer-12345"
- }
- }
- }
- ]
- }
- Summary Checklist for DEA-C01 Exam
Explicit Deny always wins regardless of where it is declared.
ABAC uses tags ( aws:PrincipalTag vs aws:ResourceTag ) to scale permissions without policy sprawl.
DynamoDB FGAC uses dynamodb:LeadingKeys (row) and dynamodb:Attributes (column).
Cross-account access uses sts:AssumeRole + sts:ExternalId .
Amazon Redshift Database Users, Groups, and Roles
IAM and Redshift database authorization operate at different layers. IAM can authorize Redshift API calls, temporary database credentials, IAM Identity Center integration, or a service role that lets the warehouse call S3 for COPY and UNLOAD. After a session reaches a database, native Redshift privileges decide whether that identity can use a schema, read a table, execute a function, or create an object.
A database user is a login identity. A group is the older way to collect users. A database role is the preferred reusable collection of privileges and can be granted to users or other roles to form a hierarchy. Grant only the required schema and object privileges—for example, a reporting role commonly needs USAGE on a schema plus SELECT on specified tables. Avoid routine superuser grants, and configure default privileges deliberately if future objects should inherit access.
The layers are not interchangeable. Granting an IAM role permission to call Redshift APIs does not grant SELECT inside a database. Granting a database role SELECT does not let COPY read an S3 object unless the warehouse also assumes an IAM role with the required S3 and KMS permissions. Troubleshoot authorization by identifying which layer denied the request: identity federation or temporary credentials, database CONNECT and object grants, or the service role used for an external AWS call.