5.5 Apache Iceberg, S3 Tables & Open Table Operations

AWS Certified Data Engineer - Associate DEA-C01

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Streaming Data Ingestion

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning 1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns 1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda 1.4 Streaming Distribution, Replay & Processing State

2 Chapter 2: Batch Data Ingestion & Migration

2.1 AWS DMS (Database Migration Service) & Change Data Capture (CDC) 2.2 S3 Batch Operations & High-Throughput Bulk Data Ingestion 2.3 Hybrid Data Transfer with Snow Family & AWS DataSync 2.4 Application, API & Managed File Ingestion

3 Chapter 3: ETL Transformation with AWS Glue

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations 3.2 Glue Crawlers, Data Catalog & Schema Evolution 3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality 3.4 Incremental Processing with AWS Glue Job Bookmarks & Partition Processing 3.5 Technical and Business Catalogs, Connections & Lineage

4 Chapter 4: Big Data Processing & Workflow Orchestration

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem 4.2 EMR Serverless & Instance Selection (On-Demand vs. Spot Optimization) 4.3 Serverless Orchestration with AWS Step Functions 4.4 Airflow Pipeline Orchestration with Amazon MWAA 4.5 Containers, Lambda & Distributed Processing Performance 4.6 Data Engineering Code, IaC & CI/CD

5 Chapter 5: Object Storage & Data Lake Architecture

5.1 Amazon S3 Storage Classes, Lifecycle Policies & Cost Optimization 5.2 S3 Data Lake Partitioning, Bucketing & Layout Strategies 5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design 5.4 S3 Performance Optimization, Multipart Uploads & Transfer Acceleration 5.5 Apache Iceberg, S3 Tables & Open Table Operations

6 Chapter 6: Data Warehousing with Amazon Redshift

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage 6.2 Redshift Data Modeling: Distribution Styles (KEY, EVEN, ALL) & Sort Keys 6.3 Loading & Unloading Data with COPY/UNLOAD & Redshift Spectrum

7 Chapter 7: NoSQL, Relational & Specialty Data Stores

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling 7.2 Amazon RDS & Aurora Operations, Read Replicas & Multi-AZ 7.3 Graph & Analytical Querying with Amazon Neptune, OpenSearch & Athena 7.4 Vector Stores, Indexes & LLM Data Processing

8 Chapter 8: Pipeline Operations, Monitoring & Observability

8.1 Monitoring Pipeline Health with Amazon CloudWatch Metrics & Logs 8.2 Event-Driven Alerting & Automation with EventBridge & CloudWatch Alarms 8.3 Centralized Audit Logging with AWS CloudTrail

9 Chapter 9: Troubleshooting, Data Quality & Performance Tuning

9.1 Diagnosing & Resolving AWS Glue, EMR & Lambda Execution Failures 9.2 Redshift & Athena Query Performance Tuning & Memory Management 9.3 Implementing Data Quality Frameworks & Automated Validation Checks 9.4 Visualization, Notebooks, SQL Analytics, Sampling & Skew

10 Chapter 10: High Availability, Backup & Cost Optimization

10.1 Disaster Recovery, Cross-Region Replication & Backup Strategies 10.2 Designing High Availability Architectures Across AWS Data Services 10.3 Cost Optimization & Resource Utilization across Analytics Workloads

11 Chapter 11: Data Security, Encryption & Identity Management

11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions 11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies 11.3 Secure Credentials & Secrets Management with AWS Secrets Manager & Parameter Store 11.4 Private Network Connectivity & Isolation with AWS PrivateLink & VPC Endpoints

12 Chapter 12: Data Governance, Privacy & Compliance

12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation 12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie 12.3 Configuration Auditing & Compliance Enforcement with AWS Config 12.4 Data Privacy, Sovereignty, Sharing & Retention Guardrails 12.5 SageMaker Unified Studio Domains, Projects & Catalog Governance

5.5 Apache Iceberg, S3 Tables & Open Table Operations

Key Takeaways Explain This Topic AI

Apache Iceberg tracks table state through snapshots and metadata rather than relying on directory listings, enabling atomic commits, time travel, and safe concurrent table operations.

Schema evolution uses stable field IDs and partition evolution changes future layout without forcing queries to know physical folder paths or rewrite all historical data.

Amazon S3 Tables stores Iceberg tables in table buckets and provides automatic compaction, snapshot management, and unreferenced-file removal with Glue Data Catalog integration.

Iceberg optimistic concurrency detects conflicting metadata commits; writers must use retry logic and should avoid maintenance policies that remove snapshots still needed for recovery or audit.

Last updated: August 2026

5.5 Apache Iceberg, S3 Tables & Open Table Operations

Parquet, ORC, and Avro define file encodings. They do not by themselves make a collection of files behave like a database table. Apache Iceberg is an open table format that adds table metadata, snapshots, atomic commits, evolution, and maintenance over files in object storage.

Iceberg metadata model

An Iceberg table points to a current metadata file. That metadata references snapshots, manifest lists, and manifests that identify the exact data and delete files belonging to the table.

Layer Purpose
Table metadata Schema, partition specification, properties, snapshots, and current snapshot pointer
Snapshot One committed table state with parent and operation information
Manifest list Identifies manifests included in the snapshot and supplies pruning summaries
Manifest Lists data or delete files plus partition and column statistics
Data/delete files Parquet, ORC, or Avro data and row-level delete information

A query plans from metadata rather than listing every S3 object. It can prune manifests and files using statistics, which is important at data-lake scale.

Atomic commits and concurrency

An Iceberg writer produces new files and metadata, then atomically swaps the catalog pointer from the old metadata version to the new one. Readers continue seeing a consistent old or new snapshot rather than a half-written table.

Iceberg uses optimistic concurrency . Two writers can work from the same base snapshot, but a conflicting commit is detected when the pointer changes. The losing writer should refresh, validate whether its operation can still apply, and retry. Blindly overwriting the metadata location can lose another writer's changes.

This pattern supports append, overwrite, merge, update, and delete behavior when the engine and catalog implement the operation. ACID at the table level does not create a transaction across several unrelated tables; multi-table publication still needs orchestration or a higher-level manifest.

Schema and partition evolution

Iceberg assigns stable IDs to fields. Renaming a column does not have to look like dropping one field and adding an unrelated field, and compatible schema changes can be applied without rewriting all existing files. Type compatibility still matters; a narrowing conversion can lose data.

Partition evolution lets a table change its partition specification for future writes. Old files retain their former layout while queries use table metadata to plan both versions. This is different from Hive-style partition paths, where the query often depends directly on folder names. Iceberg supports transforms such as day, month, bucket, and truncate so consumers filter logical columns rather than constructing physical prefixes.

Snapshots, time travel, and rollback

Every successful commit creates a snapshot. A query engine can read a table as of a snapshot or timestamp for debugging, audit, or recovery. A rollback changes the current table state to a prior snapshot; it does not necessarily delete newer files immediately.

Snapshot expiration is an operational decision. Retaining every snapshot forever increases metadata and storage. Expiring too aggressively removes the ability to time travel and can make delayed jobs unable to find their base state. Align retention with recovery, legal-hold, and audit requirements before cleanup.

File maintenance

Frequent streaming writes create small files. Compaction rewrites them into fewer larger files, reducing query planning and S3 request overhead. It must preserve table semantics, apply relevant deletes, and commit a new snapshot. Maintenance competes with application writers, so both paths need conflict retry.

Other maintenance includes snapshot expiration and removal of unreferenced files. Never delete objects by scanning an Iceberg prefix with a generic script: files not referenced by the current snapshot can still be referenced by an older retained snapshot.

Amazon S3 Tables

S3 table buckets are an S3 bucket type for tables. Every S3 table uses Apache Iceberg. S3 manages maintenance operations including compaction and snapshot management, and table-bucket maintenance can remove unreferenced files. Automatic compaction is enabled by default and uses a configurable target file size; current defaults and allowed ranges belong in service configuration, not hard-coded application assumptions.

Integrating a table bucket with the AWS Glue Data Catalog makes the tables discoverable to services such as Athena and Redshift. Open-source engines can use the S3 Tables Catalog integration for Iceberg. Table-bucket permissions use S3 Tables actions and resource types, which differ from granting GetObject on an ordinary general-purpose bucket.

Choose self-managed Iceberg in a general-purpose bucket when you need engine-specific catalog or maintenance control. Choose S3 Tables when the supported integrations fit and managed table maintenance reduces operational burden.

Exam traps

Iceberg is not a file compression codec. A crawler finding Parquet files does not automatically create an Iceberg transaction log. Partition evolution is not the same as renaming S3 folders. And time travel depends on retained snapshots and files; a lifecycle rule that independently deletes table objects can corrupt the table.