Study guide
8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
Key Takeaways Explain This Topic AI
Delta Lake is the default and correct format for every managed table in the lakehouse; CSV, JSON, and raw Parquet belong at the landing boundary, not in silver or gold.
Only Delta Lake and Apache Iceberg provide ACID transactions, time travel, schema enforcement, and the file-skipping statistics that make lakehouse queries fast; CSV and JSON provide none of them.
Azure Databricks supports Iceberg specification versions 1, 2, and 3 over Apache Parquet files, as managed Iceberg tables in Unity Catalog or as read-only foreign Iceberg tables in another catalog.
Managed Iceberg tables require a Unity Catalog workspace, Databricks Runtime 16.4 LTS or above, serverless compute enabled, and predictive optimization turned on for table maintenance.
Iceberg v2 does not support position or equality deletes on Azure Databricks; row-level deletion is supported through Iceberg v3 deletion vectors instead.
Last updated: August 2026
8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
DP-750 Exam Focus: The blueprint bullet is "Choose a data table format, such as Parquet, Delta, CSV, JSON, or Iceberg." The exam is not asking you to like Delta - it is asking you to say which format satisfies a stated requirement, and to know when Iceberg is the answer.
1. Two Different Kinds of "Format"
The bullet mixes two categories, and conflating them is the most common error:
File formats - CSV, JSON, Parquet, Avro, ORC. These describe how bytes are laid out in one file. They carry no transaction log, so concurrent writers can corrupt a read, and there is no versioning.
Table formats - Delta Lake and Apache Iceberg. These add a metadata layer over Parquet files that provides ACID transactions, schema evolution, time travel, and statistics for file skipping.
- Capability CSV JSON Parquet Delta Lake Apache Iceberg
- Columnar / compressed No No Yes Yes Yes
- Schema in the file No Inferred Yes Yes Yes
- ACID transactions No No No Yes Yes
- Time travel No No No Yes Yes
- Schema enforcement and evolution No No Limited Yes Yes
- File-skipping statistics No No Row-group only Yes Yes
- MERGE / UPDATE / DELETE No No No Yes Yes
- Concurrent writers Unsafe Unsafe Unsafe Safe Safe
- 2. Where Each Format Belongs
Layer / use Format Reasoning
Source system export landing in a volume CSV / JSON as delivered You do not control the producer; land it as-is and keep it replayable
- Bronze table Delta Ingest the raw files into Delta immediately so bronze is transactional and time-travellable
- Silver and gold tables Delta MERGE , constraints, liquid clustering, predictive optimization all require it
Interchange with a non-Databricks engine that cannot speak Delta Iceberg (managed) or Parquet export Iceberg is the open interoperability path
Reading a table another platform already owns Foreign Iceberg table Read-only, metadata retrieved through Lakehouse Federation
One-off extract for a downstream tool Parquet or CSV written to a volume A file, not a table
The trap. Scenarios describing "we keep the bronze layer as compressed JSON to save conversion cost" are describing a defect, not an optimization: no ACID guarantees, no time travel, no file skipping, and every downstream read re-parses text. Ingest into Delta at the bronze boundary.
3. Apache Iceberg in Unity Catalog
Azure Databricks supports Iceberg tables that use the Apache Parquet file format and versions 1, 2, and 3 of the Iceberg specification. Iceberg maintains atomicity and consistency by writing new metadata files for each table change, and the Iceberg catalog is the layer that returns current metadata when a table is loaded.
Two distinct kinds exist, and the exam distinguishes them sharply:
Managed Iceberg Tables (Unity Catalog owns them)
Requirements:
- A workspace with Unity Catalog enabled
- Databricks Runtime 16.4 LTS or above
A workspace with serverless compute enabled - Databricks uses serverless to maintain Iceberg table metadata, so serverless must have network connectivity to the backing storage account
Predictive optimization enabled for table maintenance; managed Iceberg tables can only be created when it is on
What you get in return: Unity Catalog handles lifecycle tasks such as snapshot expiration and file compaction, liquid clustering works, predictive optimization automates maintenance, materialized views support incremental refresh, and streaming tables support incremental loading from Kafka and cloud object storage. -- A managed Iceberg table is created like any other Unity Catalog table
- CREATE TABLE prod_retail.gold.orders_iceberg (
- order_id BIGINT,
- store_id INT,
- order_ts TIMESTAMP,
- net_amount DECIMAL(18,2)
- ) USING ICEBERG;
- Foreign Iceberg Tables (another catalog owns them)
A foreign Iceberg table is managed by a catalog outside Unity Catalog - AWS Glue, a Hive metastore, or Snowflake Horizon Catalog. The external catalog holds the current metadata, and Azure Databricks uses Lakehouse Federation (Section 3.5) to retrieve that metadata and read the table from object storage. Foreign Iceberg tables are read-only and have limited platform support.
Access From External Iceberg Engines
Every Iceberg table in Unity Catalog is reachable through the Iceberg REST Catalog API , which supports read and write from external engines such as Apache Spark, Flink, Trino, and Kafka. The REST Catalog supports credential vending , delivering temporary credentials so an external engine can reach the underlying storage without a long-lived key. Credential vending is not supported on workspaces that use default storage.
4. Iceberg Limitations Worth Memorizing
Iceberg tables support only the Apache Parquet file format.
For Iceberg v2, position deletes and equality deletes are not supported ; Azure Databricks supports Iceberg v3 deletion vectors for row-level deletion instead (Section 8.3).
Branching and tagging are not supported. Only the main branch is accessible when reading foreign Iceberg tables.
Partition evolution works on managed Iceberg tables only when driven from external Iceberg engines through the REST Catalog, not from Databricks SQL. Foreign Iceberg tables do not support it at all.
Partitioning by expression transforms ( years() , months() , days() , hours() , bucket() ) is not supported for managed Iceberg tables, and partitioning by BINARY is unsupported generally.
Views are not accessible from external Iceberg engines.
Unsupported data types: UUID , Fixed(L) , TIME , and nested STRUCT with required fields.
Managed Iceberg tables cannot use Delta Lake generated columns , constraints , or collation , because Apache Iceberg does not implement them. All managed Iceberg tables use Zstd compression and the codec cannot be changed.
Foreign Iceberg tables support time travel only for snapshots previously read in Azure Databricks.
Constraints matter more than they look. Section 9.5 builds data quality on CHECK constraints and NOT NULL invariants. Those are Delta Lake features. Choosing managed Iceberg for a silver table gives up the constraint layer - a real architectural trade-off, not a footnote.
5. The Decision Table
- Requirement Format
- Default for any managed lakehouse table Delta Lake
- MERGE upserts, CHECK constraints, generated columns Delta Lake (Iceberg does not implement the latter two)
- External Iceberg engines must write to the table Managed Iceberg via the Iceberg REST Catalog
- Another platform owns the table and Databricks only reads it Foreign Iceberg table
- Interoperability with an engine that cannot read Delta, with Databricks still owning the table Managed Iceberg
- Raw landing of vendor extracts CSV / JSON as delivered, in a volume
- A columnar file handed to a downstream tool that reads files, not tables Parquet
- 6. Exam Traps
"Use Parquet for the silver layer for performance" is wrong. Parquet without a table format has no ACID guarantees, no MERGE , and no time travel.
Managed Iceberg is not a runtime toggle. It needs UC, DBR 16.4 LTS or above, serverless compute, and predictive optimization.
Foreign Iceberg tables cannot be written to. They are read-only, like every other federated object.
Iceberg v2 row-level deletes are not the answer on Databricks. Deletion vectors under Iceberg v3 are.