Study guide
3.3 Managed Tables, External Tables, & Unity Catalog Volumes
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
3.3 Managed Tables, External Tables, & Unity Catalog Volumes
Key Takeaways Explain This Topic AI
Managed tables have their full lifecycle governed by Unity Catalog: dropping a managed table permanently deletes both metastore metadata and underlying storage files.
External tables decouple metadata from physical storage: dropping an external table removes metadata from Unity Catalog while leaving underlying ADLS Gen2 files completely intact.
Unity Catalog Volumes provide governed catalog endpoints for non-tabular, unstructured, and semi-structured files, replacing legacy DBFS root and mount (/mnt) paths.
Managed Volumes store files in schema/catalog root storage and delete files on drop; External Volumes point to external ADLS Gen2 URIs and preserve files on drop.
Files inside Volumes are accessed using standard POSIX filesystem paths: '/Volumes/<catalog>/<schema>/<volume>/<path>/<file>', compatible with Python open(), OS tools, and Spark APIs.
Last updated: August 2026
3.3 Managed Tables, External Tables, & Unity Catalog Volumes
DP-750 Exam Focus: Differentiate the lifecycle, storage management, and DROP behavior of Managed Tables versus External Tables. Understand how Unity Catalog Volumes provide fine-grained governance over non-tabular data, compare Managed versus External Volumes, and master the /Volumes/<catalog>/<schema>/<volume>/... POSIX file path syntax.
Tabular vs. Non-Tabular Storage Governance in Lakehouse
A modern enterprise lakehouse must govern two primary classes of digital assets:
Tabular Structured Data: Curated tables and analytical views querying Delta Lake, Parquet, or Iceberg formats.
Non-Tabular / Semi-Structured Files: Raw JSON landing drops, CSV extracts, log archives, image/audio files for computer vision, geospatial rasters, and machine learning model artifacts.
Unity Catalog unifies governance across both classes through two distinct securable constructs: Tables (Managed and External) and Volumes (Managed and External).
Managed Tables: Lifecycle, Storage, and Drop Behavior
A Managed Table is the default and recommended table type in Unity Catalog. When you create a managed table, Unity Catalog manages both the table metadata and the underlying physical data files in ADLS Gen2. +---------------------------------------------+
- | Managed Table |
- | CREATE TABLE prod.silver.customer_orders | |
- | (order_id INT, amount DOUBLE); |
- +---------------------------------------------+
- |
- v
- +---------------------------------------------+
- | Storage Location Inherited from: |
- | 1. Schema Managed Location (if set) |
- | 2. Catalog Managed Location (if set) |
- | 3. Metastore Root Location (fallback) |
- +---------------------------------------------+
- |
- | DROP TABLE customer_orders;
- v
- +---------------------------------------------+
- | Deletion Result: |
- | [X] Metadata deleted from Unity Catalog |
- | [X] Data files deleted from ADLS Gen2 |
- +---------------------------------------------+
- Managed Table Characteristics
Default Creation: Any CREATE TABLE statement without an explicit LOCATION clause creates a Managed Table.
Storage Location Inheritance: Data files are stored in a managed directory structured hierarchically: <managed-location>/__unitystorage/schemas/<schema-id>/tables/<table-id> .
Storage Location Resolution Priority:
Schema's MANAGED LOCATION (if explicitly set during schema creation).
Catalog's MANAGED LOCATION (if explicitly set during catalog creation).
Metastore's root storage location (if neither schema nor catalog specifies a location).
DROP Semantics (Critical Exam Topic): When a user executes DROP TABLE <managed_table>; , Unity Catalog removes the table metadata from the catalog AND permanently deletes the underlying physical Delta/Parquet data files from ADLS Gen2 (subject to Delta retention windows: tombstoned files stay in storage until VACUUM runs, and delta.deletedFileRetentionDuration defaults to 7 days, not 30). -- Creating a Managed Table in Unity Catalog
- CREATE TABLE prod_catalog.silver.sensor_readings (
- sensor_id STRING NOT NULL,
- reading_timestamp TIMESTAMP NOT NULL,
- temperature DOUBLE,
- pressure DOUBLE
- )
- USING DELTA
- PARTITIONED BY (DATE(reading_timestamp))
- COMMENT 'Managed Delta table with lifecycle governed by UC';
- External Tables: Decoupled Metadata and Storage
An External Table is a table whose metadata is registered in Unity Catalog, but whose underlying data files reside in an external cloud storage directory specified by an explicit LOCATION clause. +---------------------------------------------+
- | External Table |
- | CREATE TABLE prod.silver.legacy_logs |
- | LOCATION 'abfss://raw@adls/logs/'; |
- +---------------------------------------------+
- |
- | DROP TABLE legacy_logs;
- v
- +---------------------------------------------+
- | Deletion Result: |
- | [X] Metadata deleted from Unity Catalog |
- | [ ] Data files KEPT INTACT in ADLS Gen2 |
- +---------------------------------------------+
- External Table Characteristics
Explicit Location: Created by providing a LOCATION 'abfss://...' clause pointing to an existing directory in an authorized External Location.
Required Privilege: The creator must hold the CREATE EXTERNAL TABLE privilege on the corresponding Unity Catalog External Location object.
DROP Semantics (Critical Exam Topic): When an external table is dropped via DROP TABLE <external_table>; , Unity Catalog deletes ONLY the metadata registration . The underlying data files, Parquet directories, and transaction logs in ADLS Gen2 remain completely intact on cloud storage. -- Creating an External Table in Unity Catalog
- CREATE TABLE prod_catalog.bronze.partner_invoices (
- invoice_id STRING,
- vendor_code STRING,
- invoice_total DECIMAL(10,2),
- invoice_date DATE
- )
- USING DELTA
- LOCATION 'abfss://raw-data@adlsprod.dfs.core.windows.net/landing/invoices/';
- Comprehensive Comparison: Managed vs. External Tables
Architectural Dimension Managed Table External Table
DDL Syntax CREATE TABLE catalog.schema.table (...) (No LOCATION ) CREATE TABLE catalog.schema.table (...) LOCATION 'abfss://...'
Storage Path Managed automatically in schema/catalog root path Explicit path inside an authorized EXTERNAL LOCATION
DROP TABLE Action Deletes both metadata and physical data files Deletes metadata only ; physical files remain untouched
TRUNCATE TABLE Action Deletes table records and underlying file contents Deletes table records and underlying file contents
Required Privileges CREATE TABLE on target Schema CREATE TABLE on Schema + CREATE EXTERNAL TABLE on External Location
File Format Support Primarily Delta Lake (recommended and default) Delta Lake, Parquet, ORC, JSON, CSV, Avro
File Manipulation Direct out-of-band file edits in ADLS are prevented External systems can write files directly into the path
Unity Catalog Volumes: File-Level Governance
A Volume is a Unity Catalog securable object representing a logical volume of non-tabular, unstructured, or semi-structured storage in cloud object storage. +-----------------------------------+
- | Schema |
- | (prod_catalog.bronze) |
- +-----------------------------------+
- |
- +-----------------------+-----------------------+
- | |
- v v
- +-----------------------------------+ +-----------------------------------+
- | Managed Volume | | External Volume |
- | `landing_json_volume` | | `partner_sftp_volume` |
- +-----------------------------------+ +-----------------------------------+
- | Storage: Inherits schema root | | Storage: Explicit External Path |
- | Lifecycle: Files deleted on DROP | | Lifecycle: Files preserved on DROP|
- +-----------------------------------+ +-----------------------------------+
- Why Volumes Replace DBFS Mounts ( /mnt )
Legacy Risk: In legacy Databricks, accessing raw files required configuring DBFS mounts ( dbfs:/mnt/... ) or storing AWS/Azure credentials on clusters. DBFS mounts lacked fine-grained authorization—if a cluster could access /mnt/data , every user on that cluster had full access.
Modern Governance: Volumes provide full ANSI SQL privilege controls ( READ VOLUME , WRITE VOLUME ) on files, audit access via system tables, and eliminate shared storage credentials on compute nodes.
Managed Volumes vs. External Volumes
Dimension Managed Volume External Volume
Creation Syntax CREATE VOLUME catalog.schema.vol; CREATE EXTERNAL VOLUME catalog.schema.vol LOCATION 'abfss://...';
Storage Location Schema/catalog default managed storage Explicit path on an authorized External Location
DROP VOLUME Effect Deletes metadata and all files inside the volume Deletes metadata only ; underlying storage files are preserved
Primary Use Case Ephemeral landing zones, internal staging, temporary ML checkpoints Long-term external file ingestion, third-party SFTP/FTP dumps, archived media
Working with Volumes: SQL and POSIX File Access
Unity Catalog makes files stored in Volumes accessible via standard SQL commands and a POSIX-compliant filesystem path mapped directly into the operating system filesystem on all compute nodes.
- The Standard Volume Path Syntax
- POSIX Volume Path: /Volumes/<catalog>/<schema>/<volume_name>/<subpath>/<filename>
- 1. File Management via SQL -- Create a Managed Volume
- CREATE VOLUME prod_catalog.bronze.unstructured_landing
- COMMENT 'Managed volume for incoming PDF and JSON invoices';
- - Create an External Volume
- CREATE EXTERNAL VOLUME prod_catalog.bronze.partner_dropzone
- LOCATION 'abfss://partner-drops@adlsprod.dfs.core.windows.net/incoming/'
- COMMENT 'External volume pointing to vendor ingestion container';
- - List files in a Volume
- LIST '/Volumes/prod_catalog/bronze/unstructured_landing';
- 2. Reading Files in Python / PySpark
Because /Volumes is mounted as a local POSIX path via Databricks FUSE, standard Python file utilities, open-source libraries, and Spark APIs interact with files natively without needing special cloud storage drivers: # Standard Python file I/O
volume_file_path = "/Volumes/prod_catalog/bronze/unstructured_landing/invoices/inv_2026_08.json"
with open(volume_file_path, "r") as f:
- raw_json_data = f.read()
- print(f"Loaded {len(raw_json_data)} bytes")
- # Reading semi-structured JSON directly using PySpark DataFrame API
- df = spark.read.format("json").load("/Volumes/prod_catalog/bronze/unstructured_landing/invoices/")
- display(df)
- # Reading binary/image files for Computer Vision
- import cv2
- image = cv2.imread("/Volumes/prod_catalog/bronze/unstructured_landing/images/sample.jpg")
- 3. Granting Privileges on Volumes -- Grant read-only access to data scientists
- GRANT READ VOLUME ON VOLUME prod_catalog.bronze.unstructured_landing TO `data_scientists`;
- - Grant full write access to data engineering ETL pipelines
- GRANT READ VOLUME, WRITE VOLUME ON VOLUME prod_catalog.bronze.unstructured_landing TO `data_engineers`;
- Medallion Architecture Storage Strategy: Tables & Volumes
In a production Databricks Lakehouse architecture:
Bronze Layer: Non-tabular raw files (JSON, XML, CSV, media) arrive in Unity Catalog Volumes ( /Volumes/prod/bronze/landing_zone ). Auto Loader ( cloudFiles ) streams data from the Volume into Managed Bronze Delta Tables .
Silver Layer: Structured transformations, deduplication, and quality validations write exclusively into Managed Silver Delta Tables ( prod.silver.* ).
Gold Layer: Business aggregates, dimensional models, and curated reporting marts reside in Managed Gold Delta Tables and Unity Catalog Views ( prod.gold.* ).