Study guide
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture
Key Takeaways Explain This Topic AI
A Unity Catalog metastore is the top-level account container for metadata, deployed once per Azure region and shared across multiple regional workspaces.
The three-level namespace (catalog.schema.securable) establishes a standardized, hierarchical structure for referencing tables, views, volumes, models, and functions.
Account Admins create and bind metastores to workspaces in the Databricks Account Console (accounts.azuredatabricks.net), superseding legacy workspace-isolated Hive metastores.
Default catalogs include 'main' (default workspace catalog), 'samples' (read-only Databricks sample datasets), and 'system' (operational metadata, lineage, billing, and audit logs).
Session resolution supports USE CATALOG and USE SCHEMA statements, enabling single-identifier, two-level, or fully-qualified three-level referencing in SQL and DataFrame queries.
Last updated: August 2026
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture
DP-750 Exam Focus: Understand the relationship between Azure Databricks Account Console, Unity Catalog metastores, workspaces, and the 3-level namespace hierarchy. Be prepared to identify metastore regionality rules, workspace assignment mechanics, default catalogs ( main , samples , system ), and session namespace resolution behavior using USE CATALOG and USE SCHEMA .
Evolution of Databricks Metastore Architecture
In legacy Azure Databricks architectures, metadata management was bound to individual workspaces via the Hive Metastore (HMS) . Each workspace maintained its own isolated catalog, requiring complex external synchronization (such as shared external Azure SQL databases for HMS) or brittle credential pass-through mechanisms to share data across teams. Access control was fragmented, audit logging was siloed, and fine-grained governance at the row or column level required complex third-party add-ons.
Unity Catalog (UC) fundamentally transforms this model by elevating metadata, identity, and access governance to the Databricks Account level . Instead of each workspace operating as an isolated data island, Unity Catalog introduces a centralized governance plane that spans workspaces, clouds, and compute engines.
Legacy Hive Metastore vs. Unity Catalog Metastore
- Architectural Dimension Legacy Hive Metastore (HMS) Unity Catalog Metastore
- Governance Scope Workspace-local ( hive_metastore ) Account-level, shared across multiple workspaces
- Namespace Structure 2-level: schema.table (or database.table ) 3-level: catalog.schema.securable_object
Identity Management Workspace-local users and groups Centrally synchronized Account-level users, service principals, and groups
Access Control Table ACLs tied to workspace compute; DBFS mount permissions ANSI SQL GRANT / REVOKE on securables, unified across all compute types
Storage Connectivity Storage keys, SAS tokens, DBFS mounts ( /mnt ) Azure Access Connectors, Storage Credentials, External Locations
Auditing & Lineage Manual parsing of cluster logs and diagnostic settings Automated column-level lineage and centralized system tables
Unity Catalog Metastore Deployment & Workspace Assignment
The Unity Catalog Metastore is the top-level container of metadata in Unity Catalog. It stores metadata about data assets (catalogs, schemas, tables, views, volumes) and the access control policies that govern them. +---------------------------------------------------+
- | Databricks Account Console |
- | (accounts.azuredatabricks.net) |
- +---------------------------------------------------+
- |
- +-----------------------+-----------------------+
- | |
- v v
- +-----------------------------------+ +-----------------------------------+
- | UC Metastore: East US Region | | UC Metastore: West Europe Region|
- | - Root ADLS Gen2 Storage Path | | - Root ADLS Gen2 Storage Path |
- +-----------------------------------+ +-----------------------------------+
- | |
- +------------+------------+ v
- | | +---------------------------+
- v v | Workspace: Prod-EU |
- +-----------------------+ +-----------------------+ +---------------------------+
- | Workspace: Dev-EastUS | | Workspace: Prod-EastUS|
- +-----------------------+ +-----------------------+
- Metastore Scoping and Regionality Rules
One Metastore Per Region: A Databricks account can have exactly one Unity Catalog metastore per cloud region . For example, an organization with workloads in East US and West US will deploy one metastore in East US and another metastore in West US .
Multi-Workspace Binding: Multiple Databricks workspaces deployed within the same Azure region can (and should) be assigned to the same regional metastore . When workspaces share a metastore, they share the identical catalog namespace, table definitions, access permissions, and automated lineage.
Cross-Region Workspaces: Workspaces in different regions cannot be assigned to the same metastore. If a workspace is located in East US , it must be linked to the East US metastore.
Account Admin Privileges: Metastore creation and workspace assignment are performed exclusively in the Databricks Account Console ( accounts.azuredatabricks.net ) by an Account Admin . Once created, the Account Admin delegates administrative control by designating a Metastore Admin (ideally a group such as data-governance-admins ).
Metastore Root Storage
When an Account Admin creates a Unity Catalog metastore, they specify a cloud storage location in Azure Data Lake Storage Gen2 (ADLS Gen2):
Root Path Syntax: abfss://<container>@<storage-account>.dfs.core.windows.net/<metastore-id>
This metastore root storage serves as the default physical location for:
Metastore-level managed tables and managed volumes (when not overridden at the catalog or schema level).
System tables and telemetry logs generated by Databricks.
Internal metadata and change data feed checkpointing.
Important: Changing the metastore root storage after creation is not supported. It is critical to configure enterprise-grade lifecycle rules, Azure soft-delete policies, and infrastructure redundancy on this ADLS Gen2 account before creating the metastore.
The 3-Level Namespace Hierarchy
Unity Catalog organizes all data and analytical assets into an explicit three-level namespace : <catalog>.<schema>.<securable_object> . +-----------------------------------+
- | Catalog |
- | (Top-level boundary) |
- +-----------------------------------+
- |
- v
- +-----------------------------------+
- | Schema |
- | (Logical database) |
- +-----------------------------------+
- |
- +-------------------+-------------+-------------+-------------------+
- | | | |
- v v v v
- +-------------------+ +-------------------+ +-------------------+ +-------------------+
- | Tables | | Views | | Volumes | | Functions / |
- | (Managed/External)| | (Standard/Dynamic)| | (Managed/External)| | Models |
- +-------------------+ +-------------------+ +-------------------+ +-------------------+
- 1. Catalog (Level 1)
The Catalog is the top-level container for data isolation and organizational segregation. Typical enterprise strategies structure catalogs by:
- Environment: dev , staging , prod
- Business Unit / Domain: finance , marketing , supply_chain
- Data Architecture Tier: raw_ingest , curated_enterprise
Catalogs can have an optional dedicated managed storage location in ADLS Gen2. If specified, all managed tables and managed volumes created within any schema inside that catalog inherit this storage location instead of falling back to the metastore root.
2. Schema / Database (Level 2)
The Schema (historically interchangeable with the term Database ) is a logical grouping of data assets within a catalog. In lakehouse medallion architecture, schemas frequently map to data processing stages:
bronze : Raw, ingested event streams and append-only datasets.
silver : Cleansed, deduplicated, enriched, and standardized datasets.
gold : Aggregated, business-level metric tables and reporting views.
Like catalogs, schemas can also define their own explicit managed storage location , providing granular control over where data files physically reside in Azure storage.
3. Securable Objects (Level 3)
The third level represents the actual executable and queryable assets:
Tables: Tabular data stored in Delta Lake (or Parquet/CSV/ORC/JSON format), categorized as Managed or External .
Views: Saved SQL queries, including standard views, parameterized views, and dynamic views with row/column masking.
Volumes: Logical governance endpoints for non-tabular, unstructured, or semi-structured files (e.g., CSV, JSON, images, PDFs, parquet landing drops).
Functions: Registered User-Defined Functions (UDFs) and SQL scalar/table functions stored in the catalog.
Models: Registered machine learning models tracked in MLflow and governed under Unity Catalog.
Default Built-in Catalogs
When a Unity Catalog metastore is provisioned and linked to a workspace, Databricks automatically exposes several built-in catalogs: +---------------------------------------------------+
- | Unity Catalog Default Catalogs |
- +---------------------------------------------------+
- | |
- | +----------------+ +-------------------------+ |
- | | main | | samples | |
- | | (Default Data) | | (Read-Only Datasets) | |
- | +----------------+ +-------------------------+ |
- | |
- | +----------------+ +-------------------------+ |
- | | system | | hive_metastore | |
- | | (Observability)| | (Legacy Compatibility)| |
- | +----------------+ +-------------------------+ |
- +---------------------------------------------------+
- 1. The main Catalog
Purpose: Created automatically as the default catalog for user data.
Behavior: If a SQL query references a 2-level name ( schema.table ) without an active catalog set, or when migrating legacy workloads, Databricks defaults to main .
Production Best Practice: Enterprises typically restrict access to main or remove default permissions, establishing dedicated environment catalogs (e.g., prod_lakehouse , dev_sandbox ) with tailored storage locations and RBAC.
2. The samples Catalog
Purpose: A globally provisioned, read-only catalog provided by Databricks.
Contents: Contains standardized benchmark datasets such as samples.nyctaxi.trips , samples.tpch.orders , and samples.bakeoff .
Usage: Ideal for training, POCs, onboarding, and testing syntax without incurring storage costs or requiring cloud storage configuration.
3. The system Catalog
Purpose: The centralized operational and observability engine of Unity Catalog.
Contents: Exposes SQL-queryable audit logs, usage telemetry, and information schema views across the entire account:
system.access : Audit logs detailing user activity, logins, and permission changes.
system.billing : Granular DBU consumption and infrastructure cost metrics.
system.compute : Cluster start/stop events, node utilization, and runtime versions.
system.lakeflow : Jobs, tasks, pipeline runs, and trigger histories.
system.information_schema : ANSI standard metadata on catalogs, schemas, tables, columns, and privileges.
Access Control: Query access to system schemas is granted by Metastore Admins using standard SQL grants (e.g., GRANT USE SCHEMA ON SCHEMA system.billing TO billing_analysts ).
4. The hive_metastore Catalog
Purpose: Virtual catalog representing the legacy, workspace-local Hive metastore.
Behavior: Allows existing pipelines running on Unity Catalog-enabled compute to query legacy tables located in /user/hive/warehouse or DBFS root mounts ( dbfs:/mnt/... ) using the 3-level syntax: hive_metastore.default.my_legacy_table .
Governance Limitation: Does not support fine-grained Unity Catalog features like row filters, column masks, or automated lineage.
Session Resolution and Namespace Context
To interact with assets in Unity Catalog, queries can reference objects using fully qualified 3-level names, or set the session context to simplify query syntax.
Setting Active Context: USE CATALOG and USE SCHEMA -- Set the active catalog for the current interactive notebook or SQL session
USE CATALOG prod_catalog;
- - Set the active schema within the active catalog
- USE SCHEMA silver;
- - Now queries can resolve 1-level table names directly
- SELECT customer_id, first_name, email
- FROM customers
- WHERE is_active = true;
- Resolution Precedence Rules
Query Syntax Example Resolution Logic
Fully Qualified (3-Level) SELECT * FROM prod_catalog.silver.customers; Unambiguous. Explicitly targets prod_catalog , schema silver , table customers .
Partially Qualified (2-Level) SELECT * FROM silver.customers; Resolves against the currently active session catalog (e.g., <current_catalog>.silver.customers ).
Unqualified (1-Level) SELECT * FROM customers; Resolves against the currently active session catalog and schema (e.g., <current_catalog>.<current_schema>.customers ).
Cross-Catalog Join SELECT o.*, c.name FROM prod_catalog.gold.orders o JOIN ref_catalog.shared.customers c ON o.cust_id = c.id; Joins tables across completely distinct catalogs in a single SQL query without federation overhead. -- Cross-Catalog Querying Example
- SELECT
- ord.order_id,
- ord.order_date,
- ord.total_amount,
- dim.customer_name,
- dim.region
- FROM prod_finance.gold.fact_orders ord
- INNER JOIN prod_enterprise.gold.dim_customer dim
- ON ord.customer_id = dim.customer_id
- WHERE ord.order_date >= '2026-01-01';
- Enterprise Catalog Design Patterns
When designing Unity Catalog architectures for the DP-750 exam, consider the three primary architectural patterns: Pattern 1: Environment Isolation Pattern 2: Business Unit / Domain
- +--------------------------------+ +--------------------------------+
- | Catalogs: | | Catalogs: |
- | - dev_catalog | | - finance_catalog |
- | - staging_catalog | | - marketing_catalog |
- | - prod_catalog | | - supplychain_catalog |
- | Schemas: | | Schemas: |
- | - bronze, silver, gold | | - raw, intermediate, mart |
- +--------------------------------+ +--------------------------------+
Environment-First (Recommended for strict isolation): Separate catalogs represent environments ( dev_catalog , staging_catalog , prod_catalog ). Storage credentials and external locations are strictly isolated per storage account, preventing non-production compute from accessing production storage.
Domain/Business Unit-First: Catalogs represent business units ( finance_catalog , marketing_catalog , hr_catalog ). Schemas inside represent lifecycle layers ( bronze , silver , gold ).
Hybrid Model: Catalogs combine domain and environment (e.g., prod_finance , dev_finance , prod_marketing ), offering the highest granularity of RBAC and dedicated storage container binding.