Study guide
5.4 Secure Data Sharing within Azure Databricks
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
5.4 Secure Data Sharing within Azure Databricks
Key Takeaways Explain This Topic AI
Delta Sharing within Azure Databricks enables secure, zero-copy live data sharing across workspaces, regions, and cloud tenants without data replication or storage egress.
Databricks-to-Databricks (UC-to-UC) sharing utilizes native Unity Catalog authentication, eliminating the need to manage external credential files or bearer tokens.
A Share is a securable object containing tables, partitions, views, volumes, and models, managed by a Data Provider and granted to specific Recipients.
Dynamic views with row filters and column masks can be shared across workspaces, preserving fine-grained access control across organizational boundaries.
Delta Sharing supports Change Data Feed (CDF) and Delta Lake time travel, enabling downstream consumer workspaces to process incremental updates.
Last updated: August 2026
5.4 Secure Data Sharing within Azure Databricks
DP-750 Exam Focus: Master internal data sharing using Delta Sharing in Azure Databricks. Understand the architectural mechanics of Databricks-to-Databricks (UC-to-UC) sharing, including the Provider and Recipient model, creating and managing Shares ( CREATE SHARE , ALTER SHARE ), configuring Recipients with Sharing Identifiers, mounting shares into consumer Unity Catalog metastores ( CREATE CATALOG ... USING SHARE ), and sharing advanced securables like Dynamic Views, Volumes, and Change Data Feed (CDF).
1. Internal Enterprise Data Sharing: The Zero-Copy Paradigm
Historically, when business units, regional branches, or distinct Azure Databricks workspaces needed to share datasets, engineering teams relied on data replication pipelines : LEGACY REPLICATION ANTI-PATTERN
- [ Provider Workspace ] [ Consumer Workspace ]
- +--------------------+ +--------------------+
- | Source Delta Table | ---> [ Azure Data Factory / ETL ] --> | Copy Delta Table |
- | (10 TB Dataset) | - Scheduled Batch Copy | (Storage Duplicate)|
- +--------------------+ - Network Egress Billed +--------------------+
- Stale Data (Sync Lag) - 2x Storage Cost
- Complex Error Recovery - Out-of-Sync Lineage
This replication approach introduces severe operational liabilities:
Storage Duplication & High Cloud Costs: Paying double or triple storage costs for identical multi-terabyte datasets.
Network Egress Fees: Continuous cross-region data transfer incurs significant network bandwidth charges.
Data Drift & Stale Analytics: Downstream consumers work on snapshot copies that lag hours or days behind production updates.
Governance Gaps: Compliance teams lose track of duplicate datasets scattered across disparate storage accounts.
Delta Sharing is an open, secure protocol developed by Databricks for real-time, zero-copy data exchange. Within Azure Databricks, Databricks-to-Databricks (UC-to-UC) Sharing allows a data provider to grant direct, governed query access on live Delta tables to consumer workspaces—across regions, subscriptions, and Entra ID tenants— without copying a single byte of underlying data .
2. Databricks-to-Databricks (UC-to-UC) Sharing Architecture
In Databricks-to-Databricks sharing, both the data provider and data consumer operate Unity Catalog-enabled Azure Databricks environments. Authentication is handled natively through Unity Catalog trust handshakes, eliminating the need to generate or exchange external credential activation URLs or bearer token files. +-----------------------------------------------------------------------------------------+
- | DATABRICKS-TO-DATABRICKS (UC-TO-UC) SHARING |
- +-----------------------------------------------------------------------------------------+
- | |
- | PROVIDER ENVIRONMENT (Metastore East US): |
- | 1. Data Provider creates a Share: |
- | CREATE SHARE finance_share; |
- | ALTER SHARE finance_share ADD TABLE prod.finance.fact_revenue; |
- | |
- | 2. Data Provider creates a Recipient using Consumer's Sharing Identifier: |
- | CREATE RECIPIENT marketing_team_recipient |
- | USING ID 'azure:eastus:11111111-2222-3333-4444-555555555555:metastore-id'; |
- | |
- | 3. Provider Grants Share Access: |
- | GRANT SELECT ON SHARE finance_share TO RECIPIENT marketing_team_recipient; |
- | |
- | ------------------------------------------------------------------------------------- |
- | |
- | CONSUMER ENVIRONMENT (Metastore West Europe / Other Workspace): |
- | 4. Consumer discovers shared data & creates a local catalog: |
- | CREATE CATALOG shared_finance USING SHARE provider_metastore.finance_share; |
- | |
- | 5. Consumer Queries Data Live (Zero-Copy): |
- | SELECT * FROM shared_finance.finance.fact_revenue; |
- +-----------------------------------------------------------------------------------------+
- Provider vs. Consumer Responsibilities
Architectural Component Provider Role & Actions Consumer (Recipient) Role & Actions
Securable Ownership Owns the physical ADLS Gen2 storage and base Delta tables. Mounts shared data as a read-only local Unity Catalog catalog.
Compute Billing Incurs no compute costs for consumer queries. Pays for their own Spark compute / SQL Warehouse clusters querying the data.
Storage Billing Pays for underlying Delta Lake data storage at rest. Incurs zero storage costs (queries data in-place).
Access Revocation Can revoke share access or individual tables instantly via SQL. Loses immediate access as soon as privileges are revoked.
3. Core Delta Sharing Securables & DDL Syntax
Setting up UC-to-UC sharing involves four primary objects: Shares , Securables in the Share , Recipients , and Share Grants . DELTA SHARING OBJECT MODEL
- +-------------------------------------------------------------------------+
- | Unity Catalog Provider |
- | |
- | +-------------------------------------------------------------------+ |
- | | SHARE: sales_share | |
- | | - Table: prod.sales.customers (AS client_directory) | |
- | | - Table: prod.sales.orders (WITH CHANGE DATA FEED) | |
- | | - Table: prod.sales.regional_metrics (PARTITION: region = 'EMEA') | |
- | | - Volume: prod.sales.raw_contracts | |
- | +-------------------------------------------------------------------+ |
- | | |
- | | GRANT SELECT |
- | v |
- | +-------------------------------------------------------------------+ |
- | | RECIPIENT: analytics_division_recipient | |
- | | (Bound via Metastore Sharing Identifier) | |
- | +-------------------------------------------------------------------+ |
- +-------------------------------------------------------------------------+
- Step 1: Create a Share
- A Share is a container for data assets managed by the provider metastore: CREATE SHARE global_sales_share
- COMMENT 'Live enterprise sales and customer metrics shared with regional analytics teams';
- Step 2: Add Tables and Partitions to the Share
Providers can add full tables, table aliases, partitioned slices, or tables enabled for Change Data Feed (CDF): -- Method A: Add an entire table
- ALTER SHARE global_sales_share
- ADD TABLE prod.sales.customer_dimension;
- - Method B: Add a table with a sanitized alias (hides internal schema name)
- ALTER SHARE global_sales_share
- ADD TABLE prod.sales.fact_transactions AS sales_data.transactions;
- - Method C: Add a specific partition slice (Partition Sharing)
- ALTER SHARE global_sales_share
- ADD TABLE prod.sales.regional_inventory
- PARTITION (inventory_region = 'EMEA');
- - Method D: Share with Change Data Feed (CDF) enabled
- ALTER SHARE global_sales_share
- ADD TABLE prod.sales.order_events
- WITH CHANGE DATA FEED;
- Step 3: Add Volumes and Models to the Share
Delta Sharing is not restricted to tabular data; providers can share Volumes (unstructured files) and ML Models : -- Share an unstructured Unity Catalog Volume containing market research PDFs
- ALTER SHARE global_sales_share
- ADD VOLUME prod.marketing.research_reports;
- - Share a registered MLflow model
- ALTER SHARE global_sales_share
- ADD MODEL prod.ml_models.churn_prediction_v2;
- Step 4: Register the Recipient
In Databricks-to-Databricks sharing, the consumer provides their Sharing Identifier (obtained from their local Unity Catalog metastore). The provider creates the recipient object using this identifier: -- Create recipient using the consumer metastore sharing ID
- CREATE RECIPIENT emea_analytics_recipient
- USING ID 'azure:westeurope:a1b2c3d4-e5f6-7890-abcd-ef1234567890:metastore-998877'
- COMMENT 'Recipient for West Europe analytics team';
- Step 5: Grant Share Permissions to the Recipient GRANT SELECT ON SHARE global_sales_share
- TO RECIPIENT emea_analytics_recipient;
- 4. Consumer Experience: Mounting a Share into Unity Catalog
On the consumer side, the recipient accesses the shared data through standard Unity Catalog SQL commands without needing access keys or API credentials.
Step 1: Discover Available Providers and Shares -- List all data providers who have shared data with this metastore
SHOW PROVIDERS;
- - Inspect shares offered by a specific provider
- SHOW SHARES IN PROVIDER eastus_finance_provider;
- Step 2: Create a Local Catalog from the Share
The consumer creates a new local catalog that points directly to the provider's share: -- Mount the share into the consumer's 3-level namespace
- CREATE CATALOG shared_global_sales
- USING SHARE eastus_finance_provider.global_sales_share;
- - Grant access to internal consumer groups
- GRANT USE CATALOG ON CATALOG shared_global_sales TO `emea_data_analysts`;
- GRANT USE SCHEMA, SELECT ON SCHEMA shared_global_sales.sales_data TO `emea_data_analysts`;
- Step 3: Query Shared Tables In-Place
Consumers can run standard ANSI SQL queries, Spark DataFrames, and cross-catalog joins: -- Query the shared Delta table live
- SELECT
- c.customer_name,
- t.transaction_id,
- t.transaction_amount,
- t.transaction_date
- FROM shared_global_sales.sales_data.transactions t
- INNER JOIN local_emea_catalog.analytics.dim_store s
- ON t.store_id = s.store_id
- WHERE t.transaction_date >= '2026-01-01';
- 5. Advanced Sharing Capabilities: Dynamic Views & Change Data Feed
- 1. Sharing Governed Dynamic Views
When sharing data with external departments, providers often need to apply row filters and column masks. In Delta Sharing, providers can add Dynamic Views to a share. When consumers query the view, the provider's masking logic evaluates transparently, guaranteeing that consumers only see redacted or authorized subsets. -- Provider creates a sanitized view
- CREATE VIEW prod.sales.sanitized_orders_view AS
- SELECT
- order_id,
- customer_id,
- mask_credit_card(credit_card_num) AS credit_card_num,
- order_amount
- FROM prod.sales.orders
- WHERE order_status != 'CANCELLED';
- - Provider adds view to share
- ALTER SHARE global_sales_share
- ADD VIEW prod.sales.sanitized_orders_view;
- 2. Consuming Change Data Feed (CDF) Incrementally
When a table is shared WITH CHANGE DATA FEED , downstream consumer pipelines can read incremental inserts, updates, and deletes using Structured Streaming or batch table_changes() : -- Consumer queries table changes incrementally between commit versions
- SELECT
- order_id,
- order_amount,
- _change_type,
- _commit_version,
- _commit_timestamp
FROM table_changes('shared_global_sales.sales_data.transactions', 105, 120); # PySpark Structured Streaming consumer reading live updates from shared table
- df_stream = (spark.readStream
- .format("deltaSharing")
- .option("readChangeFeed", "true")
- .option("startingVersion", 100)
- .table("shared_global_sales.sales_data.transactions"))
- 6. Security, Auditing, & Delta Sharing Governance
Delta Sharing provides enterprise security and end-to-end auditability:
Audit Logging via system.access.audit : Every query executed by a recipient against a shared table is logged in the provider's Unity Catalog audit logs ( system.access.audit ). The logs record the recipient name, IP address, queried table, timestamp, and accessed partitions.
Direct Storage Authorization: When a consumer cluster queries a shared table, Databricks generates short-lived, down-scoped SAS URL tokens that allow the consumer's Spark executors to stream Parquet chunks directly from the provider's ADLS Gen2 storage over TLS without passing through third-party servers.
Instant Revocation: Executing REVOKE SELECT ON SHARE global_sales_share FROM RECIPIENT emea_analytics_recipient; terminates access immediately. All active queries on the consumer cluster fail upon their next batch token refresh.