Study guide
6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks
Key Takeaways Explain This Topic AI
Delta Sharing is the industry's first open, vendor-neutral protocol for secure real-time data sharing across clouds, regions, and data platforms without copying data.
In Open Delta Sharing, external recipients authenticate using a temporary bearer token embedded in a secure credential profile JSON file, querying shared data via REST APIs using tools like Power BI, pandas, Apache Spark, and DuckDB.
In Databricks-to-Databricks (Managed) Sharing, recipients authenticate natively using their Unity Catalog Metastore Global Sharing Identifier, enabling zero-credential token management and native catalog mounting.
Data providers manage access through Unity Catalog SHARE and RECIPIENT securable objects, with granular support for partition filtering, column masking, dynamic views, volumes, and Change Data Feed (CDF).
Delta Sharing avoids compute bottlenecking by having the provider's Unity Catalog metastore vend short-lived presigned cloud storage URLs (Azure SAS tokens), allowing recipients to stream Parquet data directly from the provider's ADLS Gen2 storage.
Last updated: August 2026
6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks
DP-750 Exam Focus: Understand the open Delta Sharing protocol architecture, contrasting Open Sharing (credential profile JSON, bearer tokens, REST API for Power BI/pandas/Spark) with Databricks-to-Databricks (Managed) Sharing (native Metastore Global Sharing ID, zero-token management). Master the SQL lifecycle for CREATE SHARE , CREATE RECIPIENT , partition filtering, Change Data Feed sharing, and security governance (token expiration, IP access lists, and audit logging).
1. The Delta Sharing Paradigm & Zero-Copy Architecture
Traditional data sharing across organizations or business units relied on inefficient, error-prone patterns:
Nightly SFTP / Cloud Storage Drops: High ETL maintenance, duplicate cloud storage costs, and severe data staleness.
Vendor-Locked Sharing: Proprietary data sharing platforms require both provider and consumer to license the identical closed database software.
Direct Cloud IAM Delegation: Sharing cloud storage containers directly via cross-account Azure IAM/SAS tokens introduces high security risk and exposes all unmanaged files in the bucket.
Delta Sharing is an open-source, vendor-neutral standard developed by Databricks and the Linux Foundation. It enables organizations to share live, read-only Delta Lake tables, views, volumes, and models across any cloud (Azure, AWS, GCP), region, or client platform without data copying or replication . +-----------------------------------------------------------------------------------+
- | DELTA SHARING CORE PROTOCOL FLOW |
- +-----------------------------------------------------------------------------------+
- | |
- | [ Recipient Client ] [ Provider Unity Catalog ] |
- | (Power BI / Pandas / Spark) (Governance Metastore) |
- | | | |
- | | 1. HTTPS Query + Bearer Token / ID | |
- | |-------------------------------------->| |
- | | | (Validates permissions & |
- | | | partition filters) |
- | | 2. Returns Parquet Metadata + | |
- | | Short-Lived Azure SAS URLs | |
- | |<--------------------------------------| |
- | | |
- | | 3. Direct Read via TLS / ABFSS Driver |
- | +---------------------------------------------------+ |
- | | |
- | v |
- | [ Provider ADLS Gen2 Storage ] |
- | (Underlying Parquet Data Files) |
- +-----------------------------------------------------------------------------------+
- Why Delta Sharing Does Not Suffer Compute Bottlenecks
In Delta Sharing, the provider's compute clusters are not involved in streaming data to the consumer .
The recipient makes an HTTPS request to the provider's Unity Catalog server.
Unity Catalog validates authorization, evaluates partition filters, and generates short-lived presigned Azure SAS URLs (typically valid for 15 minutes to 1 hour) for the specific Parquet files needed.
The recipient client uses these URLs to read data directly from the provider's Azure Data Lake Storage Gen2 account over high-throughput cloud network backbones. The provider incurs zero query compute costs.
2. Open Delta Sharing (External Non-Databricks Consumers)
Open Delta Sharing enables external recipients who do not use Azure Databricks to consume live data directly within their preferred analytical environments (e.g., Microsoft Power BI, Python Pandas, Apache Spark, DuckDB, Tableau, Excel, or custom REST applications). OPEN DELTA SHARING LIFECYCLE
- 1. Provider Creates Recipient
- +-------------------------------------------------------------------------+
- | CREATE RECIPIENT external_partner; |
- | (Generates one-time activation URL with secure token download) |
- +-------------------------------------------------------------------------+
- |
- v
- 2. Consumer Downloads Credential Profile File (`config.share`)
- +-------------------------------------------------------------------------+
- | { |
- | "shareCredentialsVersion": 1, |
- | "bearerToken": "dapi891240abc98412...", |
- | "endpoint": "https://eastus.azuredatabricks.net/api/2.0/delta-sharing/"|
- | "expirationTime": "2027-01-01T00:00:00Z" |
- | } |
- +-------------------------------------------------------------------------+
- |
- v
- 3. Client Consumes Data Programmatically
- +-------------------------------------------------------------------------+
- | import delta_sharing |
- | profile = "/path/to/config.share" |
- | df = delta_sharing.load_as_pandas(f"{profile}#share_sales.gold.orders") |
- +-------------------------------------------------------------------------+
- Open Sharing Client Ecosystem # Consuming Delta Sharing via Apache Spark
- spark_df = (spark.read.format("deltaSharing")
- .load("/secrets/config.share#partner_share.analytics.quarterly_revenue"))
- # Consuming Delta Sharing via Python Pandas
- import delta_sharing
- client = delta_sharing.SharingClient("/secrets/config.share")
- shares = client.list_shares()
- pandas_df = delta_sharing.load_as_pandas("/secrets/config.share#partner_share.analytics.quarterly_revenue")
- Security Controls for Open Sharing
One-Time Activation URL: The activation URL can only be accessed once to download the credential profile JSON file. Once downloaded, the activation link expires permanently.
Token Rotation & Expiration: Providers configure strict token expiration horizons (e.g., 30, 60, or 90 days) using ALTER RECIPIENT ... SET TOKEN_EXPIRATION_DAYS = 90; and can rotate tokens instantly.
IP Access Lists: Providers restrict recipient queries to approved corporate CIDR IP blocks: ALTER RECIPIENT external_partner SET IP_ACCESS_LIST ('198.51.100.0/24', '203.0.113.50/32');
3. Databricks-to-Databricks (Managed) Delta Sharing
When both the data provider and consumer use Azure Databricks with Unity Catalog , they utilize Databricks-to-Databricks Managed Sharing . DATABRICKS-TO-DATABRICKS MANAGED SHARING
- [ Provider Unity Catalog Metastore ] [ Consumer Unity Catalog Metastore ]
- Metastore ID: azure:eastus:prov-uuid Metastore ID: azure:westus:cons-uuid
- | |
- +---------------------------------------------+
- |
- (Direct UC-to-UC Trust Federation)
- No Credential JSON Files
- Zero Bearer Token Management
- Shares Mount Natively as Catalogs
- The Metastore Global Sharing Identifier
In Databricks-to-Databricks sharing, no credential profile files or bearer tokens are created. Authentication occurs via mutual cryptographic verification between Unity Catalog metastores using the recipient's Global Sharing Identifier :
Sharing Identifier Format: <cloud>:<region>:<metastore-uuid> Example: azure:eastus:8f7e2a10-3b4c-4d5e-9f0a-1a2b3c4d5e6f
Creating a Managed Recipient -- Provider executes in Unity Catalog:
- CREATE RECIPIENT enterprise_subsidiary
- USING ID 'azure:westus:8f7e2a10-3b4c-4d5e-9f0a-1a2b3c4d5e6f';
- Consumer Mounting Shared Catalogs Natively
Once the provider grants access to a share, the recipient's Metastore Admin mounts the share as a native catalog inside their workspace: -- Consumer Metastore Admin executes in recipient workspace:
- CREATE CATALOG shared_finance_hub
- USING SHARE provider_org_metastore.finance_share;
-- Consumer users can now query shared tables using standard 3-level namespace!
- SELECT * FROM shared_finance_hub.gold.monthly_pnl;
- 4. Comparison Matrix: Open Sharing vs. Databricks-to-Databricks
Architectural Dimension Open Delta Sharing Databricks-to-Databricks Managed Sharing
Target Consumer External tools (Power BI, pandas, Spark, DuckDB, Excel) Databricks workspaces with Unity Catalog
Authentication Protocol Bearer Token in JSON Credential Profile ( config.share ) Metastore Global Sharing Identifier ( azure:region:uuid )
Token Maintenance Manual download, rotation, and expiration policies Zero Token Management (Managed automatically by UC)
Shared Asset Types Delta Tables, Table Partitions, Volumes (Files) Delta Tables, Dynamic Views , Volumes , ML Models , Notebooks
Delta Change Data Feed Supported via REST API Supported natively ( table_changes() syntax)
Namespace Integration Path-based reference ( profile#share.schema.table ) Native 3-Level Namespace Catalog ( catalog.schema.table )
Security Enforcement Token expiry, IP Access Lists, TLS encryption UC RBAC, ABAC, IP Access Lists, Cross-Metastore TLS
5. Provider-Side Object Lifecycle & Governance
To share data securely, data providers create and manage three core Unity Catalog securables: SHARE , RECIPIENT , and PROVIDER . -- Step 1: Create the SHARE object container
- CREATE SHARE regional_sales_share
- COMMENT 'Live sales data for regional retail partners';
- - Step 2: Add tables to the share with optional partition filtering
- ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.orders
- COMMENT 'Consolidated global orders';
- - Step 2b: Add a table with partition filtering (Restricting to US and CA regions)
- ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.regional_inventory
- AS prod_sales.gold.regional_inventory
- WITH (PARTITION (country = 'US'), PARTITION (country = 'CA'));
- - Step 2c: Share Delta Change Data Feed (CDF) for incremental synchronization
- ALTER SHARE regional_sales_share ADD TABLE prod_sales.gold.pricing_updates
- WITH CHANGE DATA FEED;
- - Step 3: Grant SELECT on the SHARE to the RECIPIENT
- GRANT SELECT ON SHARE regional_sales_share TO RECIPIENT external_partner;
- Auditing Shared Data Consumption
Every data read request initiated by Delta Sharing recipients is logged in system.access.audit under service_name = 'deltaSharing' , recording recipient identifiers, IP addresses, accessed tables, and byte transfer volumes.