ExamFinch.com My Study

Study guide

3.3 Ingesting Data via Delta Sharing & External Connections

Practice this exam

Databricks Certified Data Analyst Associate Exam

Track your progress as you read

Finished sections get a check mark.

1 Platform Overview & Unity Catalog Foundations

1.1 Databricks Architecture & Compute Concepts 1.2 Databricks Marketplace & Data Sharing Basics 1.3 Unity Catalog Namespace & Object Hierarchy 1.4 Governed Asset Discovery & System Tables

2 Managing & Cleaning Governed Data

2.1 Dataset Discovery & Asset Tagging 2.2 Lineage Inspection & Data Traceability 2.3 SQL Data Cleaning & Transformation Techniques

3 Data Ingestion & Import Patterns

3.1 Workspace Uploads & Cloud File Ingestion 3.2 Incremental File Ingestion with Auto Loader 3.3 Ingesting Data via Delta Sharing & External Connections

4 Databricks SQL Warehouses & Core Querying

4.1 Databricks SQL Interface & SQL Assistant 4.2 SQL Warehouses Sizing & Management 4.3 Table Creation & Data Modification in Databricks SQL

5 Advanced SQL Authoring & Time Travel

5.1 Advanced Aggregations, Window Functions & Analytics 5.2 Complex Joins, CTEs & Set Operations 5.3 Views, Materialized Views & Delta Time Travel

6 Query Performance Analysis & Photon Engine

6.1 Photon Engine Architecture & Acceleration 6.2 Query History & Execution Profile Inspection 6.3 Delta Lake Audit History & Commit Logs

7 Liquid Clustering, Caching & Performance Troubleshooting

7.1 Caching Hierarchy in Databricks SQL 7.2 Data Layout Optimization: Liquid Clustering vs Partitioning & Z-Ordering 7.3 Troubleshooting Slow & Failing Queries

8 AI/BI Dashboards & Visualization Design

8.1 AI/BI Dashboards Overview & Canvas Layout 8.2 Chart Types & Visualization Selection 8.3 Notebook & SQL Editor Visualizations

9 Dashboard Interactivity, Publishing & Alerts

9.1 Dashboard Parameters & Interactive Filtering 9.2 Publishing, Sharing & Access Control 9.3 Scheduled Refresh, Email Subscriptions & Alerts

10 Developing & Maintaining AI/BI Genie Spaces

10.1 AI/BI Genie Architecture & Space Setup 10.2 Curation via Instructions, Descriptions & Sample Questions 10.3 Trusted Assets & Parameterized Functions 10.4 Monitoring, Benchmarks, Feedback & Maintenance

11 Data Modeling & Medallion Architecture

11.1 Dimensional Modeling Fundamentals 11.2 Medallion Architecture for Analytics 11.3 Constraint Enforcement & Relationships in Unity Catalog

12 Securing Data & Unity Catalog Governance

12.1 Unity Catalog Privilege Model & Grants 12.2 Ownership, Inheritance & Access Control Patterns 12.3 Fine-Grained Security: Row Filters & Column Masks

3.3 Ingesting Data via Delta Sharing & External Connections

Key Takeaways Explain This Topic AI

Delta Sharing is an open protocol providing secure, real-time data sharing across cloud platforms without copying or moving underlying data files.

Databricks-to-Open Sharing allows recipients outside Databricks to authenticate using bearer tokens in credential files, accessing data via Power BI, pandas, Spark, or Tableau.

Lakehouse Federation enables analysts to execute federated SQL queries directly against external databases like PostgreSQL, MySQL, Snowflake, and BigQuery without ETL.

In Unity Catalog Lakehouse Federation, Foreign Catalogs map external database schemas directly into Unity Catalog's three-level namespace as read-only objects.

Query pushdown optimization minimizes network overhead by executing filters, projections, and aggregations directly on the remote database engine.

Last updated: July 2026

3.3 Ingesting Data via Delta Sharing & External Connections

In traditional data architecture, acquiring external datasets or integrating with remote transactional databases required complex Extract, Transform, Load (ETL) pipelines. These pipelines created physical data copies, leading to data staleness, ballooning cloud storage costs, security vulnerabilities, and governance complexity. Modern data intelligence platforms solve these issues by introducing zero-copy data sharing and lakehouse federation . Databricks provides Delta Sharing for secure cross-organization data exchange and Lakehouse Federation for querying external database systems directly in-place.

Zero-Copy Data Access Paradigm

The zero-copy data access model eliminates data movement entirely. Instead of copying gigabytes or terabytes of data across cloud buckets or database instances, analysts query remote datasets live at the source. Traditional ETL Model:

  • Source DB / Provider ---> [Extract File] ---> [Network Transit] ---> [Load to Lakehouse] ---> Query Data
  • (High latency, duplicate storage costs, stale data)

Zero-Copy Sharing & Federation Model:

  • Source DB / Provider <------------------- Live Direct Query ------------------- Analyst Workspace
  • (Zero storage duplication, real-time access, centralized governance)

Key advantages of zero-copy architectures include:

Immediate Data Freshness : Queries reflect source updates instantaneously without waiting for batch ETL completion.

Zero Storage Overhead : Recipients do not pay cloud storage fees for duplicated data.

Centralized Security : Providers maintain complete control and can revoke access instantly.

Delta Sharing Protocol & Architecture

Delta Sharing is an open-source protocol developed by Databricks and donated to the Linux Foundation . It enables secure real-time data sharing across organizations regardless of the underlying cloud platform, computing framework, or data storage engine.

Open Sharing vs. Databricks-to-Databricks Sharing

Delta Sharing operates in two primary modes:

Databricks-to-Databricks Sharing : When both provider and recipient use Databricks with Unity Catalog, sharing occurs natively using Unity Catalog Metastore IDs . No credential files or bearer tokens are exchanged. Databricks handles authentication seamlessly, allowing recipients to mount shared data as Unity Catalog catalogs instantly.

Databricks-to-Open Sharing : When recipients use non-Databricks platforms (such as Microsoft Power BI, Tableau, Python pandas, Apache Spark, or Excel), access is governed through a secure open REST API. The provider generates a credential file ( .share ) containing a short-lived bearer token . The recipient's application uses the token to authenticate with the Delta Sharing server, which returns pre-signed cloud storage URLs for reading underlying Parquet/Delta data files directly from cloud storage.

Governed Data Sharing in Unity Catalog

Unity Catalog serves as the control plane for Delta Sharing, organizing sharing entities into a structured hierarchy:

Share : A logical container created by a data provider in Unity Catalog that groups read-only assets to be shared. A share can contain Delta tables, materialized views, standard views, dynamic views, and Unity Catalog Volumes.

Provider : An entity representing the data provider who creates shares and manages recipient permissions.

Recipient : An entity representing an external consumer authorized to access specific shares.

Shared Object Capabilities

Providers can share complex assets beyond raw tables:

Views with Row Filters & Column Masks : Providers can enforce fine-grained access control before data leaves their metastore, ensuring recipients only see records permitted by row predicates or masked columns.

Unity Catalog Volumes : Allows sharing non-tabular datasets, such as machine learning model checkpoints, image files, or PDF documents.

Analyst Workflow for Accessing Shared Data

To consume a Delta Share, a Databricks data analyst inspects available shares and mounts them directly into their local Unity Catalog namespace: -- View shares published by an external provider

SHOW SHARES IN PROVIDER external_vendor_provider;

  • - Create a local catalog mapped to the external share
  • CREATE CATALOG vendor_analytics_cat
  • USING SHARE external_vendor_provider.market_intelligence_share;
  • - Query shared tables live with zero copy overhead
  • SELECT
  • region,
  • product_category,
  • total_market_revenue
  • FROM vendor_analytics_cat.market_data.regional_summaries
  • WHERE summary_year = 2026;
  • Lakehouse Federation for External Databases

While Delta Sharing connects organizations across lakehouses, Lakehouse Federation enables data analysts to query external relational database management systems (RDBMS) and data warehouses directly from Databricks SQL Warehouses without moving data.

Lakehouse Federation supports connectivity to popular database engines including PostgreSQL , MySQL , Snowflake , Amazon Redshift , Google BigQuery , Microsoft SQL Server , and Oracle .

Unity Catalog Federation Entities

Lakehouse Federation integrates external database systems into Unity Catalog's 3-level namespace using two key abstractions:

Connection : A Unity Catalog object that encapsulates connection details and credentials (hostname, port, username, password/secrets) for an external database system.

Foreign Catalog : A Unity Catalog catalog object that maps directly to a database inside the remote system. All schemas, tables, and views in the remote database become visible inside Unity Catalog as read-only objects under foreign_catalog.schema_name.table_name . -- Step 1: Create connection to remote PostgreSQL database

  • CREATE CONNECTION postgres_prod_db
  • TYPE POSTGRESQL
  • OPTIONS (
  • host 'postgres-prod.internal.net',
  • port '5432',
  • user secret('scope_credentials', 'pg_user'),
  • password secret('scope_credentials', 'pg_pass')
  • );
  • - Step 2: Create Foreign Catalog mapping to remote database
  • CREATE FOREIGN CATALOG pg_sales_catalog
  • USING CONNECTION postgres_prod_db
  • OPTIONS (database 'sales_production');
  • - Step 3: Query remote database directly inside Databricks SQL
  • SELECT customer_id, order_status, total_amount
  • FROM pg_sales_catalog.public.orders
  • WHERE order_date >= '2026-01-01';
  • Query Pushdown Optimization

A key engineering challenge of federated queries is network latency and data transfer overhead. Lakehouse Federation solves this through Query Pushdown Optimization .

When an analyst runs a SQL query against a Foreign Catalog, the Databricks SQL query optimizer evaluates the query execution plan and pushes filtering ( WHERE ), projections ( SELECT ), joins, and aggregations ( GROUP BY ) down to the source database engine. Databricks SQL Warehouse External Database (PostgreSQL / Snowflake)

  • ├── Executes Query ------------------------> Receives Pushed-Down SQL Query
  • │ ├── Executes filter & aggregation locally
  • └── Receives Filtered Result Set <-----------└── Returns small aggregated summary table

By processing filters and aggregations at the source engine, Lakehouse Federation transfers only the compact final result set across the network, minimizing latency and compute costs.

Comparative Matrix: Ingestion & Sharing Patterns

Analysts must select the appropriate integration strategy based on data location, update frequency, and architecture:

  • Feature / Metric Delta Sharing Lakehouse Federation COPY INTO / Auto Loader
  • Data Movement Zero-copy (Direct cloud read) Zero-copy (Live federated queries) Batch or streaming file copy
  • Target Storage Provider cloud storage External database engine Local Delta Lake tables
  • Data Freshness Real-time (Instant read) Real-time (Live transactional read) Micro-batch / scheduled latency
  • Primary Use Case Cross-org & partner data sharing Operational database reporting Ingesting raw landing files

Governance Entity Unity Catalog Shares & Recipients Connections & Foreign Catalogs External Locations & Volumes