Study guide
7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
Key Takeaways Explain This Topic AI
Lakeflow Connect managed connectors produce an ingestion pipeline that is governed by Unity Catalog, powered by serverless compute and Lakeflow pipelines, and writes into streaming tables.
Managed connector families are SaaS connectors, database connectors using change data capture, query-based connectors, file source connectors, streaming connectors, and community connectors.
Database CDC connectors add two components a SaaS connector does not have: an ingestion gateway that runs as a continuous task in its own job, and staging storage for captured changes.
Lakeflow Connect performs incremental ingestion - the first run loads all selected data and subsequent runs load only what changed - and adding a schedule automatically creates a Lakeflow Job with the pipeline as a task.
Choose Azure Data Factory when the requirement is cross-service orchestration or a source with no Databricks connector, notebooks with Auto Loader when files land in ADLS Gen2, and Lakeflow Connect when a managed connector exists for the source.
Last updated: August 2026
7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
DP-750 Exam Focus: Two blueprint bullets - "Choose an appropriate data ingestion tool, including Lakeflow Connect, notebooks, and Azure Data Factory" and "Ingest data by using Lakeflow Connect, including batch and streaming." Sections 7.1 through 7.5 taught the notebook-based paths. This section adds the managed path and gives the decision rule.
1. What Lakeflow Connect Is
Lakeflow Connect is Databricks' managed ingestion service. You configure a source and a destination; Databricks operates the pipeline. Every managed connector produces an ingestion pipeline that is:
- Governed by Unity Catalog - the destination is a Unity Catalog table, with the usual privilege model
- Powered by serverless compute and Lakeflow pipelines
- Incremental - efficient incremental reads and writes rather than full reloads
- Writing into streaming tables as its destination objects
- The Six Connector Families
- Family What it ingests
- SaaS connectors Enterprise SaaS applications including Salesforce, HubSpot, Jira, and Workday
Database connectors (CDC) Relational databases including MySQL, PostgreSQL, and SQL Server, using change data capture
Query-based connectors Databases queried directly on a schedule, without configuring CDC
File source connectors Structured and unstructured files from enterprise file storage such as Google Drive and SharePoint
- Streaming connectors Message buses and event streaming sources such as RabbitMQ
- Community connectors Open-source, community-built connectors for sources without managed support
If none of these fit, you can build a custom connector and run it in your workspace.
2. Component Architecture Differs by Connector Type
This is the detail most often tested, because it explains why a database connector costs more and has more moving parts than a SaaS connector.
Component SaaS Database (CDC) Query-based Streaming
Connection (Unity Catalog securable holding endpoint + credentials) Yes Yes Yes - either a direct Unity Catalog connection or a foreign catalog via Lakehouse Federation Yes
Ingestion pipeline writing to streaming tables Yes Yes Yes - queries the source on a schedule Yes - reads continuously
- Destination streaming tables Yes Yes Yes Yes
- Ingestion gateway No Yes - runs as a continuous task in its own job No No
- Staging storage No Yes - supports continuous change capture No No
A query-based connector is the lighter-weight database option: it queries the source directly on a schedule with no gateway and no staging storage , at the cost of not capturing every intermediate change the way CDC does.
3. Incremental Ingestion and Orchestration
Incremental behavior. On the first run, the pipeline ingests all selected data from the source while simultaneously beginning to track changes. Every subsequent run uses that change tracking to load only what changed since the prior run, where the source supports it. The exact mechanism depends on the source - SQL Server can use both change tracking and CDC, while the Salesforce connector selects a cursor column from a fixed list of options. Some sources and some individual tables do not support incremental ingestion at all.
Orchestration. You can run an ingestion pipeline on one or more custom schedules. For each schedule you add, Lakeflow Connect automatically creates a Job , with the ingestion pipeline as a task inside it. You can then add further tasks to that job - a silver transformation, a data quality check, a dashboard refresh. For database connectors, the ingestion gateway runs in its own job as a continuous task.
Networking. SaaS connectors call the source's APIs and are automatically compatible with serverless egress controls. Cloud database connectors can connect over Private Link , or the gateway can be deployed inside a VNet peered with the one hosting the database. On-premises databases connect through services such as Azure ExpressRoute .
Failure recovery. As a managed service, a connector automatically retries with exponential backoff . When an error requires intervention - expired credentials, for example - the connector stores the last cursor position so it can resume from that point rather than losing data.
Monitoring. Event logs, cluster logs, pipeline health metrics, and data quality metrics are all available, and system.billing.usage tracks connector cost.
4. Creating Connections and Deploying Pipelines as Code
Ingestion pipelines can be deployed with Declarative Automation Bundles (the bundle format also referred to on the exam as Databricks Asset Bundles), which brings source control, code review, testing, and CI/CD to ingestion - and lets the same definition target dev, staging, and prod workspaces.
For connectors that use API-only authentication (all database connectors and most SaaS connectors), connections can be created programmatically instead of through Catalog Explorer: # Databricks CLI - the JSON body matches the Connections REST API schema
- databricks connections create --json '{
- "name": "sqlserver_orders_prod",
- "connection_type": "SQLSERVER",
- "options": { "host": "orders-prod.database.windows.net", "port": "1433" }
- }'
The same call can be made from a notebook through the Connections API, or from a bundle pre-deploy script or job task before the pipeline is deployed.
The OAuth exception. Connectors whose only authentication option is browser-based OAuth (OAuth U2M) cannot be created programmatically, because the initial token requires interactive sign-in. Confluence, Google Ads, HubSpot, Jira, Meta Ads, Slack, Slack Audit Logs, TikTok Ads, and Zendesk Support fall into this group - a real constraint on fully automated CI/CD.
5. The Decision Rule
Scenario Correct tool Why
Salesforce, Workday, ServiceNow, or another supported SaaS source Lakeflow Connect SaaS connector Managed, incremental, serverless, Unity Catalog governed
SQL Server / PostgreSQL / MySQL replication with full change history Lakeflow Connect database CDC connector Gateway plus staging storage captures every change
The same databases, but periodic snapshots are sufficient and CDC cannot be enabled on the source Lakeflow Connect query-based connector No gateway, no staging storage, simpler and cheaper
Files landing continuously in ADLS Gen2 Notebook with Auto Loader (Section 7.1) cloudFiles incremental discovery, schema evolution, rescued data
One-off or periodic bulk load of files already in place COPY INTO or CTAS (Section 7.3) Idempotent, no streaming infrastructure
Azure Event Hubs or Kafka events Structured Streaming in a notebook or pipeline (Section 7.4) Full control over triggers, watermarks, and output modes
Ad hoc query of an operational database, no copy permitted Lakehouse Federation (Section 3.5) Read-only foreign catalog, no data movement
Orchestrating Databricks alongside other Azure services, or moving data from a source with no Databricks connector Azure Data Factory Broad connector catalog and cross-service control flow; ADF triggers the Databricks job
Source has neither a managed nor a community connector Custom connector , or a notebook using the Spark Data Source API Escape hatch
Lakeflow Connect Versus Lakehouse Federation
When a source supports both , Databricks recommends Lakeflow Connect if performance on higher data volumes and lower latency are priorities. Federation is positioned for ad hoc reporting and proof-of-concept access.
Where Azure Data Factory Still Belongs
ADF is named in the DP-750 audience profile, and the exam expects you to know its role rather than dismiss it:
Orchestration across services - copy from an on-premises system, then trigger a Databricks job, then refresh a Power BI dataset.
Sources Databricks does not connect to - ADF's connector catalog is broader than the managed connector list.
Existing investment - many enterprises already run ADF as the enterprise scheduler, with Databricks as the compute.
What ADF should not be used for is re-implementing transformation logic that belongs in Lakeflow Spark Declarative Pipelines, or polling a folder that Auto Loader would handle incrementally and more cheaply.
6. Exam Traps
"Lakeflow Connect writes to a Delta table you create first" is wrong. The destinations are streaming tables that the ingestion pipeline creates and owns.
A SaaS connector has no gateway. Only database CDC connectors add a gateway and staging storage.
Adding a schedule creates a Job, not just a cron entry. That job is where you attach downstream tasks.
Not every source is incremental. Some sources and tables do not yet support incremental ingestion, so a scenario stating a full reload each run is not automatically a misconfiguration.
Auto Loader is not part of Lakeflow Connect. Auto Loader is a cloud-file source you use in notebooks and declarative pipelines; Lakeflow Connect is the managed connector service.