Study guide
12.1 Databricks Git Folders & Source Control Integration
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
12.1 Databricks Git Folders & Source Control Integration
Key Takeaways Explain This Topic AI
Databricks Git Folders (formerly Repos) provide bidirectional source code synchronization between Databricks workspaces and enterprise Git providers including GitHub, Azure DevOps, GitLab, and Bitbucket.
Authentication is configured per user via Git Provider Personal Access Tokens (PAT) or Azure Active Directory / Microsoft Entra ID OAuth tokens for Azure DevOps repositories.
Git Folders support arbitrary modular files (.py, .sql, .yaml, .json, .sh), enabling standard Python module packaging and relative imports alongside traditional Databricks notebooks.
Automated CI/CD pipelines use the Repos/Git Folders REST API (/api/2.0/repos) to programmatically clone repositories, update branches, and checkout specific commit SHAs in target workspaces.
Enterprise lakehouse governance enforces trunk-based development with feature branches in developer workspaces and automated deployment to staging/production rather than manual branch editing in production.
Last updated: August 2026
12.1 Databricks Git Folders & Source Control Integration
In modern lakehouse engineering, data assets—including data pipelines, transformation notebooks, modular Python libraries, and workflow definitions—must adhere to strict software development lifecycle (SDLC) standards. Early cloud notebook environments suffered from code isolation, manual copy-pasting, lack of version history, and untracked changes in production.
Databricks Git Folders (formerly known as Databricks Repos) bridges this gap by embedding full Git version control directly into the Azure Databricks workspace. Git Folders allows data engineers to develop code collaboratively, maintain branch isolation, perform code reviews via pull requests (PRs), import modular helper packages, and trigger automated continuous integration and continuous deployment (CI/CD) pipelines.
1. Git Folders Architecture & Workspace File Integration
Databricks Git Folders is not merely a notebook viewer; it is a full local Git clone hosted inside the Databricks workspace control plane and synchronized with your remote Git repository. +-------------------------------------------------------------------------+
- | DATABRICKS GIT FOLDERS ARCHITECTURE |
- +-------------------------------------------------------------------------+
- | |
- | REMOTE GIT PROVIDER DATABRICKS WORKSPACE |
- | +--------------------+ +----------------------------------+ |
- | | Azure DevOps | | /Workspace/Users/alice@corp.com/ | |
- | | GitHub Enterprise | <---HTTPS---| └── feature-sales-pipeline/ | |
- | | GitLab / Bitbucket | Sync | ├── config.yaml | |
- | +--------------------+ | ├── main_etl.py | |
- | | ├── transforms/ | |
- | | │ └── cleanse.py | |
- | | └── tests/ | |
- | | └── test_cleanse.py | |
- | +----------------------------------+ |
- +-------------------------------------------------------------------------+
- Workspace Files & Modular Python Imports
Historically, Databricks notebooks could only reference other notebooks via the legacy %run magic command. Git Folders introduces Arbitrary Workspace Files , allowing non-notebook files ( .py , .sql , .yaml , .json , .csv , .sh , .txt ) to reside alongside notebooks.
This enables standard Python software engineering patterns within notebooks and jobs: # File: /Workspace/Users/alice@corp.com/feature-sales-pipeline/main_etl.py
- import sys
- import yaml
- from transforms.cleanse import deduplicate_transactions, mask_pii_columns
- from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Load local YAML configuration file directly from Git Folder
with open("config.yaml", "r") as f:
config = yaml.safe_load(f)
- # Ingest raw Bronze Delta table
- raw_df = spark.table(f"{config['catalog']}.bronze.raw_transactions")
- # Apply modular business transformation functions
- clean_df = deduplicate_transactions(raw_df, key_col="transaction_id")
- silver_df = mask_pii_columns(clean_df, cols=["credit_card", "ssn"])
- # Persist to Silver layer
- silver_df.write.format("delta").mode("append").saveAsTable(f"{config['catalog']}.silver.transactions")
When a notebook or Python script executes within a Git Folder, Databricks automatically adds the root directory of the Git Folder to the Python sys.path . Consequently, custom packages and relative module imports function seamlessly without requiring manual sys.path.append() statements or wheel installations.
2. Git Provider Authentication Mechanisms
To synchronize code between Azure Databricks and a remote Git repository, each engineer or service principal must configure Git credentials.
Authentication Method Supported Providers Mechanism & Characteristics
Personal Access Token (PAT) GitHub, GitLab, Bitbucket, Azure DevOps User generates a scoped access token in the Git provider and pastes it into User Settings > Linked Accounts > Git Integration . Requires token rotation upon expiry.
Microsoft Entra ID OAuth (Azure DevOps) Azure DevOps Repos Native single-sign-on (SSO) integration. When Azure Databricks and Azure DevOps share the same Entra ID tenant, authentication occurs automatically using the user's Entra ID token without creating long-lived PATs.
Service Principal Authentication Azure DevOps, GitHub Actions Used for automated CI/CD runners. A Microsoft Entra Service Principal or GitHub App interacts with the Repos REST API using client credentials or OpenID Connect (OIDC). +-------------------------------------------------------------------------+
- | AUTHENTICATION WORKFLOW: AZURE DEVOPS & ENTRA ID |
- +-------------------------------------------------------------------------+
- | 1. Developer logs into Azure Databricks via Entra ID (Azure AD). |
- | 2. In User Settings > Git Integration, select 'Azure DevOps Services'. |
- | 3. Select 'Azure DevOps OAuth' (or provide PAT with Code Read/Write). |
- | 4. Databricks securely requests an OAuth token from Entra ID. |
- | 5. Workspace clones/pushes to https://dev.azure.com/org/project/_git/ |
- +-------------------------------------------------------------------------+
Exam Tip : For Azure DevOps repositories residing in the same Microsoft Entra ID tenant as Azure Databricks, Microsoft recommends using native Entra ID OAuth integration. This eliminates secret sprawl and avoids pipeline disruptions caused by expired Personal Access Tokens.
3. Branching Strategies & Lakehouse Lifecycle Management
Enterprise data teams implement disciplined branching workflows to prevent untested code from polluting staging and production lakehouses. +-------------------------------------------------------------------------+
- | TRUNK-BASED LAKEHOUSE BRANCHING MODEL |
- +-------------------------------------------------------------------------+
- | |
- | [main branch] ====================================================> |
- | \ / (PR Merge) |
- | \--> [feat/ingest-telemetry] (Alice's Git Folder) -/ |
- | * Local testing against dev catalog |
- | * Automated PR checks (pytest, linting) |
- | |
- +-------------------------------------------------------------------------+
- Comparison: Git Flow vs. Trunk-Based Development
Strategy Structure Characteristics in Lakehouse Engineering
Git Flow Long-lived main , develop , release , and hotfix branches. Complex merge overhead; frequent merge conflicts across large ETL pipelines. Not ideal for continuous delivery.
Trunk-Based (Recommended) Short-lived feature branches ( feat/name ) branched directly off main . Developers merge small, frequent commits into main via PRs. Automated CI/CD deploys main to Staging and Production workspaces. Minimizes drift and merge collisions.
Developer vs. Production Git Folder Topologies
Developer Workspace Path ( /Workspace/Users/<email>/ ) :
Each data engineer clones the repository into their personal workspace folder.
Engineers switch branches, modify files, run interactive notebook cells against development compute, commit changes, and push to the remote repository.
Isolation: Changes made by Engineer A never affect Engineer B.
Shared Production Workspace Path ( /Workspace/Repos/Production/ or DAB-managed) :
Production workspaces should never be manually edited by individual developers.
Production Git Folders are maintained on the main or release tag branch and updated exclusively via automated CI/CD service principals.
4. Programmatic Management with the Repos REST API
Azure Databricks provides the Repos REST API ( /api/2.0/repos ) to allow automated CI/CD pipelines, orchestrators, and release agents to manage Git Folders programmatically.
Core Repos REST API Endpoints
HTTP Method Endpoint Description
POST /api/2.0/repos Clones a remote repository into a specified workspace path.
GET /api/2.0/repos/{repo_id} Retrieves metadata for a Git Folder (current branch, head commit, URL).
PATCH /api/2.0/repos/{repo_id} Updates an existing Git Folder to a target branch, tag, or commit SHA.
DELETE /api/2.0/repos/{repo_id} Deletes a Git Folder from the workspace.
Automating Deployment via REST API
When a Pull Request is approved and merged into main , a CI/CD pipeline (such as Azure DevOps Pipelines or GitHub Actions) executes a PATCH request to pull the latest code into the staging/production Git Folder: # Triggered by CI/CD Runner authenticated with Databricks Service Principal Token
- curl -X PATCH \
- https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/repos/987654321098765 \
- H "Authorization: Bearer ${DATABRICKS_SP_TOKEN}" \
- H "Content-Type: application/json" \
- d '{
- "branch": "main"
- }'
- JSON Response: {
- "head_commit_id": "e4d9b2a1c0f8e7d6a5b4c3d2e1f0a9b8c7d6e5f4",
- "id": 987654321098765,
- "path": "/Repos/Production/lakehouse-etl",
- "url": "https://dev.azure.com/myorg/dataplat/_git/lakehouse-etl",
- "provider": "azureDevOpsServices",
- "branch": "main"
- }
- Checking Out Specific Release Tags
For production releases requiring strict immutability, the CI/CD pipeline can update the Git Folder to a specific immutable release tag rather than a floating branch: curl -X PATCH \
- https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/repos/987654321098765 \
- H "Authorization: Bearer ${DATABRICKS_SP_TOKEN}" \
- H "Content-Type: application/json" \
- d '{
- "tag": "v2.4.0"
- }'
- 5. Storage Layer Comparison & Operational Constraints
Understanding where code and data reside in Azure Databricks is critical for architecture design and exam questions:
Storage / Location Purpose Version Controlled? Access / Security
Git Folders ( /Workspace/Repos/ ) Source code, scripts, modular Python packages, config files, unit tests. Yes (Remote Git Provider) Governed by Git permissions and Databricks Workspace ACLs.
Workspace Files ( /Workspace/Users/ ) Ad-hoc notebooks, temporary scratch files, experiments. No (Unless inside a Git Folder) Databricks Workspace ACLs.
Unity Catalog Volumes Unstructured / semi-structured data files, ML models, raw landing files. No (Data storage) Unity Catalog 3-level namespace privilege model ( READ VOLUME , WRITE VOLUME ).
DBFS (Legacy FileStore) Legacy staging, cluster init scripts (deprecated). No Root storage (avoid for sensitive data and code).
Git Folders Constraints & Limitations
File Size Limits : Individual non-notebook files inside Git Folders cannot exceed 500 MB.
Git Submodules : Databricks Git Folders does not support Git submodules.
Git LFS (Large File Storage) : Git LFS pointers are not resolved; store large binary files in Unity Catalog Volumes instead.
Merge Conflicts : Merge conflicts cannot be resolved in the Databricks web UI; developers must resolve conflicts locally using their IDE/Git CLI and push resolved commits.