Study guide
2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts
Key Takeaways Explain This Topic AI
Notebook-scoped libraries installed via %pip or %conda execute in session-isolated Python environments without requiring cluster restarts, ideal for ad-hoc exploration and interactive development.
Compute-scoped (cluster-level) libraries are installed across the driver and all worker nodes upon cluster startup, ensuring uniform dependency availability across all users and production jobs sharing the cluster.
Cluster init scripts execute as root during node initialization before the Apache Spark JVM boots; modern governance mandates storing init scripts in Unity Catalog Volumes instead of legacy DBFS paths.
Init script execution follows a strict deterministic hierarchy: Global Workspace Init Scripts execute first, followed by cluster-scoped init scripts in their explicitly configured order.
Last updated: August 2026
Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts
Data engineering pipelines in Azure Databricks rely heavily on external Python packages, Java/Scala JARs, and custom OS-level drivers. Managing these dependencies requires selecting the appropriate installation scope to prevent version conflicts, ensure security isolation, and avoid unnecessary compute restarts.
1. Library Scoping Models in Azure Databricks
Azure Databricks provides four distinct scoping tiers for library installation: +-----------------------------------------------------------------------------------------+
- | LIBRARY SCOPING HIERARCHY |
- +-----------------------------------------------------------------------------------------+
- | 1. WORKSPACE LIBRARIES | Stored in Workspace UI / Volumes; available for attachment |
- | | to multiple clusters across the organization. |
- +---------------------------+-------------------------------------------------------------+
- | 2. COMPUTE-SCOPED | Installed on Driver & all Workers during cluster launch. |
- | (CLUSTER-LEVEL) | Uniform across all notebooks; requires restart to modify. |
- +---------------------------+-------------------------------------------------------------+
- | 3. NOTEBOOK-SCOPED | Installed dynamically via %pip / %conda in REPL session. |
- | | Session-isolated; no cluster restart; resets on disconnect. |
- +---------------------------+-------------------------------------------------------------+
- | 4. TASK / JOB-SCOPED | Defined in Lakeflow Jobs DAG task configuration. |
- | | Automatically installed on ephemeral job compute. |
- +-----------------------------------------------------------------------------------------+
- Comprehensive Scope Comparison
- Feature Workspace Libraries Compute-Scoped Libraries Notebook-Scoped ( %pip ) Task-Scoped (Jobs)
- Target Workspace Repository Specific Compute Resource Active Notebook Session Specific Workflow Task
- Availability Available to attach All users on the cluster Isolated to current user Isolated to job task
- Installation Lifecycle Manual attachment Node boot / restart Dynamic at cell runtime Ephemeral cluster start
- Restart Required? Yes, if attached to compute Yes No (Zero Downtime) No
Storage Location UC Volumes / Maven / PyPI UC Volumes / CRAN / PyPI PyPI / Wheel / Repo UC Volumes / PyPI / JAR
Best For Organization-wide standards Core team shared dependencies Fast prototyping & ML testing Production orchestration
2. Deep Dive: Notebook-Scoped %pip Management
Notebook-scoped libraries allow developers to install, update, and manage Python packages dynamically within an active notebook session without restarting the cluster or impacting other data engineers working on the same shared cluster.
How %pip Operates Internally
When you execute %pip install <package> in a Databricks notebook cell:
The Python package is downloaded and installed into the driver node's session-isolated virtual environment.
Databricks automatically generates a serialized environment snapshot and broadcasts the required wheels/binaries to all active worker nodes.
The Python REPL process updates its sys.path dynamically.
Spark workers automatically load the library into their executor Python worker daemons. # Command Cell 1: Installing notebook-scoped dependencies
%pip install azure-storage-blob==12.19.0 azure-identity==1.15.0 --quiet
- # Command Cell 2: Installing from requirements file stored in a UC Volume
- %pip install -r /Volumes/dev_catalog/data_eng/config_volume/requirements.txt
- # Command Cell 3: Verifying active environment packages
- %pip list | grep azure
- Critical %pip Rules for the DP-750 Exam
First Cell Placement: Always place %pip installation commands in the very first command cell of the notebook. Running %pip after Python code has already initialized custom classes or C-extensions can lead to classloader conflicts.
Session Isolation: If Engineer A installs pandas==1.5.0 via %pip in Notebook 1, and Engineer B installs pandas==2.1.0 via %pip in Notebook 2 on the same shared cluster, both engineers execute in complete isolation with zero dependency collision.
Restart Persistence: Notebook-scoped libraries do not persist after the cluster terminates or when the notebook is detached. For automated production jobs, define libraries at the Job/Task level or cluster level.
3. Cluster Init Scripts Architecture & Evolution
An Init Script (initialization script) is a shell script ( .sh ) that executes during the startup phase of each cluster node—running on the underlying Ubuntu virtual machine before the Spark Driver and Worker JVM daemons launch. +-----------------------------------------------------------------------------------------+
- | CLUSTER NODE BOOTSTRAP & INIT TIMELINE |
- +-----------------------------------------------------------------------------------------+
- | 1. Cloud VM Provisioning (ARM allocates VM, attaches VNet NIC, mounts OS disk) |
- | |
- | v
- | 2. Base Container Launch (Ubuntu Linux container initializes) |
- | |
- | v
- | 3. GLOBAL WORKSPACE INIT SCRIPTS EXECUTE (Configured by Workspace Admins) |
- | |
- | v
- | 4. CLUSTER-SCORED INIT SCRIPTS EXECUTE (Configured in Compute UI / API in order) |
- | - Runs as root user |
- | - Installs OS packages (apt-get), custom certificates, ODBC drivers, monitoring |
- | |
- | v
- | 5. Spark JVM & Photon Daemons Start (SparkSession initialized, cluster enters RUNNING) |
- +-----------------------------------------------------------------------------------------+
- Common Use Cases for Init Scripts
Installing system-level Linux utilities and packages via apt-get install (e.g., unixodbc-dev , libpq-dev ).
Installing enterprise network security certificates (custom SSL CA certificates for private enterprise firewalls).
Configuring custom JVM parameters, security daemons, or enterprise APM monitoring agents (Datadog, Dynatrace, New Relic).
Configuring custom Linux environment variables ( export ENV_VAR=value ).
Unity Catalog Volumes vs. Legacy DBFS Storage
Historically, init scripts were stored in the legacy root DBFS ( dbfs:/databricks/init_scripts/ or dbfs:/FileStore/ ). In modern Azure Databricks environments governed by Unity Catalog, storing init scripts in root DBFS is deprecated and strongly discouraged due to security vulnerabilities (lack of fine-grained access control, unencrypted storage, and broad workspace read access). +-----------------------------------------------------------------------------------------+
- | INIT SCRIPT STORAGE SECURITY EVOLUTION |
- +-----------------------------------------------------------------------------------------+
- | LEGACY DBFS (DEPRECATED) | UNITY CATALOG VOLUMES (BEST PRACTICE) |
- | - Stored in dbfs:/databricks/init_scripts/ | - Stored in /Volumes/catalog/schema/volume/ |
- | - No fine-grained access control | - Controlled via UC 'READ VOLUME' privilege |
- | - Visible to all workspace users | - Fully audited via UC system tables |
- | - High risk of privilege escalation | - Centralized governance & encryption |
- +-----------------------------------------------------------------------------------------+
- Configuring an Init Script from a Unity Catalog Volume
- Upload the shell script to a governed Unity Catalog Volume: # Sample init.sh script content
- #!/bin/bash
- echo "Installing Custom PostgreSQL ODBC Driver..."
- apt-get update -y
- apt-get install -y unixodbc odbc-postgresql
- echo "Setting environment variables..."
- echo "DB_TIMEOUT=300" >> /etc/environment
- In the Compute Configuration (or JSON API definition), reference the Unity Catalog Volume path: {
- "init_scripts": [
- {
- "volumes": {
- "destination": "/Volumes/production_catalog/governance_schema/init_volume/install_odbc.sh"
- }
- }
- ]
- }
- Init Script Execution Order & Failure Behavior
Order of Execution:
Global Init Scripts (configured in Admin Settings) execute first on every node in the workspace.
Cluster-Scoped Init Scripts execute second, in the exact sequence specified in the cluster configuration array.
Execution Privileges: Init scripts run as the root user on Ubuntu Linux.
Failure Handling: If an init script returns a non-zero exit code ( exit 1 ), the node bootstrap fails immediately. The cluster aborts startup and transitions to an INIT_SCRIPT_FAILURE terminated state, preventing a broken or unconfigured compute environment from serving traffic.