Study guide

13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

Microsoft Certified: Azure Databricks Data Engineer Associate DP-750

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Azure Databricks Architecture & Compute Selection

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute

2 Chapter 2: Compute Configuration, Performance & Cost Management

2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions

3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup

3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown

4 Chapter 4: Unity Catalog Privilege Model & Access Control

4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation

5 Chapter 5: Advanced Data Security, Masking & ABAC

5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks

6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing

6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming

7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance

8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg

9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture

9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment

10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables

10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs

11 Chapter 11: Workflow Orchestration with Lakeflow Jobs

11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling

12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles

12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines

13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning

13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

Key Takeaways Explain This Topic AI

Azure Databricks Diagnostic Settings enable streaming audit, security, and operational logs to Azure Log Analytics workspaces, Azure Data Lake Storage Gen2 (ADLS Gen2) for long-term retention, or Azure Event Hubs for real-time SIEM ingestion.

Diagnostic log categories capture granular workspace events: clusters (lifecycle and sizing), jobs (run triggers and failures), notebook (cell execution and exports), dbfs (file operations), unityCatalog (access and permission grants), accounts (user logins and SCIM), and workspace (ACLs and tokens).

Kusto Query Language (KQL) in Azure Log Analytics allows engineers to run high-speed queries to detect unauthorized privilege escalations, trace user data access, audit egress IP addresses, and measure job failure frequencies.

Azure Monitor Metric Alerts track infrastructure and platform health (cluster CPU usage, memory utilization, DBU consumption) and trigger automated notifications via Action Groups (email, SMS, PagerDuty, Webhooks).

Unity Catalog system tables (system.access.audit, system.billing.usage, system.compute.clusters) complement Azure Monitor by enabling direct SQL querying of audit and billing telemetry natively within the lakehouse.

Last updated: August 2026

13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

Enterprise lakehouse governance demands end-to-end operational visibility. Data engineers and platform administrators must monitor workspace health, enforce security compliance, audit data access, track Databricks Unit (DBU) consumption, and establish automated alerting for pipeline failures.

Azure Databricks integrates natively with Azure Monitor via Diagnostic Settings , allowing organizations to stream rich audit logs and telemetry to Azure Log Analytics, Azure Storage, and SIEM platforms.

1. Diagnostic Settings Architecture & Export Destinations

Diagnostic Settings capture control-plane audit events and system operations, routing them from the Azure Databricks resource to one or more enterprise destinations: +-------------------------------------------------------------------------+

  • | AZURE DATABRICKS DIAGNOSTIC TELEMETRY PIPELINE |
  • +-------------------------------------------------------------------------+
  • | |
  • | AZURE DATABRICKS CONTROL PLANE |
  • | ├── Cluster Events ├── Notebook Executions ├── Unity Catalog Grants |
  • | ├── Job Triggers ├── Account Logins ├── DBFS Operations |
  • | └───────────────────────────────┬──────────────────────────────────────┘|
  • | │ Azure Diagnostic Settings |
  • | ▼ |
  • | ┌───────────────────────────────┼──────────────────────────────────┐ |
  • | │ │ │ |
  • | ▼ ▼ ▼ |
  • | LOG ANALYTICS WORKSPACE ADLS GEN2 STORAGE ACCOUNT EVENT HUBS |
  • | - Real-time KQL Queries - Long-term Compliance (7+ yrs) - SIEM Tool|
  • | - Azure Monitor Alerting - Cold Data Lake Archival - Sentinel |
  • | - Azure Workbooks - Cost-effective Storage - Splunk |
  • +-------------------------------------------------------------------------+
  • Comparison of Export Destinations

Destination Primary Use Case Retention & Characteristics

Azure Log Analytics Workspace Operational monitoring, rapid incident investigation, interactive KQL queries, and Azure Monitor alerting rules. Configurable retention (30 to 730 days). Ingestion billed per GB.

ADLS Gen2 Storage Account Immutable historical compliance, regulatory archiving (HIPAA, SOC 2, GDPR), and audit backups. Indefinite lifecycle retention via Azure Blob lifecycle policies. Lowest storage cost.

Azure Event Hubs Real-time streaming into external Security Information and Event Management (SIEM) tools (Microsoft Sentinel, Splunk, Datadog). Real-time streaming buffer (1 to 7 days retention). Low-latency event streaming.

2. Core Diagnostic Log Categories

When configuring Diagnostic Settings on an Azure Databricks workspace, administrators select which log categories to stream:

Log Category Captured Operations & Events

clusters Cluster creation, start, resize, edit, restart, termination, and node failure events.

jobs Lakeflow Job creation, manual/scheduled run triggers, task start/finish, retries, and failure errors.

notebook Notebook cell execution, notebook imports, exports, revisions, and snapshot downloads.

dbfs File creation, read, write, delete, and mount operations within DBFS storage.

unityCatalog Metastore, catalog, schema, and table operations; data access requests; GRANT and REVOKE permission events; lineage capture.

accounts Account-level administrator logins, SCIM user/group provisioning, and account settings changes.

workspace Workspace access control list (ACL) changes, personal access token (PAT) creation and revocation.

secrets Secret scope creation, secret ACL updates, and secret access requests.

sql Databricks SQL Warehouse start/stop, query executions, and dashboard access events.

3. Auditing & Troubleshooting with Kusto Query Language (KQL)

Once logs are ingested into an Azure Log Analytics workspace, data engineers use Kusto Query Language (KQL) to query the DatabricksAccounts , DatabricksClusters , DatabricksJobs , DatabricksNotebook , and DatabricksUnityCatalog tables.

KQL Query 1: Detecting Unauthorized Unity Catalog Privilege Grants

Find all security GRANT events granting ALL PRIVILEGES or MANAGE permissions to non-admin users in the last 7 days: DatabricksUnityCatalog

  • | where TimeGenerated >= ago(7d)
  • | where ActionName in ("grantPermissions", "updatePermissions")
  • | extend Request = parse_json(RequestParams)
  • | extend TargetObject = tostring(Request.securable_type),
  • TargetName = tostring(Request.securable_name),
  • Changes = tostring(Request.changes)
  • | where Changes has "ALL_PRIVILEGES" or Changes has "MANAGE"
  • | project TimeGenerated, Identity, Source_IP_Address, ActionName, TargetObject, TargetName, Changes
  • | order by TimeGenerated desc
  • KQL Query 2: Tracking Production Job Failures & Error Messages
  • Identify failed Lakeflow Job runs over the past 24 hours to monitor pipeline SLA compliance: DatabricksJobs
  • | where TimeGenerated >= ago(24h)
  • | where ActionName == "runFailed"
  • | extend Response = parse_json(Response)
  • | extend JobId = tostring(Response.job_id),
  • RunId = tostring(Response.run_id),
  • ErrorMessage = tostring(Response.error)
  • | project TimeGenerated, Identity, JobId, RunId, ErrorMessage
  • | summarize FailureCount = count() by JobId, ErrorMessage
  • | order by FailureCount desc
  • KQL Query 3: Auditing High-Risk Notebook Data Exports

Detect instances where users exported or downloaded large notebook results to local workstations: DatabricksNotebook

  • | where TimeGenerated >= ago(30d)
  • | where ActionName in ("downloadNotebookExport", "exportNotebook")
  • | project TimeGenerated, Identity, Source_IP_Address, ActionName, RequestParams
  • | order by TimeGenerated desc
  • 4. Azure Monitor Metric Alerts & Action Groups

Azure Monitor collects platform-level performance metrics from the Azure Databricks resource and underlying infrastructure. Data engineers configure Metric Alert Rules to automatically trigger when operational thresholds are breached. +-------------------------------------------------------------------------+

  • | AZURE MONITOR ALERTING PIPELINE |
  • +-------------------------------------------------------------------------+
  • | |
  • | 1. TELEMETRY SOURCE 2. ALERT CONDITION 3. ACTION GROUP |
  • | +--------------------+ +--------------------+ +-----------------+ |
  • | | Databricks Metrics | | Condition: | | Action Group: | |
  • | | - CPU Utilization | -> | Cluster CPU > 90% |-> | - Email DataOps | |
  • | | - Memory Pressure | | for 15 minutes | | - PagerDuty | |
  • | | - Failed Job Runs | | OR Job Failure > 0 | | - Teams Webhook | |
  • | +--------------------+ +--------------------+ | - Logic App | |
  • | +-----------------+ |
  • +-------------------------------------------------------------------------+

Configuring an Alert Rule:

Scope : Target the Azure Databricks workspace resource.

Condition : Define signal logic (e.g., Signal: Cluster CPU Utilization , Aggregation: Average , Operator: Greater than , Threshold: 90% , Evaluation Period: 15 minutes ).

Actions : Attach an Azure Action Group configured with:

Email / SMS / Push / Voice notifications to on-call data engineers.

Webhook : Dispatches JSON payloads to Microsoft Teams channels or Slack.

Azure Logic App / Azure Function : Automatically triggers self-healing workflows (such as restarting a stalled cluster or scaling compute).

5. Azure Monitor vs. Unity Catalog System Tables

Modern Azure Databricks environments provide two complementary observability pillars: Azure Monitor (Log Analytics) and Unity Catalog System Tables . +-------------------------------------------------------------------------+

  • | AZURE MONITOR VS. UNITY CATALOG SYSTEM TABLES |
  • +-------------------------------------------------------------------------+
  • | FEATURE | AZURE MONITOR / LOG ANALYTICS | SYSTEM TABLES |
  • |---------------------|-------------------------------|-------------------|
  • | Query Engine | Kusto Query Language (KQL) | ANSI SQL (Spark) |
  • | Query Interface | Azure Portal Log Analytics | Databricks SQL |
  • | Data Location | Log Analytics Workspace (Azure)| Unity Catalog |
  • | Primary Persona | Azure Admins, SecOps, DevOps | Data Engineers, BI|
  • | Key Schemas/Tables | DatabricksClusters, | system.access.audit
  • | | DatabricksJobs, | system.billing.usage
  • | | DatabricksUnityCatalog | system.compute.* |
  • | Best For | Real-time infrastructure | Lakehouse lineage |
  • | | alerts, SIEM integration | joins, DBU billing|
  • +-------------------------------------------------------------------------+

Exam Tip : Use Azure Monitor / Log Analytics when integrating with enterprise-wide Azure monitoring, SIEMs (Microsoft Sentinel), and infrastructure alerting. Use Unity Catalog System Tables ( system.access.audit , system.billing.usage ) when data engineers and analysts need to join audit and cost telemetry directly with Delta tables in SQL dashboards.

Loading diagram... Azure Databricks Observability Architecture: Azure Monitor vs System Tables