Study guide
11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
Microsoft Certified: Azure Databricks Data Engineer Associate DP-750
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Azure Databricks Architecture & Compute Selection
1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture 1.2 Compute Types & Workload Matching 1.3 Cluster Access Modes, Standard/Dedicated, & Serverless Compute
2 Chapter 2: Compute Configuration, Performance & Cost Management
2.1 Autoscaling, Databricks Pools, & Node Type Selection 2.2 Photon Vectorized Engine & Cost Governance with Cluster Policies 2.3 Library Management: Workspace, Compute-Scoped, Notebook, & Init Scripts 2.4 Auto-Termination, Budget Alerts, & Tagging for Cost Allocation 2.5 Databricks Runtime & ML Runtime Selection, and Compute Access Permissions
3 Chapter 3: Unity Catalog Storage Infrastructure & Workspace Setup
3.1 Unity Catalog 3-Level Namespace & Metastore Architecture 3.2 Azure Managed Identity, Storage Credentials, & External Locations 3.3 Managed Tables, External Tables, & Unity Catalog Volumes 3.4 Unity Catalog Naming Conventions, Environment Isolation, & Data Granularity 3.5 Lakehouse Federation: Connections, Foreign Catalogs, & Query Pushdown
4 Chapter 4: Unity Catalog Privilege Model & Access Control
4.1 Hierarchical Privilege Model & Securable Objects 4.2 Account-Level Identities, SCIM Sync, & Group Management 4.3 Object Ownership, Ownership Transfer, & Administrative Delegation
5 Chapter 5: Advanced Data Security, Masking & ABAC
5.1 Dynamic Row Filters & Column Masking Implementation 5.2 Attribute-Based Access Control (ABAC) with Unity Catalog Tags 5.3 Data Isolation across Workspaces & Multiple Metastore Patterns 5.4 Secure Data Sharing within Azure Databricks
6 Chapter 6: Governance, Auditing, Lineage & Delta Sharing
6.1 Automated Data Lineage Tracking in Catalog Explorer 6.2 Unity Catalog System Tables: Audit, Billing, and Access Logs 6.3 Data Retention Policies, Time Travel Security, & Tag Governance 6.4 Delta Sharing Protocols: Open Sharing vs. Databricks-to-Databricks 6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions
7 Chapter 7: Ingestion with Auto Loader, Batch & Structured Streaming
7.1 Auto Loader (cloudFiles) Architecture, File Notification vs. Directory Listing 7.2 Auto Loader Schema Inference, Evolution, & Rescued Data Column 7.3 Batch Ingestion with COPY INTO, CTAS, & Multipart Files 7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion 7.5 Streaming Watermarking, Triggers, & Output Modes 7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory
8 Chapter 8: Delta Lake Core Mechanics, ACID & Table Maintenance
8.1 Delta Lake Architecture, Parquet Files, & the _delta_log Transaction Log 8.2 ACID Transactions, Concurrency Control, & Conflict Resolution 8.3 Compaction (OPTIMIZE), Z-Ordering, & Liquid Clustering (CLUSTER BY) 8.4 Data Retention, Snapshot Isolation, & VACUUM Operations 8.5 Delta Change Data Feed (CDF) & MERGE INTO Upsert Patterns 8.6 Choosing a Table Format: Delta, Parquet, CSV, JSON, & Apache Iceberg
9 Chapter 9: Data Cleansing, Transformation & Medallion Architecture
9.1 The Medallion Lakehouse Pattern: Bronze, Silver, and Gold Layer Design 9.2 Data Cleansing: Deduplication, Null Handling, & Type Casting in PySpark/SQL 9.3 Complex Transformations: Higher-Order Functions, Arrays, Structs, & JSON Parsing 9.4 Relational Joins, Window Functions, Set Operators, & Pivoting 9.5 Enforcing Data Quality: Check Constraints & Table Invariants 9.6 Data Profiling: Summary Statistics & Distribution Assessment
10 Chapter 10: Declarative Pipelines, Lakeflow Spark & Delta Live Tables
10.1 Declarative Pipeline Fundamentals: Streaming Tables vs. Materialized Views 10.2 Data Quality Expectations: expect, expect or drop, & expect or fail 10.3 Change Data Capture Ingestion with AUTO CDC / APPLY CHANGES INTO 10.4 Pipeline Execution Modes, Serverless DLT, & Event Logs
11 Chapter 11: Workflow Orchestration with Lakeflow Jobs
11.1 Multi-Task DAG Design in Lakeflow Jobs & Task Type Selection 11.2 Task Dependencies, Parameters, Task Values, & Dynamic Variables 11.3 Triggers (Scheduled, File Arrival, Continuous), Retries, & Timeouts 11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
12 Chapter 12: CI/CD, Git Integration & Databricks Asset Bundles
12.1 Databricks Git Folders & Source Control Integration 12.2 Databricks Asset Bundles (DAB) Architecture & databricks.yml Structure 12.3 Multi-Environment Deployment (Dev / Staging / Prod) & Target Overrides 12.4 Automated Testing, Unit Testing PySpark, & CI/CD Pipelines
13 Chapter 13: Workload Monitoring, Troubleshooting, & Performance Tuning
13.1 Spark UI Mastery: Query Execution DAG, Stages, Tasks, & Skew Analysis 13.2 Troubleshooting Common Failures: OOM, Disk Spill, Data Skew, & Network Timeouts 13.3 Performance Tuning: Adaptive Query Execution (AQE), Caching, & Broadcast Joins 13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics
11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
Key Takeaways Explain This Topic AI
Lakeflow Jobs provides native alerting on critical lifecycle events: on_start , on_success , on_failure , and on_duration_warning_threshold_exceeded (soft SLA breach warnings).
System Notification Destinations support enterprise channels including Slack, Microsoft Teams, PagerDuty, and custom HTTP Webhooks with dynamic templating ( {{job.id}} , {{job.run_id}} , {{event_type}} ).
The 'Repair and Rerun' (Matrix Rerun) feature allows data engineers to re-execute only failed or skipped tasks in a multi-task DAG while preserving the state, task values, and outputs of already succeeded tasks.
Production security governance mandates running Lakeflow Jobs under a Microsoft Entra ID Service Principal ( Run as Service Principal ) rather than an individual user identity to prevent pipeline outages caused by employee offboarding.
SLA duration warning thresholds enable proactive monitoring by dispatching notifications when a job exceeds expected runtimes without abruptly terminating the job.
Last updated: August 2026
11.4 Notifications, Alerting (Email, Slack, Webhooks), & Failure Handling
DP-750 Exam Focus: Master operational observability, notification integrations, and failure remediation in Lakeflow Jobs. Understand system notification events ( on_start , on_success , on_failure , on_duration_warning_threshold_exceeded ), notification destinations (Email, Slack, Teams, Webhooks), the exact mechanics of "Repair and Rerun" for failed DAG tasks, and enterprise identity governance using Run as Service Principal .
1. System Notification Architecture & Lifecycle Events
Real-time visibility into production workflow status is vital for meeting enterprise Service Level Agreements (SLAs). Lakeflow Jobs provides a decoupled notification framework that emits alerts on specific job lifecycle transitions. +---------------------------------------------------------------------------------------------------------+
- | LAKEFLOW JOBS NOTIFICATION ENGINE |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | JOB LIFECYCLE EVENTS DESTINATION INTEGRATIONS |
- | +---------------------------------------+ +---------------------------------------+ |
- | | 1. on_start | | 1. Email (Users & Distribution Lists) | |
- | | - Run initializes & allocates VMs | | | |
- | | 2. on_success | | 2. Slack Notification Channels | |
- | | - All DAG tasks complete cleanly | =========> | | |
- | | 3. on_failure | | 3. Microsoft Teams Webhook Cards | |
- | | - A task fails & halts the DAG | | | |
- | | 4. on_duration_warning_threshold | | 4. PagerDuty Incident Manager | |
- | | - Job exceeds expected runtime SLA | | 5. Custom HTTP Webhooks (JSON POST) | |
- | +---------------------------------------+ +---------------------------------------+ |
- +---------------------------------------------------------------------------------------------------------+
- Lifecycle Event Types
on_start : Fired immediately when the job run transitions from PENDING to RUNNING . Useful for tracking pipeline start times in external audit logs.
on_success : Fired when all tasks in the DAG complete successfully with state TERMINATED / SUCCESS .
on_failure : Fired when one or more tasks fail and the overall job run transitions to FAILED .
on_duration_warning_threshold_exceeded (Soft SLA Alert) : Fired when the active execution duration of a job run exceeds a specified threshold in seconds (e.g., 3600 seconds for a 1-hour SLA). Crucially, this alert does NOT terminate the job ; it proactively notifies operations teams of cluster degradation or data skew before a hard timeout occurs. // Example: Job-Level Notification Configuration
- {
- "email_notifications": {
- "on_start": ["dataops-audit@company.com"],
- "on_failure": ["dataops-oncall@company.com"],
- "on_duration_warning_threshold_exceeded": ["lead-engineer@company.com"],
- "no_alert_for_skipped_runs": true
- },
- "health": {
- "rules": [
- {
- "metric": "RUN_DURATION_SECONDS",
- "op": "GREATER_THAN",
- "value": 3600
- }
- ]
- }
- }
- 2. Notification Destinations (Slack, Teams, Webhooks)
Instead of managing static lists of individual user email addresses across dozens of jobs, Azure Databricks enables Notification Destinations —centralized, reusable webhook integrations managed at the workspace or account level.
Supported Destination Types
Slack : Posts formatted alerts directly to dedicated DataOps Slack channels via incoming webhooks.
Microsoft Teams : Dispatches actionable MessageCards to Microsoft 365 / Teams channels.
PagerDuty : Triggers and resolves incidents automatically for 24/7 on-call engineers.
Custom HTTP Webhooks : Sends HTTP POST requests with custom JSON payloads to internal monitoring tools, Azure Logic Apps, or ServiceNow. // Example: Custom Webhook Payload with Dynamic Templating
- {
- "job_id": "{{job.id}}",
- "run_id": "{{job.run_id}}",
- "job_name": "{{job.name}}",
- "event": "{{event_type}}",
- "run_url": "{{run_page_url}}",
- "timestamp": "{{start_time.iso_date}}"
- }
- 3. Failure Handling & The "Repair and Rerun" (Matrix Rerun) Engine
In complex multi-task DAGs, a failure in a late-stage task (e.g., Task 4 of 5) often leaves the upstream data ingestion and cleansing tasks (Tasks 1, 2, and 3) completed successfully. Re-running the entire job from scratch would waste hours of compute time, double cloud infrastructure costs, and risk duplicate processing on non-idempotent sinks.
Repair and Rerun (also known as Matrix Rerun ) allows engineers to fix the underlying issue (e.g., updating a query syntax error or granting table permissions) and re-execute only the failed tasks and their downstream dependents . +---------------------------------------------------------------------------------------------------------+
- | REPAIR AND RERUN (MATRIX RERUN) WORKFLOW |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | ORIGINAL FAILED RUN (#101): |
- | [ Task 1: Ingest ] ===> [ Task 2: Cleanse ] ===> [ Task 3: Aggregate ] ===> [ Task 4: Publish ] |
- | ( SUCCESS ) ( SUCCESS ) ( FAILED! ) ( SKIPPED ) |
- | | |
- | (Engineer fixes bug in Task 3) |
- | v |
- | REPAIR AND RERUN (#101 - Attempt 2): |
- | [ Task 1: Ingest ] [ Task 2: Cleanse ] [ Task 3: Aggregate ] ===> [ Task 4: Publish ] |
- | ( PRESERVED STATE ) ( PRESERVED STATE ) ( RE-EXECUTED ) ( EXECUTED ) |
- | - Uses cached output - Uses cached TaskValues - Runs fixed logic - Downstream completes |
- +---------------------------------------------------------------------------------------------------------+
- Operational Mechanics of Repair and Rerun
Preserved Upstream State : Succeeded tasks are not re-executed. Their outputs, logs, and Task Values ( dbutils.jobs.taskValues ) remain cached and accessible to the downstream tasks.
Selective Execution : Databricks traverses the DAG from the failed task(s), re-running the failed node and any tasks that depend on it.
Single Run History : The repair run is recorded as a new attempt attached to the original run_id , preserving audit continuity in Unity Catalog lineage and system tables.
API Integration : Can be triggered via the UI ( "Repair run" ) or programmatically via POST /api/2.1/jobs/runs/repair . # Trigger a Repair and Rerun via Databricks CLI / REST API
- databricks jobs repair-run --run-id 451289632 --tasks "aggregate_gold_sales"
- 4. Production Identity Governance: "Run As" Service Principal
In early development, jobs are frequently created by individual data engineers and run under their personal user identities. In an enterprise production environment, this is a critical anti-pattern . +---------------------------------------------------------------------------------------------------------+
- | PRODUCTION IDENTITY GOVERNANCE MATRIX |
- +---------------------------------------------------------------------------------------------------------+
- | |
- | ANTI-PATTERN: RUN AS INDIVIDUAL USER |
- | - User leaves company -> Microsoft Entra ID account disabled -> ALL PRODUCTION JOBS FAIL! |
- | - Personal tokens expire -> Pipelines break unexpectedly. |
- | - Auditing is blurred between human interactive queries and automated background ETL. |
- | |
- | ENTERPRISE BEST PRACTICE: RUN AS SERVICE PRINCIPAL |
- | - Dedicated Microsoft Entra ID (Azure AD) App Registration / Service Principal. |
- | - Permanent identity decoupled from employee lifecycles. |
- | - Granted explicit, least-privilege Unity Catalog grants (`USE CATALOG`, `SELECT`, `MODIFY`). |
- | - Secret tokens secured via Azure Key Vault-backed Databricks Secret Scopes. |
- +---------------------------------------------------------------------------------------------------------+
- Steps to Configure "Run As" Service Principal
Create Service Principal : Register an Application in Microsoft Entra ID and add it as a Service Principal in the Databricks Account Console.
Grant Unity Catalog Privileges : Grant the Service Principal explicit access to required catalogs, schemas, volumes, and SQL warehouses: GRANT USE CATALOG, USE SCHEMA ON CATALOG production TO `application-sp-client-id`;
- GRANT SELECT, MODIFY ON SCHEMA production.silver TO `application-sp-client-id`;
- GRANT USE WAREHOUSE ON SQL WAREHOUSE `analytics_serverless_wh` TO `application-sp-client-id`;
Set Job "Run As" Identity : In the Lakeflow Job settings, navigate to Job details > Run as and select the Service Principal.
Secure Credentials : Use Azure Key Vault-backed secret scopes ( dbutils.secrets.get(scope="kv-prod", key="sp-secret") ) for any required external API credentials.
5. Observability with Unity Catalog Lakeflow System Tables
Lakeflow Jobs telemetry is queryable directly within Unity Catalog through dedicated System Tables in the system.lakeflow schema:
System Table Name Architectural Contents & Query Purpose
system.lakeflow.jobs Metadata definition of all jobs in the metastore (job ID, name, creator, run-as identity, schedule, settings).
system.lakeflow.job_runs Historical record of every job run (run ID, start/end timestamps, termination status, trigger type, cleanup duration).
system.lakeflow.job_tasks Granular per-task telemetry (task key, task run ID, execution duration, task type, cluster instance ID, retry count). -- Example: Querying Average Task Duration & Failure Rates across Production Jobs
- SELECT
- t.task_key,
- t.task_type,
- COUNT(*) AS total_executions,
- SUM(CASE WHEN t.outcome = 'FAILED' THEN 1 ELSE 0 END) AS failure_count,
- AVG(t.execution_duration_seconds) AS avg_duration_sec
- FROM system.lakeflow.job_tasks t
- JOIN system.lakeflow.job_runs r
- ON t.job_run_id = r.job_run_id
- WHERE r.start_time >= CURRENT_DATE() - INTERVAL 30 DAYS
- GROUP BY t.task_key, t.task_type
- ORDER BY failure_count DESC;