Study guide
6.4 Data Quality & Pipeline Alerting
Databricks Certified Data Engineer Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark
1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines
2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns
2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact
3 Chapter 3: Data Ingestion & Acquisition
3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors
4 Chapter 4: Data Transformation, Cleansing, and Quality
4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution
5 Chapter 5: Data Sharing and Federation
5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation
6 Chapter 6: Monitoring and Alerting & Observability
6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting
7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute
7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration
8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance
8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization
9 Chapter 9: Ensuring Data Security and Compliance
9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy
10 Chapter 10: Data Governance in Unity Catalog
10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance
11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD
11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration
12 Chapter 12: Data Modelling & Lakehouse Analytics
12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization
6.4 Data Quality & Pipeline Alerting
Key Takeaways Explain This Topic AI
Delta Live Tables (DLT) stores all operational and data quality metrics in an automatically managed event log.
You can query a pipeline's event log using the table-valued function event_log('pipeline_id').
Expectation metrics, such as passed_records and failed_records, are stored within the details:flow_progress column as JSON.
Databricks SQL Alerts can be triggered by writing queries against the DLT event log to monitor data drift and quality drops.
Records failing an EXPECT OR DROP expectation are excluded from the target table, but pipeline execution continues normally.
Last updated: July 2026
6.4 Data Quality & Pipeline Alerting
In modern data engineering, delivering data on time is only half the battle; the data must also be accurate, trustworthy, and compliant with business rules. Databricks Delta Live Tables (DLT) provides built-in declarative data quality through Expectations. However, defining expectations is just the first step. To maintain a healthy lakehouse, you must actively monitor these data quality metrics, build automated dashboards, and set up alerts for when data drifts or quality thresholds are breached. This section covers how to unlock and utilize the DLT Event Log for advanced pipeline observability, enabling a robust DataOps culture.
The DLT Event Log
Every Delta Live Tables pipeline automatically generates an event log. This log is not just a dump of error messages; it records every action, state change, and data quality metric generated during the pipeline's execution. Crucially, the event log itself is stored as a standard Delta table, managed entirely by the Databricks platform.
Because it is a Delta table, you can query it using standard SQL in Databricks SQL, a notebook, or even through external BI tools. The event log is the foundational dataset for all DLT observability, acting as a unified catalog of the operational state and data fidelity of your pipelines.
Querying the Event Log
To query the event log, you use the built-in event_log table-valued function, passing the ID of your pipeline as a string argument. You can easily find the Pipeline ID in the DLT UI settings pane. -- Retrieve the most recent 100 events from a specific pipeline
- SELECT
- timestamp,
- event_type,
- level,
- message
- FROM event_log('a1b2c3d4-e5f6-7890-abcd-1234567890ab')
- ORDER BY timestamp DESC
- LIMIT 100;
The event_type column is critical for filtering. Common event types include flow_progress (which contains data quality metrics and throughput statistics), user_action (such as starting or stopping the pipeline), and plan_update (recording changes to the DAG structure). Filtering by event type allows you to isolate exactly the telemetry you need.
Extracting Expectation Metrics
When you define an expectation in DLT (e.g., EXPECT (age > 0) OR DROP ), DLT evaluates every record against this rule during the data processing phase. The results of these evaluations are logged in the event log under the flow_progress event type. This provides a historical audit trail of data quality.
The detailed metrics are stored as a complex JSON string within the details column. To analyze data quality effectively, you must parse this JSON payload and extract quantitative fields like passed_records and failed_records .
SQL Query for Data Quality Metrics
Here is how you parse the JSON to extract the pass/fail rates for specific datasets and expectations. Databricks SQL provides robust JSON parsing functions that make this straightforward: -- Extract data quality metrics from the event log
- SELECT
- timestamp,
- details:flow_progress:data_quality:dataset_name AS dataset,
- explode(from_json(details:flow_progress:data_quality:expectations,
- 'array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>'
- )) AS expectation_metrics
- FROM event_log('your-pipeline-id')
- WHERE event_type = 'flow_progress'
- AND details:flow_progress:data_quality IS NOT NULL;
By expanding the array of expectations using the explode function, you get a distinct row for every rule evaluated. This row contains the exact count of records that passed and failed during that specific micro-batch or pipeline run, providing granular insights into exactly which rules are failing and how often.
Handling Failed Records
Recall how DLT handles expectation failures based on your declaration. Understanding this behavior is critical for interpreting the metrics:
EXPECT (Warn): The record fails the rule but is still written to the target table. Metrics are logged. This is useful for observing data drift without impacting downstream consumers immediately.
EXPECT OR DROP : The record fails and is dropped (excluded) from the target table. The pipeline successfully continues processing the remaining good records. This ensures that downstream dashboards never see corrupted data.
EXPECT OR FAIL : The entire pipeline immediately halts execution and throws an exception if a single record fails. This is a severe safeguard used for mission-critical assertions where bad data is worse than no data.
By querying the event log, you can track exactly how many records are being silently dropped by an EXPECT OR DROP rule over time. If a rule typically drops 5 records a day but suddenly drops 5,000, that is a critical incident that requires investigation.
Building Automated Data Quality Dashboards
Once you have the SQL queries extracting passed_records and failed_records , you can build automated dashboards directly within Databricks SQL. These dashboards serve as the command center for data quality.
By calculating the failure rate using the formula (failed_records / (passed_records + failed_records)) * 100 , you can create time-series line charts tracking data quality over time. If the failure rate for an expectation checking user_email_format suddenly spikes from 0.1% to 15%, the dashboard will immediately visualize this data drift. This visual indication often points to an upstream schema change by a vendor or a new bug introduced in the source system software.
Triggering Alert Notifications
Visual dashboards are excellent for reporting, but they require human eyes to be actively watching. For proactive, enterprise-grade observability, you need automated alerting mechanisms.
Databricks SQL Alerts allow you to run these analytical queries on a schedule and trigger notifications (via Email, Slack, PagerDuty, or custom Webhooks) if a certain conditional threshold is met.
Setting up a Quality Drop Alert
To build a robust alerting system for data quality, follow these steps:
Write the Query: Create a query in Databricks SQL that calculates the failure rate for the last 24 hours directly from the DLT event log.
Set the Threshold: Configure the Databricks SQL Alert to evaluate this query periodically and trigger if the failure rate column evaluates to > 5.0 (i.e., a failure rate greater than 5%).
Configure Notifications: Route the alert to the Data Engineering Slack channel or an incident management tool.
With this architecture, if a vendor subtly changes the format of the incoming data, causing records to fail DLT expectations, the engineering team receives a Slack alert within minutes of the pipeline run completing. They can then check the Databricks SQL dashboard, identify the exact expectation that failed, investigate the dropped records, and remediate the issue proactively before business users consume corrupted data or make flawed business decisions based on it. This represents the pinnacle of modern data pipeline observability.