ExamFinch.com My Study

Study guide

4.1 Data Validation & Quality Frameworks

Practice this exam

Databricks Certified Data Engineer Professional Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Developing Code for Data Processing using Python and SQL - Core Spark SQL & PySpark

1.1 Complex PySpark Transformations & DataFrame API 1.2 Advanced Spark SQL Patterns 1.3 Streaming Workloads with Structured Streaming 1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

2 Chapter 2: Developing Code for Data Processing using Python and SQL - Streaming & Advanced Patterns

2.1 Advanced Delta Lake Features 2.2 Custom & Stateful Streaming Operations 2.3 Python & PySpark Code Modularization 2.4 User-Defined Functions (UDFs) & Performance Impact

3 Chapter 3: Data Ingestion & Acquisition

3.1 Auto Loader (cloudFiles) Production Ingestion 3.2 Change Data Capture (CDC) Ingestion Patterns 3.3 Batch & Streaming Connectors

4 Chapter 4: Data Transformation, Cleansing, and Quality

4.1 Data Validation & Quality Frameworks 4.2 Data Cleansing, Deduplication & Standardization 4.3 Silver and Gold Medallion Layer Refinement 4.4 Handling Schema Drift and Evolution

5 Chapter 5: Data Sharing and Federation

5.1 Delta Sharing Architecture & Governance 5.2 Lakehouse Federation

6 Chapter 6: Monitoring and Alerting & Observability

6.1 System Tables & Observability 6.2 Databricks Workflows & Job Monitoring 6.3 Spark UI & Query Execution Diagnostics 6.4 Data Quality & Pipeline Alerting

7 Chapter 7: Cost & Performance Optimisation - Data Layout & Compute

7.1 Data Storage & Layout Optimization 7.2 Compute Choice & Cluster Sizing 7.3 Photon Engine & Query Acceleration

8 Chapter 8: Cost & Performance Optimisation - Skew & Cost Governance

8.1 Partitioning & Data Skew Remediation 8.2 Cost Governance & DBU Optimization

9 Chapter 9: Ensuring Data Security and Compliance

9.1 Unity Catalog Access Control & Least Privilege 9.2 Dynamic Data Masking & Row/Column-Level Security 9.3 Secrets Management & Network Security 9.4 Compliance, Audit & Data Privacy

10 Chapter 10: Data Governance in Unity Catalog

10.1 Unity Catalog Governance Architecture 10.2 Data Lineage & Tagging in Unity Catalog 10.3 Managed vs External Tables & Storage Governance

11 Chapter 11: Debugging and Deploying - Asset Bundles & CI/CD

11.1 Databricks Asset Bundles (DABs) & Infrastructure as Code 11.2 CI/CD Automation & Databricks CLI / REST API 11.3 Troubleshooting Pipeline & Job Failures 11.4 Code Versioning & Databricks Git Integration

12 Chapter 12: Data Modelling & Lakehouse Analytics

12.1 Medallion Architecture Design Patterns 12.2 Dimensional Modeling in the Lakehouse 12.3 Analytics & BI Optimization

4.1 Data Validation & Quality Frameworks

Key Takeaways Explain This Topic AI

Delta Live Tables (DLT) provides three expectation modifiers: @dlt.expect (record metric & keep row), @dlt.expect_or_drop (drop invalid row), and @dlt.expect_or_fail (fail pipeline).

The Quarantine table pattern explicitly routes invalid records into a separate table, allowing data stewards to investigate anomalies without halting upstream ingestion pipelines.

Expectations in DLT can be defined natively via Python decorators or within SQL using constraint definitions for declarative quality management.

Great Expectations can be integrated within Databricks notebooks using Spark-native evaluations, enabling complex, multi-table validation checks outside of DLT constraints.

DLT event logs store comprehensive metrics on expectation failures, which can be queried using Databricks SQL to build enterprise-wide data quality observability dashboards.

Last updated: July 2026

Data validation and quality frameworks are crucial for ensuring that downstream analytical applications, machine learning models, and executive dashboards consume reliable and accurate information. In the context of Databricks and the Databricks Certified Data Engineer Professional exam, data quality enforcement is heavily tested around declarative pipelines, programmatic data routing, and external validation frameworks. Building robust data engineering pipelines requires moving beyond simple error handling into comprehensive data quality architectures. As data flows through the Medallion Architecture, ensuring its integrity becomes paramount, as the cost of fixing corrupted data increases exponentially the further downstream it reaches.

Delta Live Tables (DLT) Expectations

Delta Live Tables (DLT) makes managing data quality natively easy by introducing 'expectations.' Expectations are constraints that you define for your datasets. When records flow through a DLT pipeline, they are evaluated against these expectations. Depending on the syntax used, DLT handles records failing these expectations in different ways. This declarative approach allows developers to encode business rules directly into the pipeline logic, rather than writing custom filtering transformations that can become difficult to maintain over time.

By integrating data quality directly into the pipeline graph, DLT ensures that data quality is not an afterthought but a first-class citizen of the data engineering lifecycle. The event log automatically captures the results of these expectations, providing built-in observability without requiring external monitoring tools.

The Three Expectation Types

@dlt.expect(name, constraint) : This is the most lenient expectation. When a record violates the constraint, it is still processed and written to the target table. However, DLT records the failure in the event log metrics. This is extremely useful for monitoring data drift, soft validation rules, or non-critical anomalies without losing data. For instance, you might expect a discount_applied field to be less than 50%. If it exceeds 50%, you want to know about it, but you don't necessarily want to drop the transaction.

@dlt.expect_or_drop(name, constraint) : When a record fails this expectation, DLT drops the row entirely and does not write it to the target table. The pipeline continues to run normally, processing the valid records. This is frequently used for strict data cleansing where invalid records are completely unusable and can be safely ignored. Examples include dropping records with a NULL primary key or removing sensor readings that fall completely outside physical possibilities (e.g., a temperature of absolute zero in a standard engine).

@dlt.expect_or_fail(name, constraint) : This is the strictest expectation. If even a single record violates the condition, the entire pipeline execution fails and halts. No further data is processed. This is essential when data integrity is so critical that processing flawed data would corrupt the entire dataset or violate strict regulatory compliance. For instance, if a batch of financial transactions contains negative transfer amounts that should never occur, halting the pipeline prevents downstream financial dashboards from displaying incorrect revenue figures.

  • Code Example: DLT Expectations in Python import dlt
  • from pyspark.sql.functions import expr
  • @dlt.table(name="valid_transactions")
  • @dlt.expect("valid_timestamp", "timestamp IS NOT NULL")
  • @dlt.expect_or_drop("positive_amount", "amount > 0")
  • @dlt.expect_or_fail("valid_account", "account_id IS NOT NULL AND length(account_id) = 10")

def get_valid_transactions():

return dlt.read_stream("raw_transactions")

In this example, transactions with a null timestamp are kept but flagged, transactions with an amount less than or equal to zero are dropped, and if any transaction has an invalid account ID, the pipeline fails immediately.

The Quarantine Table Pattern

While @dlt.expect_or_drop is useful for discarding data, sometimes dropped records must be investigated, audited, or reprocessed. Dropping data silently can lead to compliance issues or lost business value. The Quarantine table pattern (sometimes referred to as a Dead Letter Queue or DLQ pattern in other contexts) addresses this requirement by splitting a single input stream into two distinct output streams: one for valid records (the target table) and one for invalid records (the quarantine table).

Instead of implicitly dropping data, the pipeline uses explicit transformations to route data. A common implementation involves creating a view or table that evaluates the quality rules and appends a boolean flag (e.g., is_valid ) or an array of error messages. Downstream tables then filter based on these flags.

Implementation Example @dlt.table(name="transactions_with_quality_flags")

def evaluate_transactions():

  • df = dlt.read_stream("raw_transactions")
  • # Add flags for various quality checks
  • return df.withColumn("is_valid", expr("amount > 0 AND account_id IS NOT NULL")) \
  • .withColumn("error_reason", expr(
  • "CASE WHEN amount <= 0 THEN 'Negative Amount' " +
  • "WHEN account_id IS NULL THEN 'Missing Account' ELSE NULL END"
  • ))

@dlt.table(name="valid_transactions")

def get_valid():

  • # Only forward valid records to the clean table
  • return dlt.read_stream("transactions_with_quality_flags").filter("is_valid = true")

@dlt.table(name="quarantined_transactions")

def get_quarantined():

  • # Route invalid records to the quarantine table for investigation
  • return dlt.read_stream("transactions_with_quality_flags").filter("is_valid = false")

This pattern guarantees zero data loss while preventing bad data from entering the trusted "Silver" or "Gold" layers of the lakehouse. Data stewards can periodically query the quarantine table to identify upstream system bugs, correct the data, and reinject it into the pipeline.

Great Expectations on Databricks

For teams requiring complex data quality rules that span multiple tables, historical trend analysis, or statistical profiling, third-party frameworks like Great Expectations (GE) are often integrated into Databricks workflows. Great Expectations is an open-source Python library that allows you to define programmatic assertions about your data.

While DLT expectations are evaluated on a per-record basis as data streams through the pipeline, Great Expectations is typically executed in batch mode against an entire DataFrame or an existing Delta table. GE creates "Expectation Suites" which can include complex statistical bounds (e.g., expect_column_mean_to_be_between , expect_column_kl_divergence_to_be_less_than ).

On Databricks, GE runs natively against PySpark DataFrames, leveraging the distributed compute of the Apache Spark cluster. The results of a GE validation run can be saved as JSON reports, visually rendered in HTML "Data Docs," or used to conditionally fail a Databricks Job task if the overall data quality score drops below a specific threshold. This makes GE incredibly powerful for post-load validation steps or certification of Gold layer datasets before they are published to business users.

Custom Data Quality Verification Rules

Beyond DLT and Great Expectations, data engineers often build custom data quality verification rules using standard PySpark DataFrame API methods. This might involve running aggregate queries to ensure row counts match between source and destination, or verifying that cross-table foreign key relationships remain intact after a large batch update.

When writing custom verification logic, engineers typically use assertions within a Databricks notebook. If an assertion fails, the notebook throws an exception, causing the encompassing Databricks Workflow task to fail. This is a common pattern for post-load validation steps in complex orchestration pipelines.

For instance, an engineer might write a custom script to calculate the total sum of account balances in a Silver table and compare it against a control file delivered by the upstream source system. If the values differ by more than a defined tolerance, the pipeline halts and sends an alert.

  • Feature DLT Expectations Great Expectations Custom PySpark
  • Execution Mode Row-level, Streaming/Batch Batch-level, Statistical Batch-level, Programmatic
  • Integration Native to Databricks DLT Open-source Library Native PySpark API
  • Complexity Simple constraints, SQL expressions Complex statistical assertions Unlimited custom logic
  • Reporting DLT Event Log Metrics HTML Data Docs / JSON Custom implementation
  • Querying DLT Event Logs for Quality Metrics

A crucial aspect of managing data validation in Databricks is observing the results over time. DLT captures all expectation metrics in its event log, which is itself stored as a Delta table. Data engineers can query this event log using Databricks SQL to create data quality dashboards.

By querying the flow_progress events in the DLT event log, engineers can extract the number of records that passed, failed, or were dropped for every expectation defined in the pipeline. This allows organizations to track data quality trends over time, identify specific days when data quality degrades, and calculate Data Quality Service Level Indicators (SLIs) for the enterprise.

Mastering these quality frameworks is absolutely essential for the Data Engineer Professional exam, as you must demonstrate the ability to architect resilient pipelines that handle imperfect data gracefully, ensure compliance, route data efficiently, and alert stakeholders to severe anomalies without writing overly complex manual logic.