4.3 Streaming Inference with Delta Live Tables

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

4.3 Streaming Inference with Delta Live Tables

Key Takeaways Explain This Topic AI

Delta Live Tables (DLT) is declarative: you define the tables and their queries with @dlt.table , and DLT manages the streams, dependencies, compute, and retries.

A model is applied inside a DLT pipeline by registering it as a Spark UDF with mlflow.pyfunc.spark_udf and calling that UDF in the table's query.

Streaming tables read with dlt.read_stream (or spark.readStream ), so each pipeline update processes only new rows rather than rescanning history.

DLT manages checkpoints for its own streaming tables, which is what makes exactly-once recovery automatic rather than a checkpointLocation you configure by hand.

Data-quality expectations — @dlt.expect , @dlt.expect_or_drop , @dlt.expect_or_fail — enforce contracts on both the input features and the predictions.

Last updated: August 2026

• 4.3 Streaming Inference with Delta Live Tables
• Streaming Inference with Spark Structured Streaming

Structured Streaming applies MLflow UDFs incrementally against continuous data streams with micro-batch processing.

• Implementation with Checkpointing import mlflow.pyfunc
• from pyspark.sql.functions import struct, col

• # 1. Register distributed scoring UDF
• model_uri = "models:/enterprise_ml.iot.turbine_anomaly@champion"
• anomaly_udf = mlflow.pyfunc.spark_udf(spark, model_uri=model_uri, result_type="double")

• # 2. Read streaming DataFrame from Bronze ingestion stream
• streaming_raw_df = spark.readStream \
• .format("delta") \
• .table("enterprise_ml.iot.bronze_sensor_telemetry")

• # 3. Apply inference transformation
• streaming_scored_df = streaming_raw_df.withColumn(
• "anomaly_score",
• anomaly_udf(struct("vibration_level", "rpm", "temperature", "pressure"))
• ).filter("anomaly_score > 0.85") # Alert threshold

• # 4. Write stream to Gold Delta sink with mandatory checkpointing
• query = streaming_scored_df.writeStream \
• .format("delta") \
• .outputMode("append") \
• .option("checkpointLocation", "dbfs:/checkpoints/turbine_anomaly_scoring") \
• .trigger(processingTime="10 seconds") \
• .toTable("enterprise_ml.iot.gold_critical_anomalies")
• The Critical Role of checkpointLocation

In Spark Structured Streaming, the checkpointLocation directory stores the write-ahead transaction log, offset metadata, and state information in durable cloud storage. If an executor or cluster restarts, the stream resumes from the exact recorded offset without re-scoring duplicate records or dropping events, guaranteeing exactly-once processing when combined with Delta Lake sinks.

Declarative ML Pipelines with Delta Live Tables (DLT)

Delta Live Tables (DLT) simplifies production ETL and inference by replacing manual stream and cluster management with declarative pipeline definitions. import dlt

• import mlflow.pyfunc
• from pyspark.sql.functions import struct

• # Broadcast model UDF inside DLT pipeline context
• model_uri = "models:/supply_chain.logistics.delivery_delay_model@champion"
• predict_delay = mlflow.pyfunc.spark_udf(spark, model_uri, result_type="double")

• @dlt.table(
• name="silver_cleaned_shipments",
• comment="Ingests and validates raw logistics telemetry"
• )
• @dlt.expect_or_drop("valid_shipment_id", "shipment_id IS NOT NULL")
• @dlt.expect_or_drop("valid_distance", "distance_miles > 0")

def silver_cleaned_shipments():

return dlt.read_stream("bronze_shipment_events")

• @dlt.table(
• name="gold_predicted_delays",
• comment="Applies MLflow model to forecast delivery delays"
• )
• @dlt.expect_or_fail("valid_prediction", "predicted_delay_minutes >= 0")

def gold_predicted_delays():

• features = ["distance_miles", "weather_severity", "traffic_index", "carrier_historical_delay"]
• return dlt.read("silver_cleaned_shipments").withColumn(
• "predicted_delay_minutes",
• predict_delay(struct(*features))
• )
• Why DLT Rather Than a Hand-Written Stream
• Both approaches apply the same MLflow UDF to the same events. The difference is how

much operational machinery you own.

• Concern Hand-written Structured Streaming Delta Live Tables
• Checkpointing You set checkpointLocation on every writeStream Managed by the pipeline for its streaming tables
• Dependency ordering You sequence the jobs yourself Inferred from the dlt.read / dlt.read_stream graph

Compute lifecycle You size and manage the cluster Provisioned per pipeline update; autoscaling is a pipeline setting

• Retries and recovery Custom job-level retry logic Built into the pipeline update
• Data quality Custom filters and assertions First-class @dlt.expect* decorators with recorded metrics
• Lineage and observability Assembled manually Pipeline graph and per-expectation metrics in the UI
• This is why a scenario that mentions fluctuating event volume, dynamic resizing, and
• declarative pipeline management points to DLT, while one that mentions a single

bespoke stream with custom state handling points to Structured Streaming.

Streaming Tables vs. Materialised Views

Definition Semantics Use for

dlt.read_stream("upstream") Incremental: each update processes only new rows Append-only event ingestion and scoring

dlt.read("upstream") Recomputed from the full upstream contents on each update Aggregates, joins, dimension-style tables

• Scoring an unbounded event feed should read as a stream, so the model is applied once
• per event rather than re-applied to the entire history on every update. That difference

dominates both cost and latency at scale.

Expectations on Predictions, Not Just Inputs

Quality constraints are as valuable on the output of a model as on its input: @dlt.table(name="gold_scored_events")

• @dlt.expect("score_in_range", "anomaly_score BETWEEN 0 AND 1")
• @dlt.expect_or_drop("has_entity", "device_id IS NOT NULL")

def gold_scored_events():

• return dlt.read_stream("silver_events").withColumn(
• "anomaly_score", predict_udf(struct(*feature_cols))
• )
• @dlt.expect records violations as pipeline metrics but keeps the rows — the right

choice for monitoring prediction drift.

@dlt.expect_or_drop filters violating rows out of the target table.

@dlt.expect_or_fail aborts the pipeline update, reserved for violations that must

never reach downstream consumers.

• Because the violation counts are recorded per expectation and per update, a rising
• "score out of range" rate becomes an early warning that the model or its inputs have

shifted — without a separate monitoring job.

• The Serving-Endpoint Alternative Inside a Pipeline
• A pipeline can also call a Model Serving endpoint from a UDF rather than loading the
• model into the stream. That is the right design when the model is very large, when it
• runs on a GPU, or when several consumers must share one governed deployment. It is the
• wrong design for high-volume event scoring, where a per-event HTTPS round trip adds

latency and cost that in-pipeline UDF scoring avoids entirely.

• The Name Changed; the Objective Did Not
• The exam guide dated March 1, 2025 words this objective as "Identify how streaming
• inference is performed with Delta Live Tables," and its own sample question describes
• tens of thousands of events per second with fluctuating volume and compute that must
• resize dynamically. The keyed answer there is a Delta Live Tables pipeline applying the
• algorithm as a Spark UDF — precisely the tradeoff above. The pipeline supplies the
• autoscaling and the managed update semantics, while applying the model in process avoids
• an HTTPS round trip per event. Calling a serving endpoint from the pipeline would add a
• network hop to every event and force two components to scale independently where one

suffices.

• Since the 2025 Data + AI Summit, Databricks has folded Delta Live Tables into Lakeflow;
• the documentation now calls these Lakeflow pipelines , and the declarative framework
• itself was open-sourced as Spark Declarative Pipelines. Databricks states that no
• migration is required — existing DLT code still runs — and the concepts carry over
• directly: streaming tables, materialized views, expectations, and pipeline updates all

mean what they meant before. In current Python, import dlt is superseded by from pyspark import pipelines as dp , where @dp.table creates a streaming table, @dp.materialized_view creates a materialized view, and the old @dlt.view becomes @dp.temporary_view . Expect the exam itself to use the older Delta Live Tables

vocabulary, because that is the language of the live exam guide.

Loading diagram... Distributed Batch and Streaming Inference Execution Lifecycle