2.1 Computing Summary Statistics on a Spark DataFrame

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

2.1 Computing Summary Statistics on a Spark DataFrame

Key Takeaways Explain This Topic AI

df.summary() returns count, mean, stddev, min, 25%, 50%, 75%, and max as a Spark DataFrame, and accepts custom percentiles such as df.summary("count", "5%", "95%") .

df.describe() is the narrower method: count, mean, stddev, min, and max only — it computes no percentiles, so it cannot give you a median or an IQR.

dbutils.data.summarize(df) renders an interactive Databricks profile with null percentages, distinct counts, zero counts, and per-column distribution charts.

Summary output is itself a DataFrame, so it must be displayed or collected; percentile statistics on large data are computed approximately for scalability.

Reading the null percentage, distinct count, and min/max bounds before modelling is what surfaces sentinel values, constant columns, and impossible ranges.

Last updated: August 2026

2.1 Computing Summary Statistics on a Spark DataFrame

Exploratory Data Analysis (EDA) is the indispensable first phase of any production machine learning workflow on the Databricks Lakehouse. In a Lakehouse architecture, data scientists interact directly with massive raw, bronze, or silver Delta Lake tables using Apache Spark. Because Lakehouse datasets frequently scale to millions or billions of rows across distributed clusters, traditional single-node EDA tools (such as native pandas or matplotlib) cannot operate directly without memory bottlenecks or aggressive downsampling. Databricks resolves this challenge by providing distributed data profiling and interactive visualization utilities natively inside Databricks notebooks.

Rigorous EDA achieves several mission-critical objectives before a single algorithm is trained:

Data Quality & Integrity Verification : Identifying missing values, unexpected null clusters, sentinel values (such as -999 or "N/A" ), and data type mismatches.

Target Variable Diagnostics : Assessing class balance for classification problems and evaluating distribution skewness, kurtosis, and multi-modality for regression targets.

Feature Distribution & Variance Analysis : Uncovering constant or near-zero variance features that contribute no predictive signal, as well as extreme outliers.

Feature Interactions & Collinearity : Detecting multi-collinearity among predictor variables and uncovering non-linear relationships with the target label.

Temporal & Spatial Partitioning Patterns : Verifying that temporal distributions are consistent across time to prevent future data leakage.

Automated Data Profiling with dbutils.data.summarize()

Databricks provides a powerful built-in utility specifically engineered for rapid tabular profiling: dbutils.data.summarize() . When executed against an Apache Spark DataFrame, Databricks leverages the distributed Spark Catalyst optimizer to compute aggregate statistics and renders an interactive, visual data profile directly in the notebook output cell. # Load feature dataset from Unity Catalog Silver table

df = spark.table("ml_catalog.feature_store.customer_churn_features")

• # Generate an automatic interactive profile
• dbutils.data.summarize(df) # Comparison with native Spark DataFrame summary methods
• # df.summary() returns a Spark DataFrame containing standard statistics
• summary_df = df.summary("count", "mean", "stddev", "min", "25%", "50%", "75%", "max")
• display(summary_df)
• Metrics Computed by dbutils.data.summarize()

The dbutils.data.summarize() command automatically categorizes columns by data type and generates distinct statistical profiles:

Column Data Type Computed Summary Statistics & Visualizations

Numerical Columns (Integer, Double, Float, Decimal) Total row count, missing/null value count and percentage, number of zeros, distinct value count, mean, standard deviation, minimum, 25th percentile ( Q 1 Q1 Q 1 ), median ( Q 2 Q2 Q 2 ), 75th percentile ( Q 3 Q3 Q 3 ), maximum, and an interactive distribution histogram with dynamic binning.

Categorical / String Columns Total row count, missing/null count and percentage, empty string count, distinct value cardinality, and a frequency bar chart depicting the top most frequent category values along with long-tail distributions.

Boolean Columns Total row count, missing/null count, true count, false count, and true/false proportion percentage bar.

Timestamp / Date Columns Total row count, missing/null count, distinct count, minimum timestamp, maximum timestamp, and temporal distribution histogram across time intervals.

dbutils.data.summarize() vs. df.describe() vs. df.summary()

Understanding the operational differences between these profiling approaches is frequently tested on the Databricks Machine Learning Associate exam:

df.describe(*cols) : A standard PySpark DataFrame method that calculates basic descriptive statistics: count , mean , stddev , min , and max . It does not calculate percentiles (such as the median or interquartile range).

df.summary(*statistics) : An extended PySpark DataFrame method that calculates count , mean , stddev , min , 25% , 50% , 75% , and max by default, or allows custom percentile specifications (e.g., df.summary("count", "5%", "95%") ). It returns a tabular Spark DataFrame.

dbutils.data.summarize(df) : A Databricks-specific UI command that executes distributed computations across the cluster and renders a comprehensive visual dashboard with interactive charts, null indicators, and distribution shapes without requiring manual charting.

Choosing Between the Three Commands

• Need Command Returns
• A quick count/mean/stddev/min/max table in code df.describe() Spark DataFrame — no percentiles

Quartiles, median, or custom percentiles in code df.summary() Spark DataFrame including 25%, 50%, 75% by default

A visual profile with nulls, cardinality, and distributions dbutils.data.summarize(df) Interactive rendering in the notebook cell # describe(): no percentiles at all

display(df.describe("tenure_months", "monthly_charges"))

• # summary(): quartiles by default, or name exactly what you want
• display(df.summary())
• display(df.summary("count", "mean", "5%", "50%", "95%"))
• Both describe() and summary() return a DataFrame , not printed output. A cell

that ends in df.summary() shows nothing useful until it is wrapped in display() or .show() — a detail that appears in exam code snippets.

• Behaviour worth remembering
• describe() and summary() operate on numeric and string columns; string columns

get count, min, and max (lexicographic) but no mean or stddev.

• Percentiles are computed with an approximate algorithm so the work stays
• distributed. For an exact quantile use df.stat.approxQuantile(col, probs, 0.0) ,

which is exact at a relative error of 0 but far more expensive.

• Neither method reports null counts directly; the count row shows non-null counts,
• so a column whose count is lower than the DataFrame's row count has nulls. This is

the reason dbutils.data.summarize exists — it reports missingness explicitly.

Reading the Profile for Defects

A summary table is only useful if you know what to look for:

• Signal in the profile Likely defect Action
• count well below the row count Missing values Plan imputation (Section 2.6)

min is negative on a quantity that cannot be negative Sentinel encoding ( -1 , -999 ) Convert sentinels to null before imputing

• max far beyond the 75th percentile Right skew or genuine outliers Inspect with IQR fences (Section 2.4)
• stddev of exactly 0 Constant column Drop it — no predictive signal
• Distinct count ≈ row count on a string column Identifier masquerading as a category Exclude from features
• mean far from the median Skewed distribution Prefer median imputation; consider a log transform (Section 2.5)