Study guide
2.6 Comparing and Applying Mean, Median, and Mode Imputation
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
2.6 Comparing and Applying Mean, Median, and Mode Imputation
Key Takeaways Explain This Topic AI
Median imputation is the default for skewed numeric features because the median is robust to the outliers that drag the mean; mean imputation suits symmetric numeric features.
Mode or a constant token is the strategy for categorical features, since a mean has no meaning over unordered levels.
Every univariate imputation shrinks the feature's variance and weakens its covariance with other columns — the price paid for keeping the rows.
pyspark.ml.feature.Imputer supports mean , median , and mode ; fit it on the training split and apply the fitted ImputerModel to validation and test data.
Adding a binary missingness indicator preserves the signal that a value was absent, which matters most when data is Missing Not at Random.
Last updated: August 2026
2.6 Comparing and Applying Mean, Median, and Mode Imputation
Missing data is one of the most pervasive data quality challenges in production machine learning. Null values can originate from faulty sensors, database join mismatches, user opt-outs on web forms, system timeouts, or upstream pipeline failures. Many popular machine learning algorithms (including standard scikit-learn estimators, linear models, and neural networks) will throw immediate runtime errors when presented with missing entries NaN or None .
To build robust, production-grade ML workflows on Databricks, machine learning practitioners must understand the theoretical mechanisms of missingness, the mathematical tradeoffs of various imputation strategies, and how to execute distributed imputation in PySpark without introducing data leakage.
Theoretical Taxonomy of Missing Data Mechanisms
In statistical theory (established by Donald Rubin), missing data is classified into three distinct mechanisms. Identifying the mechanism dictates whether dropping data introduces bias: MISSING DATA MECHANISMS
- │
- ┌────────────────────────────┼────────────────────────────┐
- ▼ ▼ ▼
- MCAR MAR MNAR
- Missing Completely at Random Missing at Random Missing Not at Random
- Independent of all data • Depends on observed data • Depends on the missing
- No bias if dropped • Biased if dropped • Highest risk of bias
- Purely random glitch • Can impute via features • Must use indicators
- Missing Completely at Random (MCAR)
Definition : The probability of a value being missing is completely independent of both observed variables and unobserved values ( P ( M ∣ Y o b s , Y m i s ) = P ( M ) P(M \mid Y_{obs}, Y_{mis}) = P(M) P ( M ∣ Y o b s , Y mi s ) = P ( M ) ).
Real-World Example : A lab technician accidentally drops a random test tube, or a temporary network glitch drops arbitrary IoT sensor packets.
Consequence : Deleting missing rows (Complete Case Analysis) reduces statistical power and sample size, but does not introduce systematic bias into model parameters.
Missing at Random (MAR)
Definition : The probability of missingness depends systematically on other observed features in the dataset, but not on the unobserved missing value itself ( P ( M ∣ Y o b s , Y m i s ) = P ( M ∣ Y o b s ) P(M \mid Y_{obs}, Y_{mis}) = P(M \mid Y_{obs}) P ( M ∣ Y o b s , Y mi s ) = P ( M ∣ Y o b s ) ).
Real-World Example : Male survey respondents are statistically less likely to answer a question regarding depressive symptoms, but gender is recorded and observed.
Consequence : Dropping rows introduces severe selection bias. However, because the missingness is explained by observed features, multivariate imputation or conditional statistical imputation can restore unbiased estimates.
Missing Not at Random (MNAR)
Definition : The probability of missingness depends directly on the unobserved missing value itself ( P ( M ∣ Y o b s , Y m i s ) ≠ P ( M ∣ Y o b s ) P(M \mid Y_{obs}, Y_{mis}) \ne P(M \mid Y_{obs}) P ( M ∣ Y o b s , Y mi s ) = P ( M ∣ Y o b s ) ).
Real-World Example : Individuals with extremely high annual incomes or heavy debt burdens decline to disclose their financial numbers on credit applications.
Consequence : The most dangerous mechanism. Imputing standard means or medians suppresses variance and introduces extreme bias. Data scientists must engineer missingness indicator features to explicitly retain the signal that the value was withheld.
Strategies for Handling Missing Values
Strategy Mechanism / Precondition Advantages Disadvantages & Risks
Complete Case Analysis ( dropna() ) MCAR only; < 3 − 5 % < 3-5\% < 3 − 5% missingness Simple to execute; preserves true observed covariance structure without fabrication. Discards entire rows; can drastically reduce training volume; introduces catastrophic bias under MAR or MNAR.
Feature Column Dropping > 40 − 60 % > 40-60\% > 40 − 60% missingness Removes uninformative, noisy columns with minimal observed signal. Destroys potential predictive signal if missingness itself is predictive (MNAR).
Mean Imputation MCAR/MAR; symmetric Gaussian numerical features Computationally trivial; preserves feature sample mean. Artificially suppresses feature variance ( σ 2 \sigma^2 σ 2 ); distorts covariance with other variables; sensitive to outliers.
Median Imputation MCAR/MAR; skewed numerical features with outliers Robust to extreme outliers; preserves central tendency of non-normal features. Still shrinks overall feature variance and distorts joint probability distributions.
Mode / Constant Imputation Categorical features ( "__MISSING__" or "Unknown" ) Prevents discarding rows; creates a distinct categorical bucket for missingness. Can create an artificially dominant modal category if missingness is high.
Forward / Backward Fill Time-series and sequential data Propagates the most recent valid observation (Last Observation Carried Forward). Inapplicable to non-sequential tabular data; risk of lookahead bias if backward fill is misused.
Picking Among Mean, Median, and Mode
Feature Distribution Strategy Reason
Numeric Roughly symmetric, few outliers Mean Preserves the sample mean; the mean is an efficient estimate of the centre under symmetry
Numeric Skewed, or contains outliers Median The 50th percentile is unaffected by extreme values, so imputed rows are not pushed toward the tail
- Numeric Counts where absence means zero Constant 0 The null is semantically a zero, not an unknown
- Categorical Any Mode , or a constant "unknown" token An average over unordered levels is undefined
Categorical Missingness is itself meaningful Constant token + indicator Keeps "not provided" as its own learnable level
- The exam frequently poses this as: a continuous feature must be imputed with the least
- effort and correct results — which strategy? The correct reasoning is to look at the
- distribution first and choose accordingly, rather than reaching for the mean by
default. There is no imputer that inspects the distribution and decides for you.
What every univariate imputation costs
Replacing k k k missing values with a single constant concentrates mass at that point.
The consequences are consistent across mean, median, and mode:
Variance shrinks — the imputed values have zero spread, so the feature's standard
deviation is understated.
Covariance is diluted — imputed rows carry no relationship to other features, so
correlations move toward zero.
The distribution gains a spike — a histogram after imputation shows an artificial
peak at the imputed value, which is worth checking.
- These costs are acceptable because the alternative — dropping the rows — is usually
- worse. But they are the reason a missingness indicator is so often paired with the
imputation.
Preserving Signal: Missingness Indicator Features
When values are Missing Not at Random (MNAR), the fact that a value is missing is itself a powerful predictive feature. If a data scientist simply imputes the median, the model cannot distinguish between a customer who genuinely had the median value versus a customer who refused to answer.
To solve this, we create a binary missingness indicator : I ( x = null ) I(x = \text{null}) I ( x = null ) . # Creating explicit missingness indicators in PySpark
from pyspark.sql import functions as F
- df_with_indicators = df.withColumn(
- "income_is_missing",
- F.when(F.col("annual_income").isNull(), 1).otherwise(0)
- ).withColumn(
- "credit_score_is_missing",
- F.when(F.col("credit_score").isNull(), 1).otherwise(0)
- )
In scikit-learn, this is automated using SimpleImputer(add_indicator=True) or the dedicated MissingIndicator transformer.
Distributed Imputation in PySpark ML ( pyspark.ml.feature.Imputer )
Apache Spark ML provides a native, distributed transformer/estimator: pyspark.ml.feature.Imputer . It computes imputation metrics in parallel across the cluster and supports mean , median , and mode strategies. from pyspark.ml.feature import Imputer
- # 1. Instantiate the PySpark Imputer estimator
- imputer = Imputer(
- strategy="median", # Options: "mean", "median", "mode"
- missingValue=float("nan"), # Target missing representation
- inputCols=["age", "annual_income", "credit_score"],
- outputCols=["age_imputed", "annual_income_imputed", "credit_score_imputed"]
- )
- # 2. Fit strictly on the training DataFrame
- imputer_model = imputer.fit(train_df)
- # 3. Transform both train and test DataFrames using the fitted model
- train_imputed = imputer_model.transform(train_df)
- test_imputed = imputer_model.transform(test_df)
- Scikit-Learn SimpleImputer Example from sklearn.impute import SimpleImputer
- import numpy as np
- # Instantiate median imputer with missingness indicator generation
- num_imputer = SimpleImputer(strategy="median", add_indicator=True)
- # Fit on training split ONLY, transform train and test
- X_train_imputed = num_imputer.fit_transform(X_train)
- X_test_imputed = num_imputer.transform(X_test)
- Preventing Data Leakage During Imputation
Data leakage occurs when information from outside the training dataset (such as validation or test splits) is inadvertently used to train or parameterize a machine learning model. This produces unrealistically optimistic offline evaluation metrics that fail to replicate in production. INCORRECT (Data Leakage!):
[Full Dataset: Train + Test] ──> Compute Global Median ($65,000) ──> Split Train / Test
CORRECT (Leak-Free ML Pipeline):
- [Full Dataset] ──> Split ──┬──> [Train Set] ──> Fit Imputer (Train Median = $62,000)
- └──> [Test Set] ──> Transform using Train Median ($62,000)
Rules for Leak-Free Imputation:
Split First, Preprocess Second : Always partition raw data into train_df , val_df , and test_df before fitting any imputer.
Fit on Train, Transform on Test : Never call .fit() or .fit_transform() on the test dataset. The test set must be treated as unseen real-world production data.
Encapsulate in ML Pipelines : Embed transformers inside a pyspark.ml.Pipeline or sklearn.pipeline.Pipeline so that cross-validation folds automatically fit imputers strictly on internal training folds.