2.4 Removing Outliers with Standard Deviation and IQR

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

2.4 Removing Outliers with Standard Deviation and IQR

Key Takeaways Explain This Topic AI

The IQR rule flags values outside [ Q 1 − 1.5 × I Q R , Q 3 + 1.5 × I Q R ] [Q1 - 1.5 \times IQR,\; Q3 + 1.5 \times IQR] [ Q 1 − 1.5 × I QR , Q 3 + 1.5 × I QR ] and is non-parametric, so it holds on skewed data.

The standard-deviation (Z-score) rule flags ∣ z ∣ > 3 |z| > 3 ∣ z ∣ > 3 but assumes approximate normality, and its mean and standard deviation are themselves distorted by the outliers being sought.

Compute the fences in Spark with percentile_approx or approxQuantile , then filter with a boolean predicate — never collect the column to the driver.

Removal is one option among several: trimming deletes rows, Winsorizing caps values at chosen percentiles, and an indicator column preserves the fact that a value was extreme.

Tree-based models are invariant to feature outliers because splits depend on rank order, so outlier work matters most for linear, distance-based, and gradient-descent models.

Last updated: August 2026

2.4 Removing Outliers with Standard Deviation and IQR

In machine learning feature engineering, an outlier is an observation that deviates substantially from the overall distribution of the dataset. Outliers can arise from measurement errors, sensor failures, data entry typos, data corruption, or genuine rare real-world phenomena (e.g., high-net-worth individuals in wealth management or flash-crash volume spikes in financial trading).

Understanding the mathematical nature of outliers and their downstream impact on specific algorithms is a core competency tested on the Databricks Certified Machine Learning Associate exam.

Algorithm Sensitivity to Outliers

Machine learning algorithms exhibit vastly different degrees of sensitivity to extreme values. The decision to invest time in outlier detection and remediation depends entirely on the chosen model family:

Algorithm Family Sensitivity Level Mathematical Rationale

Ordinary Least Squares (OLS) Linear Regression High OLS minimizes the sum of squared residuals ( ∑ ( y i − y ^ i ) 2 \sum (y_i - \hat{y}_i)^2 ∑ ( y i ​ − y ^ ​ i ​ ) 2 ). An extreme outlier exerts high leverage, pulling the fitted hyperplane toward itself and distorting slope coefficients for the rest of the dataset.

Regularized Linear Models (Ridge, Lasso, ElasticNet) High Although penalties shrink coefficients, the underlying loss function remains squared error, leaving the model vulnerable to outlier distortion.

Logistic Regression & Neural Networks High Gradient descent weight updates are proportional to the magnitude of feature values ( x i j x_{ij} x ij ​ ). Unbounded outlier values cause explosive gradient steps or push sigmoid/softmax activations into saturation zones where gradients vanish.

Distance-Based Models (k-NN, K-Means, SVM with RBF) High Distances (e.g., Euclidean metric ∑ ( x i a − x i b ) 2 \sqrt{\sum (x_{ia} - x_{ib})^2} ∑ ( x ia ​ − x ib ​ ) 2 ​ ) are dominated by dimensions with extreme magnitudes, rendering nearest neighbor and cluster assignment calculations inaccurate.

Principal Component Analysis (PCA) High PCA maximizes variance ( σ 2 \sigma^2 σ 2 ). A single extreme outlier can define the direction of the first principal component entirely by itself.

Tree-Based Models (Decision Trees, Random Forests, XGBoost, LightGBM) Low (Robust / Invariant) Tree split algorithms evaluate monotonic ordering of features rather than absolute magnitudes. A split at x ≤ 50 x \le 50 x ≤ 50 produces identical partitions regardless of whether the maximum value is 100 100 100 or 1 , 000 , 000 1,000,000 1 , 000 , 000 . (Note: Outliers in continuous regression targets y can still affect tree leaf mean predictions, but feature x outliers do not).

• Mathematical Outlier Detection Methods
• The Interquartile Range (IQR) Rule (Tukey's Fences)

The IQR rule is a robust, non-parametric method that makes no assumptions regarding the underlying probability distribution of the data. It is resistant to extreme skewness. IQR = Q 3 − Q 1 \text{IQR} = Q_3 - Q_1 IQR = Q 3 ​ − Q 1 ​ Lower Inner Fence = Q 1 − 1.5 × IQR \text{Lower Inner Fence} = Q_1 - 1.5 \times \text{IQR} Lower Inner Fence = Q 1 ​ − 1.5 × IQR Upper Inner Fence = Q 3 + 1.5 × IQR \text{Upper Inner Fence} = Q_3 + 1.5 \times \text{IQR} Upper Inner Fence = Q 3 ​ + 1.5 × IQR

Observations falling outside [ Lower Inner Fence , Upper Inner Fence ] [\text{Lower Inner Fence}, \text{Upper Inner Fence}] [ Lower Inner Fence , Upper Inner Fence ] are classified as mild outliers. Observations beyond 3.0 × IQR 3.0 \times \text{IQR} 3.0 × IQR are classified as extreme outliers. # Calculating IQR Fences using PySpark percentile_approx

from pyspark.sql import functions as F

• quantiles = df.select(
• F.percentile_approx("annual_income", [0.25, 0.75], 10000).alias("q")
• ).collect()[0]["q"]

• q1, q3 = quantiles[0], quantiles[1]
• iqr = q3 - q1
• lower_fence = q1 - 1.5 * iqr
• upper_fence = q3 + 1.5 * iqr

• print(f"Q1: {q1}, Q3: {q3}, IQR: {iqr}")
• print(f"Valid Range: [{lower_fence}, {upper_fence}]")

• # Flag outliers in PySpark
• df_flagged = df.withColumn(
• "is_income_outlier",
• (F.col("annual_income") < lower_fence) | (F.col("annual_income") > upper_fence)
• )
• The Z-Score Method (Standard Deviation Rule)

The Z-score measures how many standard deviations ( σ \sigma σ ) an observation x x x lies away from the arithmetic mean ( μ \mu μ ): z = x − μ σ z = \frac{x - \mu}{\sigma} z = σ x − μ ​

Under a standard Gaussian (normal) distribution:

• 68.27 % 68.27\% 68.27% of values lie within ∣ z ∣ ≤ 1 |z| \le 1 ∣ z ∣ ≤ 1
• 95.45 % 95.45\% 95.45% of values lie within ∣ z ∣ ≤ 2 |z| \le 2 ∣ z ∣ ≤ 2
• 99.73 % 99.73\% 99.73% of values lie within ∣ z ∣ ≤ 3 |z| \le 3 ∣ z ∣ ≤ 3

Threshold : An observation is flagged as an outlier if ∣ z ∣ > 3.0 |z| > 3.0 ∣ z ∣ > 3.0 .

Exam Warning : The Z-score assumes the underlying data is normally distributed. Furthermore, the calculation of μ \mu μ and σ \sigma σ is itself distorted by extreme outliers (the masking effect). If data is non-normal or skewed, use the IQR method or the Median Absolute Deviation (MAD) rather than standard Z-scores. MAD = median ( ∣ x i − median ( x ) ∣ ) \text{MAD} = \text{median}(|x_i - \text{median}(x)|) MAD = median ( ∣ x i ​ − median ( x ) ∣ ) Modified Z i = 0.6745 × ( x i − median ( x ) ) MAD \text{Modified } Z_i = \frac{0.6745 \times (x_i - \text{median}(x))}{\text{MAD}} Modified Z i ​ = MAD 0.6745 × ( x i ​ − median ( x )) ​

Choosing Between the Two Rules

• IQR / Tukey fences Standard deviation / Z-score
• Assumption None — non-parametric Approximately normal distribution
• Statistics used Q1, Q3 (rank-based) Mean, standard deviation

Robustness High — quartiles barely move when a few extreme points are added Low — the outliers inflate the very σ \sigma σ used to detect them (masking)

• Typical threshold 1.5 × IQR (mild), 3.0 × IQR (extreme) ∣ z ∣ > 3 \lvert z \rvert > 3 ∣ z ∣ > 3
• Best for Skewed features: income, spend, latency, counts Symmetric, roughly Gaussian measurements
• On a right-skewed feature such as income, the Z-score rule flags almost nothing on the
• left and far too little on the right, because a handful of very large values has
• already pushed the mean up and the standard deviation out. The IQR rule is the safer

default and is what Databricks material generally demonstrates.

Filtering in Spark, end to end from pyspark.sql import functions as F

• q1, q3 = df.approxQuantile("annual_income", [0.25, 0.75], 0.01)
• iqr = q3 - q1
• lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr

clean_df = df.filter((F.col("annual_income") >= lower) & (F.col("annual_income") <= upper))

• # Standard-deviation alternative, computed in one distributed pass
• stats = df.select(F.mean("annual_income").alias("mu"),
• F.stddev("annual_income").alias("sigma")).first()
• z_clean_df = df.filter(F.abs((F.col("annual_income") - stats["mu"]) / stats["sigma"]) <= 3)

Two implementation rules the exam rewards:

• Compute the thresholds on the training split only. Fences derived from the full
• dataset leak test-set distribution information into training, exactly as an imputer

fitted on everything would.

• Never collect() the column. approxQuantile and the aggregate functions run
• distributed; pulling millions of values to the driver to sort them is the wrong

answer even when it is written in valid Python.

Outlier Remediation Strategies

Once outliers are detected, data scientists select an appropriate treatment strategy:

Trimming / Filtering (Row Deletion) :

Approach : Drop rows containing outlier values: df.filter((col("x") >= lower) & (col("x") <= upper)) .

When to use : Obvious data corruption or recording errors where the observation is invalid.

Risks : Can remove genuine rare events, reduce sample size, and introduce selection bias.

Winsorization (Percentile Capping) :

Approach : Cap extreme values at predetermined lower and upper percentiles (e.g., 1st and 99th percentiles, or 5th and 95th percentiles).

When to use : When the observation is valid and the row must be preserved, but extreme magnitudes would destabilize parametric models.

• PySpark implementation : # Winsorize / Cap feature values at 1st and 99th percentiles
• p01, p99 = df.select(
• F.percentile_approx("transaction_amount", [0.01, 0.99], 10000).alias("p")
• ).collect()[0]["p"]

• df_capped = df.withColumn(
• "transaction_amount_capped",
• F.when(F.col("transaction_amount") < p01, p01)
• .when(F.col("transaction_amount") > p99, p99)
• .otherwise(F.col("transaction_amount"))
• )

Outlier Indicator Feature Encoding :

Approach : Create a binary indicator column ( 1 1 1 if outlier, 0 0 0 otherwise) before capping or imputing.

Benefit : Preserves the informational signal that an anomaly occurred while preventing numerical instability in the primary feature column.