Study guide
2.5 When Log-Scale Transformation Is Appropriate
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
2.5 When Log-Scale Transformation Is Appropriate
Key Takeaways Explain This Topic AI
A log transform is appropriate for strictly positive, right-skewed, multiplicative quantities — prices, incomes, counts, durations, and latencies.
log1p ( ln ( x + 1 ) \ln(x+1) ln ( x + 1 ) ) is the standard choice when the feature contains true zeros, because ln ( 0 ) \ln(0) ln ( 0 ) is undefined.
A log transform is inappropriate for features with negative values (use Yeo-Johnson), for already-symmetric features, and for features consumed only by tree models, which are invariant to it.
Logging the target changes what the model optimises: errors become proportional rather than absolute, so predictions must be inverted with expm1 before reporting business metrics.
A log transform compresses large values, which shrinks the leverage of extreme observations — often a better response to skew than deleting rows.
Last updated: August 2026
- 2.5 When Log-Scale Transformation Is Appropriate
- Distribution Transformations for Skewed Features
Many real-world features (e.g., pricing, income, click counts, server latencies) follow long-tailed, right-skewed distributions. Transforming these features toward a bell-shaped Gaussian distribution stabilizes variance, improves linear model convergence, and satisfies homoscedasticity assumptions. Raw Right-Skewed: [||||||||||||| | | | | ] --> Long right tail
- Log-Transformed ln(x): [ ||| |||||||||||||||| ||| ] --> Bell-shaped normal
- Common Transformation Mathematical Formulations
Transformation Formula Domain / Constraints Primary Use Case
Natural Log y = ln ( x ) y = \ln(x) y = ln ( x ) x > 0 x > 0 x > 0 strictly Compresses severe right skew when all values are strictly positive.
Log-Plus-One ( log1p ) y = ln ( x + 1 ) y = \ln(x + 1) y = ln ( x + 1 ) x ≥ 0 x \ge 0 x ≥ 0 Standard choice for non-negative data containing true zeros (e.g., counts, charges). Prevents ln ( 0 ) = − ∞ \ln(0) = -\infty ln ( 0 ) = − ∞ .
Square Root y = x y = \sqrt{x} y = x x ≥ 0 x \ge 0 x ≥ 0 Moderate compression of right skew; milder effect than log transformation.
Box-Cox y ( λ ) = { x λ − 1 λ if λ ≠ 0 ln ( x ) if λ = 0 y^{(\lambda)} = \begin{cases} \frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \ne 0 \\ \ln(x) & \text{if } \lambda = 0 \end{cases} y ( λ ) = { λ x λ − 1 ln ( x ) if λ = 0 if λ = 0 x > 0 x > 0 x > 0 strictly Optimizes parameter λ \lambda λ via maximum likelihood to maximize normality.
Yeo-Johnson Generalization of Box-Cox Supports all real numbers ( x < 0 , x = 0 , x > 0 x < 0, x = 0, x > 0 x < 0 , x = 0 , x > 0 ) Stabilizes variance and normalizes features containing negative, zero, and positive values.
- PySpark and scikit-learn Transformation Implementations # PySpark Log-Plus-One Transformation
- from pyspark.sql.functions import log1p, expm1
df_transformed = df.withColumn("log_revenue", log1p("raw_revenue")) # Scikit-learn PowerTransformer for Box-Cox and Yeo-Johnson
- from sklearn.preprocessing import PowerTransformer
- import numpy as np
- # Yeo-Johnson handles zero and negative values
- pt_yj = PowerTransformer(method="yeo-johnson", standardize=True)
- X_train_yj = pt_yj.fit_transform(X_train)
X_test_yj = pt_yj.transform(X_test) # Fit on train, transform on test!
The Decision Rule
Apply a log transform when all of the following hold:
The feature is strictly positive (or non-negative, using log1p ).
Its distribution is right-skewed with a long tail — the mean sits well above the
median, and the maximum is orders of magnitude beyond the 75th percentile.
- The quantity is naturally multiplicative : a change from $1,000 to $2,000
- means what a change from $100,000 to $200,000 means. Prices, incomes, revenues,
- population counts, page views, session durations, and API latencies all behave this
way.
The downstream model is sensitive to scale and to leverage — linear and logistic
regression, regularised linear models, k-NN, SVM, K-Means, or a neural network.
Skip it when any of the following hold:
Situation Why a log transform is wrong Better option
The feature contains negative values ln ( x ) \ln(x) ln ( x ) is undefined for x ≤ 0 x \le 0 x ≤ 0 Yeo-Johnson
The feature contains true zeros ln ( 0 ) = − ∞ \ln(0) = -\infty ln ( 0 ) = − ∞ log1p — ln ( x + 1 ) \ln(x+1) ln ( x + 1 ) maps 0 to 0
The distribution is already roughly symmetric The transform introduces left skew Leave it, or standardise
The model is a decision tree or tree ensemble Splits depend on rank order, and log is monotonic, so the splits are unchanged No transform needed
- The feature is bounded and narrow (e.g. a 1–5 rating) Nothing to compress No transform needed
- Exam framing : "a strictly positive, heavily right-skewed feature feeding a linear
- model" is the canonical yes. "A right-skewed feature feeding XGBoost" is the
canonical no — a monotonic transform cannot change a tree's split points.
Choosing Among the Power Transforms
- Transform Domain Strength Notes
- ln ( x ) \ln(x) ln ( x ) x > 0 x > 0 x > 0 Strong compression Fails on zeros
ln ( x + 1 ) \ln(x+1) ln ( x + 1 ) ( log1p ) x ≥ 0 x \ge 0 x ≥ 0 Strong compression The default for counts and currency
x \sqrt{x} x x ≥ 0 x \ge 0 x ≥ 0 Mild compression Good for moderate skew, e.g. Poisson-like counts
Box-Cox x > 0 x > 0 x > 0 Fits λ \lambda λ by maximum likelihood Chooses the strength for you; still fails on zeros
Yeo-Johnson All reals Fits λ \lambda λ ; handles negatives and zeros The general-purpose choice when the sign is mixed from pyspark.sql import functions as F
- # Feature transform: log1p tolerates the zeros in a revenue column
- df = df.withColumn("log_revenue", F.log1p("raw_revenue"))
- Transforming the Target Is a Different Decision
- Logging a feature rescales an input. Logging the target changes the loss the
- model minimises: squared error on ln ( y + 1 ) \ln(y+1) ln ( y + 1 ) penalises relative error, so a $50k
- miss on a $1M house counts the same as a $500 miss on a $10k one. That is often
- exactly what a business wants — and it is also why a log-target model is not directly
comparable to a raw-target model until predictions are inverted.
Two consequences follow, and both are examinable:
Every prediction must be inverted with np.expm1 (or F.expm1 ) before it is
reported or compared in original units.
Metrics computed on the log scale are log-scale metrics (effectively RMSLE), not
dollars. Section 3.13 covers this in full.
Target Variable Transformation & Prediction Inversion
When a continuous target variable y y y is heavily right-skewed, training a regression model on the log-transformed target y ~ = ln ( y + 1 ) \tilde{y} = \ln(y + 1) y ~ = ln ( y + 1 ) frequently achieves superior optimization stability.
However, the model will output predictions on the transformed log scale ( y ~ ^ \hat{\tilde{y}} y ~ ^ ). Before computing business metrics (such as Mean Absolute Error in dollars) or reporting results to stakeholders, predictions must be inverted back to the original domain: y ^ r a w = exp ( y ~ ^ ) − 1 \hat{y}_{raw} = \exp(\hat{\tilde{y}}) - 1 y ^ r a w = exp ( y ~ ^ ) − 1 import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error
- # 1. Transform target during training
- y_train_log = np.log1p(y_train)
- model.fit(X_train, y_train_log)
- # 2. Predict on test set (outputs log-scale predictions)
- y_pred_log = model.predict(X_test)
- # 3. Invert predictions back to original domain
- y_pred_original = np.expm1(y_pred_log)
- # 4. Evaluate true performance against actual original target
- rmse_dollars = np.sqrt(mean_squared_error(y_test, y_pred_original))
- mae_dollars = mean_absolute_error(y_test, y_pred_original)
- print(f"Test RMSE: ${rmse_dollars:,.2f}, MAE: ${mae_dollars:,.2f}")