ExamFinch.com My Study

Study guide

3.13 Exponentiating Log-Transformed Predictions Before Evaluation

Practice this exam

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

3.13 Exponentiating Log-Transformed Predictions Before Evaluation

Key Takeaways Explain This Topic AI

A model trained on z = ln ⁡ ( y + 1 ) z = \ln(y+1) z = ln ( y + 1 ) predicts on the log scale; its raw predictions are not in the target's units.

Metrics computed on z ^ \hat{z} z ^ measure logarithmic error — RMSE on the log scale is effectively RMSLE, not dollars.

Invert with y ^ = exp ⁡ ( z ^ ) − 1 \hat{y} = \exp(\hat{z}) - 1 y ^ ​ = exp ( z ^ ) − 1 ( np.expm1 , F.expm1 ) before computing business metrics or reporting predictions.

Use the exact inverse of the transform applied: expm1 undoes log1p , exp undoes log , and 10 z ^ 10^{\hat{z}} 1 0 z ^ undoes log ⁡ 10 \log_{10} lo g 10 ​ .

Comparing a log-target model against a raw-target model is only valid after both are evaluated in the same original units.

Last updated: August 2026

  • 3.13 Exponentiating Log-Transformed Predictions Before Evaluation
  • When a skewed target is log-transformed for training (Section 2.5), everything the
  • model emits lives on the log scale. Forgetting to come back is one of the most common

silent errors in applied regression, and the exam names it explicitly.

Evaluating Models Trained on Log-Transformed Targets

When predicting highly skewed targets (e.g. house prices, customer spend, income), practitioners frequently apply a logarithmic transformation during data preparation: z = ln ⁡ ( y + 1 ) ( or z = log ⁡ 10 ( y ) ) z = \ln(y + 1) \quad (\text{or } z = \log_{10}(y)) z = ln ( y + 1 ) ( or z = lo g 10 ​ ( y )) +-----------------------------------------------------------------------------+

  • | LOG-TRANSFORMED TARGET EVALUATION WORKFLOW |
  • | |
  • | Training Phase: |
  • | Target y ---> [ z = ln(y + 1) ] ---> Model Fits on z ---> Predicts z_hat |
  • | |
  • | INCORRECT EVALUATION (Log-Scale Error): |
  • | RMSE(z, z_hat) = 0.35 <-- Reflects Root Mean Squared Log Error (RMSLE) |
  • | DO NOT compare directly to un-logged RMSE! |
  • | |
  • | CORRECT EVALUATION (Original Dollar/Unit Scale): |
  • | z_hat ---> [ y_hat = exp(z_hat) - 1 ] ---> RMSE(y, y_hat) = $42,500 |
  • | * Allows direct apples-to-apples comparison with baseline dollar models! |
  • +-----------------------------------------------------------------------------+

Important

Exam Rule for Log Transformations: Computing metrics on predicted log values z ^ \hat{z} z ^ measures error on the logarithmic scale (equivalent to RMSLE). To evaluate real-world business impact and compare fairly against models trained on the raw target, you must apply the inverse transformation y ^ = exp ⁡ ( z ^ ) − 1 \hat{y} = \exp(\hat{z}) - 1 y ^ ​ = exp ( z ^ ) − 1 (for log1p ) and calculate metrics between true y y y and y ^ \hat{y} y ^ ​ .

  • Matching the Inverse to the Transform
  • The inverse must exactly undo what was applied. Mixing them produces predictions that
  • are wrong by a constant factor, which is easy to miss because they still look

plausible.

  • Applied during preparation Inverse before evaluation NumPy PySpark
  • z = ln ⁡ ( y ) z = \ln(y) z = ln ( y ) y ^ = e z ^ \hat{y} = e^{\hat{z}} y ^ ​ = e z ^ np.exp F.exp

z = ln ⁡ ( y + 1 ) z = \ln(y+1) z = ln ( y + 1 ) ( log1p ) y ^ = e z ^ − 1 \hat{y} = e^{\hat{z}} - 1 y ^ ​ = e z ^ − 1 np.expm1 F.expm1

z = log ⁡ 10 ( y ) z = \log_{10}(y) z = lo g 10 ​ ( y ) y ^ = 10 z ^ \hat{y} = 10^{\hat{z}} y ^ ​ = 1 0 z ^ np.power(10, z) F.pow(F.lit(10), z)

  • z = y z = \sqrt{y} z = y ​ y ^ = z ^ 2 \hat{y} = \hat{z}^2 y ^ ​ = z ^ 2 np.square F.pow(z, 2)
  • Using np.exp where np.expm1 was required shifts every prediction by exactly 1 unit —

negligible on a $500,000 house, material on a 3-unit demand forecast.

  • Two Distinct Failures
  • Reporting the wrong number. "The model's RMSE is 0.28" is meaningless to a

stakeholder expecting dollars. On a natural-log target it describes a multiplicative error: a typical prediction is off by a factor of e 0.28 ≈ 1.32 e^{0.28} \approx 1.32 e 0.28 ≈ 1.32 , so about

  • 32% high or low. The shorthand "an RMSE of r r r on a log target is about an r × 100 % r \times
  • 100\% r × 100% relative error" is only a small- r r r approximation, and it understates the true

figure as r r r grows — 0.28 reads as 28% but is really 32%.

  • Comparing incomparable models. A log-target model showing RMSE 0.28 and a
  • raw-target model showing RMSE 42,000 cannot be ranked against each other. Invert the

first, then compute both RMSEs in dollars.

A note on the mean. Because exp ⁡ \exp exp is convex, exp ⁡ ( E [ z ] ) ≤ E [ exp ⁡ ( z ) ] \exp(\mathbb{E}[z]) \le

\mathbb{E}[\exp(z)] exp ( E [ z ]) ≤ E [ exp ( z )] (Jensen's inequality), so naively exponentiating a prediction of

  • the log-mean slightly under-predicts the mean on the original scale. The exam expects
  • the straightforward expm1 inversion; the bias correction matters mainly when the

residual variance is large.

A Worked Number

Take a house whose true price is $400,000, on a log1p target.

Wrong — report the log-scale residual. The true value on the log scale is ln ⁡ ( 400,001 ) ≈ 12.899 \ln(400{,}001) \approx 12.899 ln ( 400 , 001 ) ≈ 12.899 . A prediction of z ^ = 12.9 \hat{z} = 12.9 z ^ = 12.9 gives a log-scale

error of about 0.001 0.001 0.001 : a number that sounds like near-perfection and means nothing

to a stakeholder.

Right — invert first. y ^ = e 12.9 − 1 ≈ 400,311 \hat{y} = e^{12.9} - 1 \approx 400{,}311 y ^ ​ = e 12.9 − 1 ≈ 400 , 311 , so the

dollar-scale error is roughly $311.

Now move the prediction to z ^ = 13.2 \hat{z} = 13.2 z ^ = 13.2 . The log-scale error grows to about 0.30 0.30 0.30 ,

which still reads as small, but e 13.2 − 1 ≈ 540,364 e^{13.2} - 1 \approx 540{,}364 e 13.2 − 1 ≈ 540 , 364 — a miss of roughly

  • $140,000. That is the property which makes log-scale metrics deceptive: a constant
  • error on the log scale is a constant proportional error on the original scale , so

the same 0.30 0.30 0.30 is $311-sized on a cheap house and six figures on an expensive one.

  • Why log1p Rather Than Plain log
  • log1p exists for the zero problem. A target with legitimate zeros — units sold, claim
  • amounts, session minutes — cannot be passed to ln ⁡ \ln ln , which is undefined at 0. Shifting

by one maps ln ⁡ ( 0 + 1 ) = 0 \ln(0 + 1) = 0 ln ( 0 + 1 ) = 0 , keeping those rows in the data set at a finite value. The

cost is that the shift must be remembered on the way back: expm1 , never exp .

  • Interpreting Predictions, Not Only Metrics
  • The objective names interpretation alongside metric calculation, and the trap runs in
  • both directions. On a log target an effect is multiplicative, not additive : a

prediction that rises by 0.10 0.10 0.10 on the log scale corresponds to roughly a 10.5% increase in the target ( e 0.10 − 1 = 0.105 e^{0.10} - 1 = 0.105 e 0.10 − 1 = 0.105 ), not a 0.10-unit increase. A prediction

interval has to be exponentiated endpoint by endpoint, which is why a correctly

inverted interval is asymmetric in dollars around the inverted point estimate.

  • Worked Implementation: Inverting a Log Target in PySpark
  • PySpark ML: Multi-Metric Evaluation & Log-Target Inversion from pyspark.sql import functions as F
  • from pyspark.ml.evaluation import RegressionEvaluator
  • # Load predictions DataFrame containing log-transformed target
  • # actual: 'log_price', predicted: 'prediction' (log scale)
  • scored_df = spark.table("lakehouse_gold.real_estate_predictions")
  • # 1. Evaluate metrics on Logarithmic Scale (RMSLE)
  • evaluator_rmse = RegressionEvaluator(labelCol="log_price", predictionCol="prediction", metricName="rmse")
  • evaluator_r2 = RegressionEvaluator(labelCol="log_price", predictionCol="prediction", metricName="r2")
  • log_rmse = evaluator_rmse.evaluate(scored_df)
  • log_r2 = evaluator_r2.evaluate(scored_df)
  • print(f"Log-Scale RMSE: {log_rmse:.4f}, Log-Scale R2: {log_r2:.4f}")
  • # 2. Invert predictions back to original dollar scale: y = exp(z) - 1
  • inverted_df = scored_df.withColumn(
  • "actual_price", F.expm1(F.col("log_price"))
  • ).withColumn(
  • "predicted_price", F.expm1(F.col("prediction"))
  • )
  • # 3. Evaluate metrics on True Original Dollar Scale
  • evaluator_dollar_rmse = RegressionEvaluator(
  • labelCol="actual_price",
  • predictionCol="predicted_price",
  • metricName="rmse"
  • )
  • evaluator_dollar_mae = RegressionEvaluator(
  • labelCol="actual_price",
  • predictionCol="predicted_price",
  • metricName="mae"
  • )
  • dollar_rmse = evaluator_dollar_rmse.evaluate(inverted_df)
  • dollar_mae = evaluator_dollar_mae.evaluate(inverted_df)
  • print(f"True Dollar Scale RMSE: ${dollar_rmse:,.2f}")
  • print(f"True Dollar Scale MAE: ${dollar_mae:,.2f}")