2.7 One-Hot Encoding and When It Is or Is Not Appropriate

Databricks Certified Machine Learning Associate Study Guide Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Databricks Machine Learning (38% of the Exam)

1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion

2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)

2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate

3 Chapter 3: Model Development (31% of the Exam)

3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff

4 Chapter 4: Model Deployment (12% of the Exam)

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

2.7 One-Hot Encoding and When It Is or Is Not Appropriate

Key Takeaways Explain This Topic AI

One-hot encoding represents a nominal feature as binary indicator columns, imposing no artificial ordering between levels.

It is appropriate for low-to-moderate cardinality nominal features feeding linear, logistic, distance-based, or neural models.

It is a poor fit for high-cardinality features, where it explodes dimensionality, and for tree ensembles, where sparse binary splits weaken each tree.

Ordinal features should keep their order via integer encoding; one-hot encoding discards the ranking information the model could use.

In Spark ML the sequence is StringIndexer → OneHotEncoder → VectorAssembler , with handleInvalid="keep" protecting inference against unseen categories.

Last updated: August 2026

2.7 One-Hot Encoding and When It Is or Is Not Appropriate

Raw enterprise data in the Lakehouse is rarely formatted as pure numerical vectors ready for estimator ingestion. Categorical variables (such as zip codes, device types, education levels, and transaction statuses) must be mathematically transformed into numeric representations, and continuous numeric features of varying scales must be standardized or normalized.

Mastering feature transformation pipelines—and understanding how they behave in distributed Spark ML vs. single-node scikit-learn environments—is a critical domain on the Databricks Machine Learning Associate exam.

Categorical Data Taxonomy: Nominal vs. Ordinal

Before selecting an encoding strategy, the data scientist must identify whether categorical features possess an inherent mathematical ordering: CATEGORICAL DATA TYPES

• │
• ┌────────────────────────────┴────────────────────────────┐
• ▼ ▼
• NOMINAL ORDINAL
• (No Natural Order) (Explicit Hierarchy)
• State: [CA, NY, TX] • Education: [High School < BS < MS < PhD]
• Device: [iOS, Android, Web] • Customer Tier: [Bronze < Silver < Gold]
• Encoding: One-Hot Encoding, • Encoding: Ordinal Integer Mapping
• Target Encoding, Frequency Encoding (e.g., Bronze=1, Silver=2, Gold=3)

Exam Trap : Applying simple integer/label encoding (e.g., Red=1, Green=2, Blue=3 ) to nominal variables imposes an artificial linear hierarchy. A linear regression model would incorrectly assume that Blue is three times the value of Red and that Green is the exact mathematical midpoint. Use One-Hot Encoding for nominal data.

Categorical Encoding Techniques & Tradeoffs

Encoding Technique Best Suited For Spark ML Transformer Advantages Drawbacks / Risks

One-Hot Encoding (OHE) Low-to-moderate cardinality nominal variables ( < 50 < 50 < 50 distinct categories) OneHotEncoder (after StringIndexer ) No artificial ordering imposed; preserves distinct category identities. High cardinality causes dimensionality explosion and extreme memory sparsity.

Ordinal / Integer Encoding Explicitly ordered ordinal variables Custom mapping or StringIndexer Preserves ordering; compact single-column representation without expanding dimensionality. Imposes linear spacing between ordinal ranks that may not reflect true intervals.

Target / Impact Encoding High-cardinality nominal variables (e.g., zip codes, cities) Custom PySpark aggregation or category_encoders Replaces category with mean target value; maintains single-column dimensionality. Severe risk of target leakage and overfitting without k-fold out-of-fold smoothing.

Frequency / Count Encoding High-cardinality features Custom PySpark groupBy().count() Replaces category with its occurrence count/frequency; captures prevalence. Different categories with identical frequencies receive identical encoded values.

Top-K + Other Grouping Long-tailed nominal categories Custom PySpark when().otherwise() Retains top K K K most frequent categories and consolidates the long tail into "Other" . Discards granular category distinctions in the tail.

• When One-Hot Encoding Is the Wrong Tool
• The exam asks explicitly about the model types and datasets where one-hot encoding is

inappropriate. There are four recognised cases.

• High-cardinality features
• A zip_code column with 40,000 levels becomes 40,000 sparse columns. Memory and
• training time grow with the level count, most columns are almost always zero, and each
• individual indicator carries too little signal to be selected by a model. Prefer target

encoding with out-of-fold smoothing, frequency encoding, hashing, or a top-K plus "Other" grouping.

• Ordinal features
• Bronze < Silver < Gold carries an ordering that a model can exploit with a single
• integer column. One-hot encoding erases that ordering and spends three columns to

convey less information.

• Tree-based models
• A decision tree splits one column at a time. With one-hot encoding, a single
• informative categorical feature becomes many binary columns, and each split can only
• separate one level from the rest — so the tree needs greater depth to express what a
• single multi-way categorical split would capture, and under column subsampling
• ( colsample_bytree , max_features ) the individual indicators are often not even
• sampled. LightGBM and CatBoost handle categorical columns natively and generally do

better with the raw or index-encoded column.

• Features that are really identifiers
• A near-unique column such as order_id produces one indicator per row. It cannot

generalise and should be dropped, not encoded.

Model family One-hot encoding?

• Linear / logistic regression, Ridge, Lasso Yes — required; integer codes would imply a false ordering
• k-NN, K-Means, SVM Yes — distances need each level to be equidistant
• Neural networks Yes for low cardinality; embeddings for high cardinality
• Random forest, Spark GBT Workable, but index encoding is usually preferable
• LightGBM, CatBoost No — use native categorical support
• Distributed Categorical Pipelines in PySpark ML

In Apache Spark ML, categorical transformation is typically executed in three sequential stages:

StringIndexer : Maps string categorical values to numerical index columns ordered by category frequency (the most frequent category receives index 0.0 ).

OneHotEncoder : Converts category indices into binary vector representations ( SparseVector ).

VectorAssembler : Combines multiple numeric features and one-hot encoded vectors into a single unified features vector column required by Spark ML estimators. [Raw Strings] ──> StringIndexer ──> [Category Indices] ──> OneHotEncoder ──> [Sparse Vectors] ──> VectorAssembler ──> [features Vector]

Handling Unseen Categories at Inference Time ( handleInvalid )

When a model deployed in production encounters a category string that was never observed during training, StringIndexer must know how to respond. The handleInvalid parameter controls this behavior:

handleInvalid="error" (Default in some versions) : Throws a runtime JVM exception and halts execution. Unsafe for real-time production pipelines.

handleInvalid="skip" : Filters out and discards the entire row containing the invalid/unseen category. Useful for batch training, but dangerous in real-time inference endpoints.

handleInvalid="keep" (Recommended for Production) : Maps all unseen/invalid categories into a dedicated extra bucket (index equal to the number of distinct training categories). This allows downstream one-hot encoders and models to score new categories gracefully.

• Complete End-to-End PySpark ML Pipeline from pyspark.ml import Pipeline
• from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler

• categorical_cols = ["contract_type", "payment_method", "internet_service"]
• numeric_cols = ["age", "tenure_months", "monthly_charges"]

• # Step 1: Index categorical strings to frequency-ranked numbers
• indexers = [
• StringIndexer(
• inputCol=c,
• outputCol=f"{c}_idx",
• handleInvalid="keep", # Keep unseen categories in separate bucket
• stringOrderType="frequencyDesc" # Most frequent category = index 0.0
• )
• for c in categorical_cols
• ]

• # Step 2: One-Hot Encode indexed columns into SparseVectors
• encoder = OneHotEncoder(
• inputCols=[f"{c}_idx" for c in categorical_cols],
• outputCols=[f"{c}_vec" for c in categorical_cols],
• dropLast=True, # Drop last category to avoid dummy variable trap
• handleInvalid="keep"
• )

• # Step 3: Assemble all continuous and encoded vector columns into single features column
• assembler = VectorAssembler(
• inputCols=numeric_cols + [f"{c}_vec" for c in categorical_cols],
• outputCol="features",
• handleInvalid="keep"
• )

• # Step 4: Construct and fit the unified Pipeline
• pipeline = Pipeline(stages=indexers + [encoder, assembler])
• pipeline_model = pipeline.fit(train_df)

• # Transform train and test splits
• train_prepared = pipeline_model.transform(train_df)
• test_prepared = pipeline_model.transform(test_df)

The dropLast Parameter : In PySpark's OneHotEncoder , dropLast=True (default) drops the last category entry, producing a vector of length k − 1 k-1 k − 1 for k k k categories. This prevents the dummy variable trap (exact multi-collinearity) in unregularized linear models. For tree-based models or regularized linear models, dropLast=False can be used.

Feature Scaling Techniques & Formulations

Continuous features often exhibit drastically different numerical ranges (e.g., age between 18 and 90 vs. annual_income between $20,000 and $500,000). Features with large magnitudes can dominate distance metrics and gradient steps. Raw Features: Age: [18 ── 90] Income: [$20,000 ──────────── $500,000]

• StandardScaler: Age: [-1.8 ── +2.1] (mean=0, std=1) Income: [-1.5 ── +3.2] (mean=0, std=1)
• MinMaxScaler: Age: [0.0 ── 1.0] Income: [0.0 ──────────────── 1.0]
• RobustScaler: Age: [-1.2 ── +1.5] (median/IQR) Income: [-0.9 ── +2.4] (median/IQR)
• Mathematical Formulations

Standardization ( StandardScaler ) : z = x − μ σ z = \frac{x - \mu}{\sigma} z = σ x − μ ​ Centers features at μ = 0 \mu = 0 μ = 0 with unit variance σ = 1 \sigma = 1 σ = 1 . Does not bound values into a fixed range. Outliers are preserved. PySpark Note : StandardScaler(withMean=True, withStd=True) . Setting withMean=True on sparse vector inputs converts them to dense vectors, which can consume massive memory.

Min-Max Normalization ( MinMaxScaler ) : x s c a l e d = x − x m i n x m a x − x m i n × ( max − min ) + min x_{scaled} = \frac{x - x_{min}}{x_{max} - x_{min}} \times (\text{max} - \text{min}) + \text{min} x sc a l e d ​ = x ma x ​ − x min ​ x − x min ​ ​ × ( max − min ) + min Compresses all feature values strictly into [ 0.0 , 1.0 ] [0.0, 1.0] [ 0.0 , 1.0 ] (or custom bounds). Highly sensitive to extreme outliers, which compress standard observations into a narrow band.

Robust Scaling ( RobustScaler ) : x s c a l e d = x − median IQR x_{scaled} = \frac{x - \text{median}}{\text{IQR}} x sc a l e d ​ = IQR x − median ​ Uses median and Interquartile Range ( Q 3 − Q 1 Q3 - Q1 Q 3 − Q 1 ). Resilient to outliers because centering and scaling metrics are not influenced by extreme tails.

Algorithm Scaling Sensitivity Comparison

Algorithm Family Requires Feature Scaling? Operational Reason

k-Nearest Neighbors (k-NN) Mandatory Computes Euclidean distance ∑ ( x a − x b ) 2 \sqrt{\sum (x_a - x_b)^2} ∑ ( x a ​ − x b ​ ) 2 ​ . Unscaled features with large ranges dominate neighbor selection entirely.

K-Means Clustering Mandatory Cluster centroids are calculated using Euclidean distance. Unscaled features warp spherical cluster geometries into elongated ellipsoids.

Support Vector Machines (SVM) Mandatory Maximizes the margin separating support vectors. Distance to the separating hyperplane is distorted by unscaled features.

Regularized Linear Models (Lasso L 1 L_1 L 1 ​ , Ridge L 2 L_2 L 2 ​ ) Mandatory Penalty terms λ ∑ ∣ w j ∣ \lambda \sum \lvert w_j \rvert λ ∑ ∣ w j ​ ∣ or λ ∑ w j 2 \lambda \sum w_j^2 λ ∑ w j 2 ​ penalize all weights equally. Unscaled features receive artificially small weights and escape penalization.

Neural Networks (MLP / Deep Learning) Mandatory Accelerates gradient descent convergence; prevents vanishing/exploding gradients in activation functions.

Unregularized Linear / Logistic Regression (OLS) Recommended (Not strictly mandatory) Scale does not change the optimal analytical hyperplane, but scaling dramatically improves gradient descent optimization speed.

Decision Trees, Random Forests, XGBoost, LightGBM Scale-Invariant (Not Required) Splits evaluate only the relative ordering of values along one feature at a time. Multiplying or shifting feature values does not alter optimal split points.

Loading diagram... PySpark ML Feature Transformation and Vector Assembly Pipeline