Study guide
3.2 Methods to Mitigate Data Imbalance in Training Data
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
3.2 Methods to Mitigate Data Imbalance in Training Data
Key Takeaways Explain This Topic AI
Imbalance is mitigated at three levels: resampling the data, weighting the loss function, and tuning the decision threshold after training.
Cost-sensitive weighting — class_weight='balanced' , XGBoost scale_pos_weight , Spark ML weightCol — changes the objective without fabricating or discarding rows.
SMOTE synthesises minority points by interpolating between nearest neighbours; it must be applied to the training fold only, never before splitting.
scale_pos_weight is the ratio of negative to positive instances, so 95,000 negatives against 5,000 positives gives 19.
Under severe imbalance, evaluate with PR-AUC or F β F_\beta F β rather than accuracy or ROC-AUC, which large true-negative counts inflate.
Last updated: August 2026
3.2 Methods to Mitigate Data Imbalance in Training Data
In enterprise machine learning applications—such as financial fraud detection, customer churn prediction, ad click-through rate modeling, and cyber threat identification—the target class distribution is often heavily skewed. Positive cases of interest may represent less than 1% (or even 0.01%) of total observations. Training models on severely imbalanced datasets without mitigation leads to catastrophic failure: standard loss functions incentivize algorithms to classify all records as the majority class to maximize raw accuracy. Addressing imbalance requires a multi-layered strategy across data resampling, algorithmic weighting, metric selection, and decision threshold optimization.
The Accuracy Paradox & Class Imbalance Dynamics
Consider a credit card fraud detection dataset with 1,000,000 transactions, where only 1,000 transactions (0.1%) are fraudulent ( y = 1 y=1 y = 1 ) and 999,000 are legitimate ( y = 0 y=0 y = 0 ). +-----------------------------------------------------------------------------+
- | THE ACCURACY PARADOX DEMONSTRATED |
- | |
- | TRIVIAL MAJORITY PREDICTOR: Predicts y = 0 for 100% of transactions |
- | |
- | Confusion Matrix: |
- | +--------------------+--------------------+ |
- | | TN = 999,000 | FP = 0 | Accuracy = 99.90% (Superb?) |
- | +--------------------+--------------------+ Precision = Undefined (0/0) |
- | | FN = 1,000 | TP = 0 | Recall = 0.00% (Useless!)|
- | +--------------------+--------------------+ F1-Score = 0.00% |
- | |
- | Business Impact: $10,000,000 in fraudulent transactions missed completely!|
- +-----------------------------------------------------------------------------+
The Accuracy Paradox occurs because standard cross-entropy and 0-1 loss functions treat every misclassification equally. A naive model that predicts the majority class for every instance achieves 99.9% accuracy while having zero operational utility.
Multi-Level Imbalance Mitigation Strategies
Imbalance mitigation can be applied at three distinct stages of the machine learning lifecycle: +-----------------------------------------------------------------------------+
- | THREE-TIER IMBALANCE MITIGATION HIERARCHY |
- | |
- | [1] DATA-LEVEL STRATEGIES (Resampling before Training) |
- | +---------------------------------------------------------------------+ |
- | | - Random Undersampling (majority class reduction) | |
- | | - Random Oversampling (minority class replication) | |
- | | - SMOTE / ADASYN (synthetic minority interpolation) | |
- | +---------------------------------------------------------------------+ |
- | |
- | v
- | [2] ALGORITHM-LEVEL STRATEGIES (Cost-Sensitive Loss Optimization) |
- | +---------------------------------------------------------------------+ |
- | | - Scikit-learn: class_weight='balanced' | |
- | | - XGBoost / LightGBM: scale_pos_weight = N_negative / N_positive | |
- | | - PySpark ML: weightCol="weight" (sample weight per row) | |
- | +---------------------------------------------------------------------+ |
- | |
- | v
- | [3] DECISION-LEVEL STRATEGIES (Post-Prediction Threshold Tuning) |
- | +---------------------------------------------------------------------+ |
- | | - Shift classification cutoff tau from 0.5 to tau* | |
- | | - Cost Matrix Optimization: Minimize C_FP * FP(tau) + C_FN * FN(tau)| |
- | | - Precision-Recall Curve Cutoff Selection | |
- | +---------------------------------------------------------------------+ |
- +-----------------------------------------------------------------------------+
- Data-Level Resampling Techniques
Random Undersampling: Randomly discards instances from the majority class until the desired ratio is reached.
Advantage: Decreases dataset size, significantly accelerating training speed.
Disadvantage: Discards potentially critical majority class boundary information, increasing model variance.
Random Oversampling: Randomly duplicates instances from the minority class.
Advantage: Preserves all majority information.
Disadvantage: Duplicating identical minority samples leads to severe overfitting on specific minority feature regions.
SMOTE (Synthetic Minority Over-sampling Technique): Synthesizes new minority instances by selecting a minority sample x i \mathbf{x}_i x i , finding its k k k -nearest minority neighbors x z i \mathbf{x}_{zi} x z i , and generating synthetic points along the connecting line segment: x new = x i + λ ( x z i − x i ) , λ ∼ Uniform ( 0 , 1 ) \mathbf{x}_{\text{new}} = \mathbf{x}_i + \lambda (\mathbf{x}_{zi} - \mathbf{x}_i), \quad \lambda \sim \text{Uniform}(0, 1) x new = x i + λ ( x z i − x i ) , λ ∼ Uniform ( 0 , 1 )
Spark Consideration: SMOTE requires distance calculations in feature space, making it expensive on distributed datasets. In Databricks, SMOTE is typically executed on sampled single-node data or partition-by-partition using Pandas UDFs.
Algorithm-Level Cost-Sensitive Learning
Cost-sensitive learning modifies the objective loss function to penalize minority class errors more heavily than majority class errors:
Scikit-learn class_weight='balanced' : Automatically assigns inverse class frequency weights: w j = N K ⋅ N j w_j = \frac{N}{K \cdot N_j} w j = K ⋅ N j N where N N N is total samples, K K K is number of classes, and N j N_j N j is samples in class j j j .
XGBoost / LightGBM scale_pos_weight : Scales the gradient and hessian of positive instances: scale_pos_weight = Number of Negative Instances Number of Positive Instances \text{scale\_pos\_weight} = \frac{\text{Number of Negative Instances}}{\text{Number of Positive Instances}} scale_pos_weight = Number of Positive Instances Number of Negative Instances
PySpark ML weightCol : Most PySpark ML classification estimators ( LogisticRegression , RandomForestClassifier ) accept a weightCol parameter pointing to a computed column in the DataFrame.
Decision Threshold Optimization & Cost Matrix Analysis
By default, binary classifiers apply a decision threshold of τ = 0.5 \tau = 0.5 τ = 0.5 to predicted probabilities: y ^ = { 1 if P ( y = 1 ∣ x ) ≥ 0.5 0 if P ( y = 1 ∣ x ) < 0.5 \hat{y} = \begin{cases} 1 & \text{if } P(y=1|\mathbf{x}) \ge 0.5 \\ 0 & \text{if } P(y=1|\mathbf{x}) < 0.5 \end{cases} y ^ = { 1 0 if P ( y = 1∣ x ) ≥ 0.5 if P ( y = 1∣ x ) < 0.5
However, τ = 0.5 \tau = 0.5 τ = 0.5 is only optimal when classes are balanced and the cost of a False Positive ( C F P C_{FP} C FP ) equals the cost of a False Negative ( C F N C_{FN} C FN ). +-----------------------------------------------------------------------------+
- | COST MATRIX & THRESHOLD OPTIMIZATION |
- | |
- | ACTUAL: Fraud (y=1) ACTUAL: Legitimate (y=0) |
- | PREDICT: Fraud (y_hat=1) TP (Cost = $0) FP (Cost = $C_FP) |
- | Transaction Caught False Alarm/Friction|
- | |
- | PREDICT: Legitimate (y_hat=0) FN (Cost = $C_FN) TN (Cost = $0) |
- | Fraud Loss Incurred Normal Transaction |
- | |
- | Total Business Cost Function: |
- | Cost(tau) = C_FP * FP(tau) + C_FN * FN(tau) |
- +-----------------------------------------------------------------------------+
- Impact of Adjusting Threshold τ \tau τ
Lowering Threshold (e.g., τ = 0.15 \tau = 0.15 τ = 0.15 ): The model predicts positive more aggressively. Recall increases (fewer False Negatives; more fraud detected), but Precision decreases (more False Positives; more false alarms). Ideal when C F N ≫ C F P C_{FN} \gg C_{FP} C FN ≫ C FP (e.g. cancer detection, severe fraud).
Raising Threshold (e.g., τ = 0.85 \tau = 0.85 τ = 0.85 ): The model predicts positive only when highly confident. Precision increases (fewer False Positives), but Recall decreases (more False Negatives). Ideal when False Alarms are extremely costly (e.g. automated account termination).
Metric Selection Under Imbalance: PR-AUC vs. ROC-AUC
Evaluating imbalanced models with ROC-AUC can be dangerously misleading:
Dimension ROC-AUC (Receiver Operating Characteristic) PR-AUC (Precision-Recall / Average Precision)
Plotted Axes True Positive Rate ( Y Y Y ) vs. False Positive Rate ( X X X ) Precision ( Y Y Y ) vs. Recall ( X X X )
FPR Formula FPR = FP FP + TN \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} FPR = FP + TN FP Precision = TP TP + FP \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} Precision = TP + FP TP
Effect of Large TN High volume of True Negatives inflates denominator, keeping FPR artificially small even with thousands of False Positives. True Negatives are not included in Precision or Recall; focuses strictly on positive class performance.
Baseline Curve Diagonal line at 0.5 0.5 0.5 (random guess). Horizontal line at positive prevalence P = N + N P = \frac{N_+}{N} P = N N + .
Recommended Use Balanced datasets or when both classes are equally important. Mandatory standard for rare event detection ( < 5 % <5\% < 5% prevalence).
- Implementation Code Examples
- PySpark ML: Class Weighting with weightCol from pyspark.sql import functions as F
- from pyspark.ml.classification import LogisticRegression
- from pyspark.ml.evaluation import BinaryClassificationEvaluator
df = spark.table("lakehouse_gold.fraud_transactions")
- # Calculate class distribution
- total_count = df.count()
- pos_count = df.filter(F.col("is_fraud") == 1).count()
- neg_count = total_count - pos_count
- # Calculate balanced class weights
- pos_weight = total_count / (2.0 * pos_count)
- neg_weight = total_count / (2.0 * neg_count)
- # Add weight column
- weighted_df = df.withColumn(
- "class_weight",
- F.when(F.col("is_fraud") == 1, pos_weight).otherwise(neg_weight)
- )
train_df, test_df = weighted_df.randomSplit([0.8, 0.2], seed=42)
- # Train cost-sensitive Logistic Regression
- lr = LogisticRegression(
- featuresCol="features",
- labelCol="is_fraud",
- weightCol="class_weight",
- maxIter=50
- )
- lr_model = lr.fit(train_df)
- predictions = lr_model.transform(test_df)
- # Evaluate with PR-AUC
- evaluator = BinaryClassificationEvaluator(
- labelCol="is_fraud",
- rawPredictionCol="rawPrediction",
- metricName="areaUnderPR"
- )
- pr_auc = evaluator.evaluate(predictions)
- print(f"Weighted Model PR-AUC: {pr_auc:.4f}")
- Scikit-learn: Cost-Based Threshold Optimization import numpy as np
- from sklearn.metrics import precision_recall_curve
- # Predicted probabilities and ground truth
- y_true = np.array([...])
- y_prob = np.array([...])
- # Business cost specification
- COST_FP = 15.0 # $15 customer service friction per false alarm
- COST_FN = 500.0 # $500 average loss per missed fraud transaction
precisions, recalls, thresholds = precision_recall_curve(y_true, y_prob)
- best_threshold = 0.5
- min_total_cost = float("inf")
for threshold in np.linspace(0.01, 0.99, 100):
y_pred = (y_prob >= threshold).astype(int)
- fp = np.sum((y_pred == 1) & (y_true == 0))
- fn = np.sum((y_pred == 0) & (y_true == 1))
total_cost = (COST_FP * fp) + (COST_FN * fn)
if total_cost < min_total_cost:
- min_total_cost = total_cost
- best_threshold = threshold
- print(f"Optimal Decision Threshold: {best_threshold:.3f}")
- print(f"Minimized Financial Cost: ${min_total_cost:,.2f}")
- Where Each Technique Belongs in the Pipeline
A recurring exam trap is applying a resampling technique in the wrong place.
Technique Applied to Never applied to
Random undersampling / oversampling The training split (or the training fold inside CV) The validation or test split — resampling them destroys the true prevalence the metric depends on
SMOTE The training fold only, after the split The full dataset before splitting; synthetic points derived from validation rows leak
Class weighting The estimator, via class_weight , scale_pos_weight , or weightCol Nothing to worry about — weighting does not touch the data
Threshold tuning Predicted probabilities, using a validation split The test split used for the final unbiased estimate
- Class weighting is the lowest-risk option because it neither discards majority rows nor
- invents minority ones; it simply tells the loss function that a minority error costs
- more. It is the technique Databricks material reaches for first, and the one that most
- often appears as the correct answer when a question asks how to "directly mitigate the
model's bias toward the majority class".