Study guide
2.3 Comparing Two Categorical or Two Continuous Features
Databricks Certified Machine Learning Associate Study Guide Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Databricks Machine Learning (38% of the Exam)
1.1 MLOps Best Practices on Databricks 1.2 Advantages of Databricks ML Runtimes and Compute Selection 1.3 Promoting Code vs. Promoting Models 1.4 How AutoML Facilitates Model and Feature Selection 1.5 AutoML Advantages in the Model Development Process 1.6 Feature Tables in Unity Catalog vs. the Workspace Feature Store 1.7 Creating a Feature Table and Writing Data to It 1.8 Training a Model with Features from a Feature Table 1.9 Scoring a Model with Features from a Feature Table 1.10 Online vs. Offline Feature Tables 1.11 Manually Logging Metrics, Artifacts, and Models in an MLflow Run 1.12 Identifying the Best Run with the MLflow Client API 1.13 Information Available in the MLflow UI 1.14 Registering a Model in the Unity Catalog Registry 1.15 Model Tags, Aliases, and Champion/Challenger Promotion
2 Chapter 2: Data Processing and ML Workflows (19% of the Exam)
2.1 Computing Summary Statistics on a Spark DataFrame 2.2 Creating Visualizations for Categorical and Continuous Features 2.3 Comparing Two Categorical or Two Continuous Features 2.4 Removing Outliers with Standard Deviation and IQR 2.5 When Log-Scale Transformation Is Appropriate 2.6 Comparing and Applying Mean, Median, and Mode Imputation 2.7 One-Hot Encoding and When It Is or Is Not Appropriate
3 Chapter 3: Model Development (31% of the Exam)
3.1 Selecting the Appropriate Algorithm for a Scenario 3.2 Methods to Mitigate Data Imbalance in Training Data 3.3 Comparing Estimators and Transformers 3.4 Developing a Training Pipeline 3.5 Tuning Hyperparameters with Hyperopt's fmin 3.6 Grid, Random, and Bayesian Search Strategies 3.7 Parallelizing Single-Node Models for Hyperparameter Tuning 3.8 Cross-Validation vs. a Train-Validation Split 3.9 Counting the Models Trained by Grid Search and Cross-Validation 3.10 Using Common Classification Metrics 3.11 Using Common Regression Metrics 3.12 Choosing the Most Appropriate Metric for the Objective 3.13 Exponentiating Log-Transformed Predictions Before Evaluation 3.14 Model Complexity and the Bias-Variance Tradeoff
4 Chapter 4: Model Deployment (12% of the Exam)
4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving 4.2 Performing Batch Inference with pandas and Spark UDFs 4.3 Streaming Inference with Delta Live Tables 4.4 Deploying a Custom Model to a Model Endpoint 4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints
2.3 Comparing Two Categorical or Two Continuous Features
Key Takeaways Explain This Topic AI
Two continuous features are compared with a correlation coefficient: Pearson for linear association, Spearman for any monotonic association including non-linear ones.
Two categorical features are compared with a contingency table ( df.stat.crosstab ) and a chi-square test of independence; Cramér's V turns the chi-square statistic into a 0–1 effect size.
df.stat.corr(a, b) computes a single pairwise correlation, and pyspark.ml.stat.Correlation.corr computes a full matrix over an assembled vector column.
pyspark.ml.stat.ChiSquareTest.test(df, 'features', 'label') returns the p-value, degrees of freedom, and statistic for each feature against a categorical label.
A correlation near zero rules out a linear relationship, not a relationship — always confirm with a scatter plot before dropping a feature.
Last updated: August 2026
- 2.3 Comparing Two Categorical or Two Continuous Features
- Comparing two features means quantifying whether knowing one tells you anything about
the other. The right statistic is determined entirely by the pair of types involved, and picking the wrong one is the mistake the exam is testing.
Feature A Feature B Appropriate method PySpark implementation
Continuous Continuous Pearson correlation (linear), Spearman correlation (monotonic), scatter plot df.stat.corr , Correlation.corr
Categorical Categorical Contingency table, chi-square test of independence, Cramér's V df.stat.crosstab , ChiSquareTest.test
Categorical Continuous Grouped summary statistics, grouped box plot, ANOVA F-test groupBy().agg() , ChiSquareTest after bucketing
Two Continuous Features
Pearson's r r r measures the strength and direction of a linear relationship on [ − 1 , 1 ] [-1, 1] [ − 1 , 1 ] . Spearman's ρ \rho ρ is Pearson's correlation computed on the ranks, so it
- detects any monotonic relationship and is robust to outliers and to non-linear but
- order-preserving transforms. from pyspark.ml.feature import VectorAssembler
- from pyspark.ml.stat import Correlation
- # Single pair
- print(df.stat.corr("monthly_charges", "total_charges")) # Pearson
- print(df.stat.corr("monthly_charges", "total_charges", method="spearman"))
- # Full matrix over several numeric columns
- numeric = ["tenure_months", "monthly_charges", "total_charges", "support_tickets"]
- vec_df = VectorAssembler(inputCols=numeric, outputCol="features").transform(df)
- pearson_matrix = Correlation.corr(vec_df, "features", "pearson").head()[0]
- spearman_matrix = Correlation.corr(vec_df, "features", "spearman").head()[0]
- print(pearson_matrix.toArray())
- Choosing between them
- Situation Use
- Both features roughly linear and outlier-free Pearson
- Relationship is monotonic but curved (e.g. exponential growth) Spearman
- Heavy outliers, or one feature is an ordinal rank Spearman
- You intend to fit a linear model on the raw features Pearson — it measures exactly what the model can exploit
- The trap : r ≈ 0 r \approx 0 r ≈ 0 does not mean "no relationship". A perfectly symmetric
- U-shaped relationship has a Pearson correlation of approximately zero while being
completely deterministic. Confirm with a scatter plot before dropping a feature.
Two Categorical Features
Start with the contingency table — the joint counts of every level pair: crosstab = df.stat.crosstab("contract_type", "payment_method")
display(crosstab)
Then test whether the two are independent. The chi-square test of independence compares observed cell counts against the counts expected if the features were
unrelated: χ 2 = ∑ i , j ( O i j − E i j ) 2 E i j , E i j = R i C j N \chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}}, \qquad E_{ij} = \frac{R_i C_j}{N} χ 2 = i , j ∑ E ij ( O ij − E ij ) 2 , E ij = N R i C j
- A small p-value means the features are not independent — knowing one shifts the
- distribution of the other. from pyspark.ml.feature import StringIndexer, VectorAssembler
- from pyspark.ml.stat import ChiSquareTest
- indexed = StringIndexer(inputCols=["contract_type", "payment_method"],
- outputCols=["contract_idx", "payment_idx"]).fit(df).transform(df)
- vec = VectorAssembler(inputCols=["contract_idx"], outputCol="features").transform(indexed)
- result = ChiSquareTest.test(vec, "features", "payment_idx").head()
- print("p-values:", result.pValues)
- print("degrees of freedom:", result.degreesOfFreedom)
- print("statistics:", result.statistics)
- Statistical significance is not effect size
- On a 50-million-row table, almost every chi-square test returns a p-value near zero,
because the test's power grows with N N N . Report an effect size alongside it — Cramér's V rescales the statistic to [ 0 , 1 ] [0, 1] [ 0 , 1 ] : V = χ 2 N ⋅ min ( r − 1 , c − 1 ) V = \sqrt{\frac{\chi^2}{N \cdot \min(r - 1,\; c - 1)}} V = N ⋅ min ( r − 1 , c − 1 ) χ 2
where r r r and c c c are the number of levels in each feature. Values near 0 mean the
association is negligible regardless of how small the p-value is.
- Cell-count requirement
- The chi-square approximation degrades when expected cell counts are very small (the
- conventional guidance is at least 5 per cell). With many rare levels, collapse the tail
into an "Other" bucket before testing.
Detecting Multi-Collinearity & Feature Interactions
Multi-collinearity occurs when two or more independent predictor features are highly linearly correlated. While tree-based ensembles (such as Random Forests and XGBoost) are relatively resilient to collinearity during inference, linear and logistic regression models suffer from inflated standard errors, unstable coefficient weights, and degraded interpretability.
Computing Pearson Correlation in PySpark
To compute the pairwise Pearson correlation coefficient between two numeric columns in PySpark without collecting the entire dataset to the driver: # Distributed pairwise Pearson correlation calculation
- corr_charges = df_features.stat.corr("monthly_charges", "total_charges")
- print(f"Pearson Correlation: {corr_charges:.4f}")
- # Cross-tabulation for categorical feature interaction
- crosstab_df = df_features.stat.crosstab("contract_type", "churn_label")
- display(crosstab_df)
- Full Correlation Matrix via pandas API on Spark
When exploring dozens of features, computing a full correlation matrix is streamlined using the pyspark.pandas (formerly Koalas) API, which scales across Spark worker nodes: import pyspark.pandas as ps
- # Convert Spark DataFrame to distributed pandas-on-Spark DataFrame
- ps_df = df_features.select("tenure_months", "monthly_charges", "total_charges").pandas_api()
- # Compute distributed correlation matrix
- corr_matrix = ps_df.corr(method="pearson").to_pandas()
- print(corr_matrix)
- Target Distribution Analysis: Imbalance & Skewness
Diagnosing the distribution of the target variable is one of the most critical steps in the EDA phase, as it dictates the entire downstream modeling strategy.
Classification Target Imbalance
In classification tasks (e.g., fraud detection, anomaly detection, churn prediction), extreme class imbalance (such as 99% majority vs. 1% minority) causes standard accuracy metrics to become deceptive. An algorithm predicting the majority class exclusively would achieve 99% accuracy while exhibiting 0% recall on the minority target.
During EDA, if class proportions diverge significantly from a 50/50 or 60/40 ratio, the data scientist must plan for:
Utilizing Precision-Recall AUC (PR-AUC) , F1-Score , or Balanced Accuracy rather than ROC-AUC or standard classification accuracy.
Applying class-weight balancing (e.g., scale_pos_weight in XGBoost/LightGBM or weightCol in Spark ML).
Evaluating oversampling (SMOTE) or stratified sampling techniques during train/validation splitting.
Regression Target Skewness
In regression tasks (e.g., home price prediction, lifetime customer value, transaction volume), target distributions are frequently right-skewed with long positive tails. Severe skewness violates the normality assumptions of linear regression residuals and causes gradient descent updates to be dominated by extreme values.
EDA visual inspection via histograms and box plots reveals whether power transformations (such as ln ( y + 1 ) \ln(y+1) ln ( y + 1 ) ) should be applied prior to model fitting.
Pre-Modeling Data Quality Assessment Checklist
Quality Dimension Common Defect Observed in EDA Diagnostic Technique Planned Preprocessing Remedy
Completeness High percentage of nulls or empty strings dbutils.data.summarize() null % column Drop column if > 50 % >50\% > 50% null; impute median/mode or add indicator if < 50 % <50\% < 50% .
Validity Impossible values (e.g., age = -5 or tenure = 999 ) Histogram min/max bounds & box plots Filter invalid rows or winsorize/cap bounds.
Uniqueness / Cardinality High-cardinality categorical IDs (e.g., user_uuid treated as category) Distinct count vs. total row count Drop unique identifiers from feature set to avoid overfitting.
Consistency Inconsistent categorical strings ( "Male" , "male" , "M" ) Value frequency table in display() Standardize casing with lower(trim(col)) in PySpark.
Temporal Leakage Features populated after the target event occurs Correlation with target and temporal timestamps Remove future-dated features from training feature store.