Model Diagnosis, Bias-Variance Tradeoff & Hyperparameter Tuning

Explanation

A fundamental objective of machine learning is preventing models from overfitting training noise. High bias is diagnosed when learning curves plateau with high training and cross-validation errors (remedied by adding features, using non-linear models, or reducing regularization). High variance occurs when a large gap exists between training error and validation error (remedied by gathering more training samples, removing redundant features, or increasing regularization strength). In classification, when classes are imbalanced (e.g., fraud detection), raw accuracy is misleading; models must be evaluated using Precision = TP / (TP + FP) and Recall = TP / (TP + FN).

Key points

• High Bias (Underfitting): Model is overly simplistic and fails to capture underlying data patterns; poor train and test accuracy.
• High Variance (Overfitting): Model memorizes training noise and generalizes poorly; high train accuracy but poor test accuracy.
• Regularization techniques: L1 (Lasso) promotes feature sparsity by adding lambda * sum(|w|); L2 (Ridge) shrinks weights toward zero by adding lambda * sum(w^2).
• Evaluation Metrics: Precision, Recall, F1-Score, ROC curve (Receiver Operating Characteristic), and Area Under the Curve (AUC).
• K-Fold Cross-Validation splits data into K subsets to ensure robust performance estimation without data leakage.

Example

Code examples

from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge

# Hyperparameter Tuning using Grid Search
param_grid = {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0]}
grid = GridSearchCV(Ridge(), param_grid, cv=5, scoring='neg_mean_squared_error')
# grid.fit(X_train, y_train)

Exam questions

• [8 Marks] Explain the Bias-Variance tradeoff with appropriate learning curves. How do L1 (Lasso) and L2 (Ridge) regularization mitigate variance?
• [7 Marks] Differentiate between Precision and Recall. In which practical scenarios is Recall prioritized over Precision?