Scoring models deserve a rant. Somewhere along the way the standard advice became chasing AUC on a single training set, and that habit is exactly what breaks models in production. I have taken handoff of too many “perfect” models that fell apart in their first week inside a high-volume WooCommerce install.
The cause is nearly always weak Robust Variable Selection. A variable that only predicts well on your particular training slice is a liability rather than an asset. What follows is the selection logic I use so a model still works once the data gets messy.
Stability matters more than raw performance
A variable is robust only if it stays significant across every subset of your data, and testing on the full dataset hides that. So we split the training data into four folds with StratifiedKFold. Each fold then carries the same share of defaults as the whole population.
from sklearn.model_selection import StratifiedKFold
# Initialize stratified cross-validation
skf = StratifiedKFold(n_splits=4, shuffle=True, random_state=42)
train_imputed["fold"] = -1
# Assign folds based on default year to preserve distribution
for fold, (_, test_idx) in enumerate(skf.split(train_imputed, train_imputed["def_year"])):
train_imputed.loc[test_idx, "fold"] = foldThe rule is blunt: a variable survives only if it passes our criteria on all four folds. One failure and it is out. That keeps leakage out of the pipeline and stops the model from fitting noise. If volume is your problem too, I wrote about that in scaling WordPress data.
The filter method: four rules for robust variable selection
Statistical measures of association beat complex models at the selection stage because they run fast and you can audit them. We apply four rules, in this order.
Rule 1: drop uncorrelated continuous variables
Run a Kruskal-Wallis test between each continuous variable and the default target. If the p-value climbs above 5% on a single fold, the variable is out. On a recent credit risk project all seven continuous variables, income and loan amount among them, cleared the threshold on every fold.
Rule 2: filter weak categorical links
Categorical variables are where scoring models quietly go wrong. Measure association strength with Cramér’s V and keep the threshold low, at 10%. A weak link in even one fold is enough to purge the variable. This is the step that catches variables that look fine on paper and have no temporal stability.
Rule 3: purge redundant continuous data
Multicollinearity wrecks interpretability. Use Spearman correlation, and when two variables reach 60% correlation on any fold, one of them goes. Keep whichever has the stronger link to the default, meaning the lower p-value from Rule 1. Same reasoning I used in turning WP bloat into business assets.
Rule 4: surface categorical redundancy
The last rule runs the same redundancy check on pairs of categorical variables using Cramér’s V. Above 50% association, the weaker one is removed. What you end up with is a leaner model you can explain to a stakeholder or a regulator without hedging.
If this kind of variable selection work is eating your dev hours, I can take it on. I have been working with WordPress and heavy data loads since the 4.x days.
What this buys you
Getting a model to fit is not the hard part. Getting one that stays stable is. Enforcing these statistical rules across several folds means the selection survives data it has never seen, so your production scoring still holds when the underlying population moves. That is worth more than a higher AUC on the training set.
