The advice most beginners get in credit risk modeling is short: train an algorithm, look at the AUC or the Gini coefficient, ship it. It leaves out credit scoring model stability, which is the part that decides whether the model survives contact with production. I have watched models that looked excellent in a Jupyter notebook fall apart on real data because the variables underneath them were not stable over time.
Performance and reliability are different questions. If your variables do not tell a consistent story about risk, the score they produce is not worth much no matter how it benchmarks. Two things are worth validating before you get anywhere near final estimation: monotonicity and stability.
Monotonicity is the first check worth running
Monotonicity is a formal word for asking whether a variable makes sense. Every variable in a credit score should have a logical risk direction. As a person’s income rises, their probability of default should generally fall, which is a negative risk direction. If the data says higher earners are more likely to default, something is off, either in data quality or in how the sample was drawn.
During selection, person_age is the variable that misbehaves most often for me. The very young and the very old are expected to carry higher risk, and the middle of the range frequently shows no clear trend at all. When a variable inverts its risk direction from one year to the next, I cut it. You cannot refactor a bad foundation.
Discretizing variables for analysis
To study this, bin the continuous variables (terciles are usually enough) and calculate the empirical default rate for each bin. This is how I normally do it in Python with Pandas:
import pandas as pd
def bbioon_analyze_monotonicity(df, variable, target, bins=3):
# Create quantiles
df['bin'] = pd.qcut(df[variable], q=bins, duplicates='drop')
# Calculate default rate per bin
analysis = df.groupby('bin')[target].mean().reset_index()
analysis.columns = ['Bin Range', 'Default Rate']
return analysis
# Example usage for income
# income_risk = bbioon_analyze_monotonicity(train_df, 'person_income', 'default')
# print(income_risk)Measuring credit scoring model stability with PSI
Once the risk direction holds, the next question is whether the variables stay put over time and across your train, test and out-of-time samples. That is what the Population Stability Index (PSI) measures. It is the industry standard here because it puts one number on a distributional shift using a logarithmic formula.
It compares the share of observations in each bin between a reference dataset and a target one. Below 0.10, you are fine. Between 0.10 and 0.25, keep watching it. Above 0.25, the population has changed enough that the model may be out of date before it ever ships.
On the data side, I have also written about credit scoring EDA and robust credit scoring models with Python.
Implementing the PSI formula in Python
This function calculates the shift directly instead of leaving it to judgement. It uses the standard PSI formula from the credit risk modeling documentation:
import numpy as np
def calculate_psi(expected, actual, buckets=10):
def scale_to_proportions(arr):
return arr / np.sum(arr)
# Calculate proportions
expected_percents = scale_to_proportions(np.histogram(expected, bins=buckets)[0])
actual_percents = scale_to_proportions(np.histogram(actual, bins=buckets)[0])
# Avoid division by zero
expected_percents = np.where(expected_percents == 0, 0.0001, expected_percents)
actual_percents = np.where(actual_percents == 0, 0.0001, actual_percents)
# PSI Formula: sum((actual - expected) * ln(actual / expected))
psi_value = np.sum((actual_percents - expected_percents) * np.log(actual_percents / expected_percents))
return psi_valueIf this kind of validation work is eating your dev hours, I can take it on. I have been working with WordPress, Python integrations and messy data logic since the 4.x days.
Before you ship
Most of the real work in a scoring model happens in validation rather than training. Checking monotonicity and stability is what keeps the result interpretable and keeps it working a year later. A good algorithm will happily hide a bad variable from you. Run the PSI numbers, confirm the risk directions, then ship.
