Drift detection is the part of deployment most guides skip. The advice usually stops at the ship button: the model is live, the API answers in under 100ms, done. That is roughly when it starts getting worse, and nothing in your dashboard says so.
In WordPress work I worry about race conditions and stale transients. With a deployed model the worry is the world moving while the weights sit still. A recommender that pushes winter coats in July has drifted. Drift detection is just the practice of noticing when production data stops looking like training data.
Data drift and concept drift are different problems
Which one you have decides what you do about it, so it is worth separating the inputs changing from the underlying relationship changing.
- Data drift is a change in the distribution of your features, P(X). Picture a WooCommerce store where traffic suddenly arrives from a demographic that was barely in the training set. The model still knows how to predict; it is being asked about values it never saw.
- Concept drift is the worse case, P(y|X). The input still looks familiar but its meaning moved. A pattern that meant a legitimate transaction last month can mean fraud this month because whoever is committing it changed tactics.
If the input side is what is shifting on you, I wrote about handling covariance shift separately. It often shows up before the model fails outright.
Two statistical tests worth running
You cannot eyeball drift. I have watched teams compare weekly averages and call it monitoring, which hides everything happening in the tails of the distribution. Two tests do the job better.
1. Kolmogorov-Smirnov (K-S) test
The K-S test is the usual first pick for a single numeric feature. It measures the largest gap between the cumulative distribution function of your reference sample and your live sample. A small p-value says the two samples probably did not come from the same distribution.
from scipy import stats
def bbioon_detect_drift(reference_data, live_data):
# Perform the two-sample K-S test
statistic, p_value = stats.ks_2samp(reference_data, live_data)
if p_value < 0.05:
return "Drift Detected: Distribution shift is statistically significant."
return "Status: Stable."2. Population stability index (PSI)
PSI is less twitchy than K-S about a few stray outliers. It buckets both samples into bins and compares the proportion in each one. The convention I have always worked to is that anything above 0.25 means retrain rather than keep watching.
Put the check in the pipeline, not in your calendar
The failure that convinced me was a model whose labels arrived days late. The performance metrics looked fine because they were still scoring old data, while the live predictions had already gone bad. That is the case for running drift detection in the same pipeline that does your deploys, not as a task someone remembers on a Friday.
On a WordPress project that usually means a WP-CLI command that calls a Python script against the database logs. A non-zero exit code fires a webhook into Slack, and if it is bad enough the site falls back to the old rule-based path until someone looks at it.
If the vocabulary here is new, my post on AI vs machine learning sorts out which metrics belong to which kind of system.
If drift monitoring is eating your week, I take this kind of work on as a contract. I have been building on WordPress since the 4.x days.
What to do on Monday
Pick your two or three most important features and put a K-S test on each of them against a frozen reference sample. For anything with a lot of correlated inputs, reconstruction error from an autoencoder catches shifts that per-feature tests miss. Rough monitoring beats none, and the alternative is finding out from the revenue chart.
