Fourteen years of production systems has taught me that a model which looks perfect in a Jupyter notebook is usually a liability once it is live. Teams read production ML failures as modeling problems. Almost every one I have debugged turned out to be a data and time problem instead.
Ship a fraud detection system or a recommendation engine, then watch its metrics fall apart three weeks later, and you know the feeling. The dashboards are green, server latency is fine, and the business value drains away regardless. What you are looking at is a set of hidden assumptions meeting reality at 3 a.m.
The time travel trap: data leakage in plain sight
The most common cause of production ML failures is what I call “time travel.” In the lab we flatten time, joining tables from the past against outcomes from the future, and nobody notices that the model is cheating. Train a fraud model with a “chargeback_count” feature and, if that count includes a report filed after the transaction, the model is reading the future.
A naive SQL join is where this usually creeps in. The model learns that users with chargebacks are risky, which is true and also useless, because at the moment you score the transaction the chargeback has not happened yet.
-- The "Naive" Approach that leaks future data
SELECT
t.transaction_id,
t.amount,
COUNT(c.id) OVER (PARTITION BY t.user_id) as chargeback_count -- DANGER: Future data leakage
FROM transactions t
LEFT JOIN chargebacks c ON t.user_id = c.user_id;
-- The "Senior" Approach (Point-in-Time Join)
SELECT
t.transaction_id,
t.amount,
(SELECT COUNT(*) FROM chargebacks c
WHERE c.user_id = t.user_id
AND c.created_at < t.created_at) as valid_chargeback_count -- Accurate for production
FROM transactions t;My earlier guide on Drift Detection covers how to catch these discrepancies before they reach your bottom line.
When missing data becomes a signal
Missing values get treated as a hygiene task: fill them with zeros or medians and ship. In a production system, though, “missing” is rarely random. It usually encodes a status, like a new user or an inactive account.
When your pipeline returns zero for avg_spend_last_30_days because a user has no history, the model does not read that as missing data. It reads it as a signal. If new users happen to be lower risk during your training window, the model learns that zero history means safe. Then a downstream service times out, returns zeros for active users, and the model cheerfully approves every high-risk transaction as a safe newcomer. Absence has to be encoded separately from value when you build the features.
Population shift: when the statistics look fine
Most monitoring watches distribution shift, the classic covariance shift, using histograms and Kolmogorov-Smirnov tests. The sneakier cause of production ML failures is population shift with no distribution shift at all.
The numbers look the same while the people behind them are not. Expand a store from New York to London: a $200 transaction can land in the same spot in both distributions even though the risk profile of the two cohorts has nothing in common. A model that was never taught to care about cohort context will apply New York logic to London users and get it wrong. Huyen Chip’s write-up on monitoring data shifts is worth the read here.
A shift like this never shows up on a dev dashboard the way an ordinary bug does. You have to watch performance per segment rather than an aggregate AUC curve. My notes on handling covariance shift cover the fast version.
If production ML failures are eating your dev hours, I can take it on. I have been working on WordPress and awkward backend integrations since the 4.x days.
Design for when the data arrives
Strong offline metrics only prove that the model fits the assumptions you handed it. The real work starts when those assumptions meet live traffic, so design around when information actually arrives and how it changes afterwards. Skip that and the model keeps fitting the past while missing the present. Ship it anyway, with your eyes open.
