If you validate a model on WooCommerce data with a shuffled split, the score you report is probably fiction. Neighbors leak. The same customer and the same promo week end up on both sides of the split, and the model passes a test it has already taken. The idea comes from a geospatial piece by Jessé Burlamaque on Towards Data Science about small data machine learning for Amazon rainforest mapping, where a single field plot costs as much as the computer you’d train on. Swap forest plots for labeled orders and the same failure sits in your order table.
Autocorrelation is not a forest problem. Any store with repeat customers has it.
Why the score is fiction
In the article, nearby field plots share environment, history and sensor artifacts, so random cross-validation inflates the metrics. Store data has the same shape. Two orders from one customer share the address, the device, the price tolerance and the coupon history. Split them randomly between train and test, and the model memorizes the customer instead of the signal. The article puts it plainly:
in reality, it is simply interpolating within a neighborhood already very similar to what it saw during training
The second lie is the sample count. The source treats 100 to 200 samples as comfortable only until the map splits into strata, and orders behave the same way. Three hundred labeled orders across fourteen categories and five countries is several small datasets wearing the same table name, and some of them have twelve rows. That’s the mechanism behind why high accuracy lies on store models.
GroupKFold vs TimeSeriesSplit
The two splits answer different questions. GroupKFold, with groups set to the customer ID, keeps every order from one customer inside a single fold, so the test set contains only customers the model has never seen. That is what you want for churn risk, lifetime value or segmentation. TimeSeriesSplit trains on the past and tests only on the future, which is what demand forecasting and restock planning get judged on. The scikit-learn docs are blunt about why it exists: other cross-validation methods would train on future data.
Which one you pick depends on the store. A subscription shop, or any store with a repeat-purchase core, gets GroupKFold first, because customer leakage is what inflates its numbers the most. For a store selling one-off items with heavy seasonality, gifts or seasonal apparel, time is the axis the model will be judged on, so TimeSeriesSplit comes first. When you have enough data for both, run both. A large gap between the two scores is the most useful finding in the project.
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GroupKFold, cross_val_score
cv = GroupKFold(n_splits=5)
scores = cross_val_score(
RandomForestRegressor(n_estimators=300, min_samples_leaf=5),
X, y,
cv=cv,
groups=orders["customer_id"],
)
print(scores.mean(), scores.std())With few labels, the source argues for tree models with honest regularization over bigger architectures, and that carries over untouched. Shallow trees, a real min_samples_leaf, and a lean feature set where each column means something in the business (category, price band, days since last order) beat fifty near-duplicate derivatives. Small datasets punish feature pileups.
If you have a store sitting on a few hundred labeled orders and a model someone quoted a number for, I’m glad to check the split before anyone builds on that number.
Averages hide the segment you care about
The strata argument matters most for store owners. A global metric blends the category with two hundred labeled orders into the country with twelve, and the average looks fine while the thin segment is guesswork. As the source puts it, “a good global average does not guarantee stable behavior across all parts of the map.” On a store, the map is categories, countries and price bands.
So report per segment and keep the error with the prediction. Store the out-of-fold error per segment in a custom table or in meta, and let the admin screen say “predicted 340 units, reliable” and “predicted 340 units, we have twelve orders like this one”. That is the article’s uncertainty map translated for a store, and it pairs with how counts mislead in WooCommerce order analysis.
What I’d do on a client site this week
Re-split by customer and by time, run both, and put the honest number beside the old one. If it still earns its place, ship it with per-segment error attached. If it collapses, better to learn that this week than after the client has acted on it.
