Most advice on synthetic data validation comes down to driving KL Divergence as low as it will go. That advice is how you end up with a model that scores beautifully in testing and then walks straight past edge-case fraud three months into production. I have seen pipelines that looked bulletproof, 91% TSTR accuracy and low membership inference risk, fail anyway, because the generator had quietly stripped the awkward parts of reality out of the training data.
The usual metrics are not wrong, they are incomplete. They check whether each column looks right and stop there. If you are working through a Python data science roadmap, this is worth internalizing early: synthetic data that survives production has to do more than resemble the source one column at a time.
1. Fidelity metrics ignore correlation drift
Fidelity usually gets proven with a Kolmogorov-Smirnov test or Wasserstein distance. Both do a fine job on marginal distributions, one feature at a time, and neither says anything about how features move together. Take a healthcare dataset where age and severity each match the real distribution exactly. If the correlation between the two has drifted, the model still learns the wrong interaction, and no marginal test will catch it.
What I run instead is a correlation drift score: the Frobenius norm of the difference between the two correlation matrices. One number, cheap to log on every generation run. Past 0.5, the generator is inventing relationships that were never in your data.
import numpy as np
import pandas as pd
def bbioon_correlation_drift(real_df, synth_df):
# Compute Frobenius norm of the difference
real_corr = real_df.corr().fillna(0).values
synth_corr = synth_df.corr().fillna(0).values
return np.linalg.norm(real_corr - synth_corr, 'fro')
# If score > 0.5, your validation failed.
print(f"Drift Score: {bbioon_correlation_drift(df_real, df_synth)}")2. The tail loss problem in synthetic data validation
Train on Synthetic, Test on Real (TSTR) is the default benchmark, and the average AUC it reports is where people get burned. A generator that optimizes for the dense part of a distribution shaves down the rare events a little more with every round, which is model collapse. In fraud detection that shows up as a model which catches ordinary scams reliably and misses the high-value transactions sitting in the lowest decile.
So stratify the TSTR results by target decile instead of reading one number. When the rarest deciles fall well behind a model trained on real data, the synthetic set is losing utility in exactly the region you built the model for.
3. Privacy metrics and attribute inference risk
Membership inference risk answers “was this record in the training set?”, and it gets measured everywhere because it is cheap to compute. It is also the wrong shield for the GDPR re-identification standard. Regulators care about attribute inference: whether an attacker can work out a sensitive attribute such as income from quasi-identifiers such as age and region that the synthetic set still carries.
The test I use here is an attribute inference lift. Train something like Gradient Boosting from scikit-learn on the synthetic data, predicting the sensitive column from the quasi-identifiers, then compare it against the majority-class baseline. If it beats that baseline by a wide margin, your dataset teaches an attacker more than it teaches your models.
from sklearn.ensemble import GradientBoostingClassifier
def bbioon_check_privacy_lift(synth_df, real_test, q_identifiers, sensitive_col):
clf = GradientBoostingClassifier().fit(synth_df[q_identifiers], synth_df[sensitive_col])
acc = clf.score(real_test[q_identifiers], real_test[sensitive_col])
baseline = real_test[sensitive_col].value_counts(normalize=True).max()
return acc - baseline # This is your lift score.Older guidance treats zero risk as proof of a good dataset. No such point exists. Synthetic data validation is a set of trade-offs you pick deliberately. And if models keep breaking while the code gets harder to touch, some of that may be your data science codebase smelling sour from leaning on whatever metrics the tooling prints by default.
If validation work like this is eating your development hours, I can take it off your plate. I have been wrestling with WordPress and complicated backend logic since the 4.x days.
Validation is a measurement problem
The fraud model from the top of this post did not fail on bad data. It failed because nobody asked the right questions of the dataset first. Write down who gets access to it, what the downstream task actually is, and which features carry the weight in your utility metrics. Then build the checks yourself with NumPy and scikit-learn instead of trusting the generator’s own report card.
