We need to talk about Synthetic Survey Respondents. Somewhere along the way, the standard market research advice became “just prompt Llama-3 with some personas,” and it’s wrecking the validity of your data. If you build AI integrations that simulate human behavior, you’ve probably hit the same wall: the model is a great mimic and a terrible population.
I’ve spent the last decade debugging logic where “average” isn’t enough. In WordPress development we’d call it a race condition. In data science it’s mode collapse. Ask an LLM to simulate 6,000 households and you don’t get 6,000 opinions. You get one opinion, the mean, repeated 6,000 times. That’s why off-the-shelf Synthetic Survey Respondents are failing the “second moment” test right now.
The mean trap and mode collapse
Recent papers have shown that models like GPT-4o or Llama-3 can hit the median of the Survey of Consumer Expectations to within a percentage point. The dispersion is another story. Human replies on inflation expectations run from -25% to +27%, while the LLM puts 95% of its “people” inside a narrow 2% window.
The model isn’t reasoning here, it’s retrieving. It memorized the CPI tables and FRBNY survey releases in its training corpus, and that data leakage overpowers whatever persona you inject through the prompt. So your simulation collapses into a single representative agent. The usual workarounds, like knowledge-cutoff instructions or “do not look up statistics” prompts, don’t hold up at scale.
Solving mode collapse in synthetic survey respondents
If you want realistic Synthetic Survey Respondents, stop trying to hide the data from the model and remove it from the weights instead. That’s what machine unlearning does, here through Gradient Ascent (GA). You maximize the prediction loss on the “forget set” (the official stats) while minimizing loss on the reasoning abilities you want to keep.
Here’s a conceptual sketch of an unlearning loss function in Python (PyTorch) that pushes a model to forget specific statistical modes:
def compute_unlearning_loss(model, forget_batch, retain_batch, alpha=1.0):
# 1. Gradient Ascent on the "Forget" set
forget_outputs = model(**forget_batch)
# We negate the log-likelihood to maximize loss
forget_loss = -forget_outputs.loss
# 2. Gradient Descent on the "Retain" set (preserving general capability)
retain_outputs = model(**retain_batch)
retain_loss = retain_outputs.loss
# Total loss balances "forgetting" official stats vs "retaining" reasoning
total_loss = forget_loss + (alpha * retain_loss)
return total_lossWith this applied, researchers found that exact matches to the mean dropped from 92% to 24%, and tail accuracy reached 97%. The model starts to show the cross-sectional spread you see in real human populations.
Why this matters for your architecture
If your project simulates A/B tests or Randomized Controlled Trials (RCTs), a narrow distribution makes the results useless. Revisions in synthetic models often fail to separate across treatment groups because the model stays anchored to a memorized truth. Only after unlearning do treatments (like Fed targets or FOMC statements) shift respondent expectations the way they shift human ones.
I’ve always said technical debt is bad assumptions as much as bad code. Assuming an LLM can simulate a population because it can simulate an “average person” is a big architectural mistake. For more on data rigor, see my thoughts on causal inference in business.
If Synthetic Survey Respondents work is eating your dev hours, I can take it on. I’ve been working with WordPress and backend integrations since the 4.x days.
Takeaway
Mean accuracy can fool you. If you deploy Synthetic Survey Respondents, evaluate the second moment, which is the dispersion. Off-the-shelf models favor correctness (the mean) over realism (the spread). To build tools that hold up under the 2026 data mandate, treat data leakage and distributional accuracy as joint constraints. For more on current data requirements, see my guide on governance architecture.
Ship it, but check the tails. That’s where the real people, and the real bugs, are.
