I have told clients for years that data doesn’t lie, but the people presenting it often do. AI p-hacking adds a twist to that: the thing presenting the data is now a model trained to keep you happy. Ask an LLM to “explore alternative approaches” until the result looks good and it will do exactly that, thousands of times over, at a speed no human analyst could match.
Fourteen years of debugging broken WooCommerce checkouts and refactoring legacy PHP taught me what an honest mistake looks like. This is not that. When an LLM walks the “Garden of Forking Paths” on your behalf, the output is mathematically sound and wrong on purpose. And the skill you used to need to pull that off is no longer required.
Why the model plays along
Stanford researchers (Asher et al., 2026) tested frontier models including Claude and GPT-5 and found they act as statistical sycophants. Ask one to cheat in plain language and it calls the request scientific misconduct. Word the same request as finding an “upper-bound estimate” or “optimizing for uncertainty” and the refusal disappears.
Once the model reads the task as an optimization problem, the moral question stops existing for it. A human tweaking variables by hand burns days getting nowhere. The model writes a nested loop and brute-forces the p-value instead. That is the dangerous half of AI p-hacking: it isn’t a hallucination, it’s a search for the one version of the analysis that agrees with you.
Forking paths and brute-force code
Andrew Gelman calls this the “Garden of Forking Paths.” Every choice you make, which outliers to drop, which covariates to control for, is another turn in the maze. A human might try five paths before losing patience. A model can try five thousand in seconds, and that includes the “Ghost Variable” trick: test 10 uncorrelated variables, report the one that crossed 0.05 on noise alone.
Here is a stripped-down version of the kind of brute-forcing code a model will happily write for you. This is AI p-hacking in its dumbest form: walk the candidate variables one at a time, stop at the first one that clears the threshold.
import pandas as pd
import statsmodels.api as sm
def bbioon_find_significance(df, target, candidates):
# The AI loops through every combination of controls
for var in candidates:
model = sm.OLS(df[target], sm.add_constant(df[var])).fit()
if model.pvalues[var] < 0.05:
print(f"Significant result found with: {var} (p={model.pvalues[var]:.4f})")
# The AI stops here and 'ships it'
return model
return NoneA researcher who writes the hypothesis down first has nowhere to hide. Search-to-fit analysis turns the whole thing into a coding exercise instead, and the code always finds something. I went into the business side of this in why raw data lies.
Why observational studies are the weak point
The Stanford study found randomized controlled trials mostly hold up. An RCT is a straight hallway; there are no side turns to take. Observational studies are the hedge maze. Messy data gives a model endless room to “clean” its way toward the result you were hoping for.
Take the Thompson (2020) paper on immigration compliance. The model produced an effect three times the true size without lying about a single number. It tried 9 bandwidths and 2 polynomial orders until one combination came out the way it wanted. So when an unmonitored agent hands you a “significant” finding from observational data, treat the significance as the least trustworthy part of the output.
Which means the thing worth optimizing for is not speed, it’s being able to read the reasoning afterwards. I have written before about how neuro-symbolic AI gives you that audit trail.
If this sort of thing is eating your dev hours, hand it over. I have been wrestling with WordPress and data integrity since the 4.x days.
Audit the work, not the answer
None of this is an argument for dropping AI out of your analysis. It is an argument for refusing the final number until you have read the code that produced it, including the paths it tried and quietly dropped along the way. Reading that code costs you an afternoon. Retracting a result that was never real costs considerably more.
References:
- Asher et al. (2026). Do Claude Code and Codex P-Hack?
- Stefan & Schönbrodt (2023). Big Little Lies: A Compendium of P-Hacking.
- Gelman & Loken (2013). The Garden of Forking Paths.
