The default analytics habit in the WordPress ecosystem runs roughly like this: the number is green, the p-value is under 0.05, ship it. That habit gets expensive. We fixate on the number without knowing the p-value meaning behind it, and you end up convinced that a small CSS tweak doubled your conversion rate when what actually happened was Tuesday.
I have watched clients scrap an entire checkout refactor because a three-day A/B test showed a “significant” revenue drop at a p-value of 0.04. Dig into the logs and the drop came from one whale customer whose credit card check failed on the old version. That was a single failed payment, not a trend. Knowing what a p-value is actually claiming is what stops you from paying for that kind of mistake.
The “weirdness score”: a better way to read p-values
Formally, a p-value is the probability of getting results at least as extreme as the ones you observed, assuming the null hypothesis is true. For developers it works better as a weirdness score. Think of the warning in your error_log that only fires when a user hits one specific transient at exactly midnight: rare enough that you stop and look at it twice.
A low p-value is not “the probability that my hypothesis is correct.” It answers a narrower question: if nothing at all were going on here, how weird would this data be? A score of 0.03 says that if your two versions were identical, you would see a gap this big about 3% of the time out of pure luck.
My earlier guide on how to stop betting on noise in your experiments covers the practical side of this.
Why 0.05 is not a magic number
0.05 might be the most arbitrary number in modern engineering. Ronald Fisher popularized it as a rule of thumb, and it sits between two failure modes: false positives, where you believe something works when it doesn’t, and false negatives, where you throw away a real winner.
On a high-traffic WooCommerce store running 50k transactions a day, 0.01 is a more sensible target, because the cost of being wrong is a broken checkout. A small blog has the opposite problem, where 0.05 is strict enough that you may never find any pattern at all. Wikipedia’s p-value article goes further into the distribution curves if you want the math.
Coding a significance sanity check
If you are building your own A/B testing logic inside a plugin, print more than the conversion rate. Wrap it in something that asks whether the sample size justifies looking at a p-value at all. Here is a small PHP utility that makes the sample-size problem visible before anyone starts arguing about the p-value meaning.
<?php
/**
* Simple Sanity Check for A/B Test Data
* Prevents over-analyzing small datasets.
*/
function bbioon_check_test_reliability( $observations, $conversions ) {
$conversion_rate = ( $observations > 0 ) ? ( $conversions / $observations ) * 100 : 0;
// If we have fewer than 1000 observations, the p-value is basically a coin flip.
if ( $observations < 1000 ) {
return [
'status' => 'unreliable',
'rate' => $conversion_rate,
'message' => 'Sample size too small. Stop looking at p-values!'
];
}
return [
'status' => 'ready',
'rate' => $conversion_rate,
'message' => 'Proceed with statistical analysis.'
];
}
// Example usage:
$result = bbioon_check_test_reliability( 150, 12 );
// Output: Sample size too small. Stop looking at p-values!
Practical takeaways for developers
- Don’t confuse significance with importance: A p-value of 0.001 can sit on top of a 0.01% revenue improvement. Weigh that against the technical debt of another plugin and it usually loses.
- Check the context first: A Chi-Square test on your categorical data is a cheaper first pass than any architectural change you are tempted to make.
- Don’t p-hack: Refreshing the results every five minutes and stopping the test the moment p hits 0.049 is lying to yourself. Set the sample size, wait for it, then look at the data.
If working out what your test results mean is eating your dev hours, I can take it off your hands. I’ve been wrestling with WordPress since the 4.x days.
How I read test results now
A p-value has one job: telling you how surprised to be by your data if nothing real is happening. Surprising and worth shipping are two different things, so when a change is statistically odd but commercially pointless, leave it alone. Treat test output the way you treat inherited code, which means reading it slowly and checking where each number came from before you trust it.
