Most A/B testing advice in the WordPress world boils down to installing a plugin and watching for a green arrow. That arrow will happily appear on a handful of conversions and mean nothing at all. Without a chi-square test on your conversion counts, you are spending a client’s budget on what might be a fluke.
Last Tuesday I picked up a ticket from a client who had already spent $12k on a redesign because a plugin said their “High-Cost” cover design was winning. I went into the raw database transients and the margin was thin enough that any variance check would have flattened it. What they had was noise with a label on it. The question worth answering first is when a pile of category counts turns into evidence.
Why your A/B test needs a chi-square test
WooCommerce work is full of categorical variables: cover type (High-Cost vs. Low-Cost) and sales outcome (Sold vs. Not Sold). The chi-square test for independence answers whether those two columns move together, or whether the gap in your conversion rate sits inside the range randomness produces on its own.
Say 1,000 books go out, 670 of them sell, and the covers are split 50/50. You expect 335 sales in each group. One group comes in at 350. Is that extra 15 the design working, or a race condition in human behavior? I went into how these imbalances behave at scale in Senior Dev Insights on Applied Statistics.
Calculating expected frequencies in PHP
Before you commit a refactor to a production theme, work out the expected count under the null hypothesis, the assumption that your design change does nothing at all. The formula is (Row Total * Column Total) / Grand Total. A 2×2 contingency table fits comfortably into a small WordPress helper.
<?php
/**
* Simple Chi-Square Statistic for 2x2 Tables
* Prefixing with bbioon_ for safety.
*/
function bbioon_get_chi_square_stat( $observed_data ) {
$row_totals = [
array_sum($observed_data[0]),
array_sum($observed_data[1])
];
$col_totals = [
$observed_data[0][0] + $observed_data[1][0],
$observed_data[0][1] + $observed_data[1][1]
];
$grand_total = array_sum($row_totals);
$chi_square = 0;
foreach ($row_totals as $r => $rtotal) {
foreach ($col_totals as $c => $ctotal) {
$expected = ($rtotal * $ctotal) / $grand_total;
$observed = $observed_data[$r][$c];
$chi_square += pow(($observed - $expected), 2) / $expected;
}
}
return round($chi_square, 2);
}
// Example usage:
// [ [LowCost_Sold, LowCost_NotSold], [HighCost_Sold, HighCost_NotSold] ]
$data = [[320, 180], [350, 150]];
$stat = bbioon_get_chi_square_stat($data); // Result: 4.07
?>Degrees of freedom and the critical threshold
The chi-square test statistic here comes out at 4.07, and that number means nothing until you compare it to a critical value. A 2×2 table has 1 degree of freedom, because once the totals are fixed, changing one cell forces the other three to move with it. There is only one direction left to push. At df=1 and a significance level of 0.05, the number to beat is 3.84.
Since 4.07 clears 3.84, the null hypothesis goes and the design actually worked. Plenty of developers stop at the mean and never look at the distribution, which is where most of these arguments get lost. On the wider question of what does and does not make an experiment valid, I looked at why Covariate Balance doesn’t always define success.
When the stats fail (assumptions)
Messy data breaks the math quietly. The chi-square test rests on four requirements, and ignoring any one of them makes the result meaningless:
- No customer can be counted in both groups.
- Every expected cell count has to be at least 5. Below that, use Fisher’s Exact Test instead.
- You are counting hits, not measuring load times.
- The sample has to be random, not skewed toward one referral source.
For high-frequency data, reach for the official PHP stats extension or a library like PhpSpreadsheet, which give you sturdier distribution functions than a hand-rolled loop.
If this kind of statistics work is eating your dev hours, I can take it on. I have been wrestling with WordPress since the 4.x days.
Don’t trust the dashboard
The next time a WooCommerce log shows a winning variant, run the numbers before you ship it. Check the degrees of freedom, then check that the p-value sits under 0.05. Skip that and you get clean-looking code sitting on top of a decision nobody verified. Treat the statistics the way you treat a hook you do not trust: look at the evidence first.
