The usual answer when a team outgrows hand-rolled A/B tests is to buy an experimentation platform: interview stakeholders, run demos, score the finalists, and sign. That is what a data team at ManyChat did, and their retrospective on picking between Eppo and Statsig is worth your time. I agree with most of it. What it leaves out, for anyone doing WooCommerce A/B testing on a store they run, is that the most useful part of the process can happen on the stack you already own, at no cost, months before you open a sales conversation.
The best finding in the piece is a tie. The two platforms scored 4.36 and 4.47 across more than twenty weighted criteria, and the analysts who knew the stack best said they would be happy with either. Analyst-grade rigor is table stakes now. The value moved to who can run an experiment without an analyst holding their hand, and to everything after the signature, where the author counts a lifecycle, certified metrics and training as the real work. A platform is plumbing: if your store has no stable visitor identity and no logged assignments, the pipes arrive and nothing flows.
Most of the decision is not about the tools
For a store owner, the cheap way to prepare for that kind of decision is to run two or three real experiments in-house and watch what breaks. The author covered the same ground with interviews, and by his account those carried the whole process. A live test tells you the same things faster, and you can start this week.
It also tells you who wants to experiment. In the piece, PMs were supposed to score both platforms during the proof of concept and mostly did not, and that missing feedback is what pushed the final call toward the more guided product. You only learn which of your stakeholders will use the thing when a checkout button has to change because a test said so.
Running WooCommerce A/B testing without a platform
The smallest version I would ship, in order:
- Pick one unit of identity and stick to it. The user ID for logged-in customers, a first-party cookie for everyone else, set on init while headers are still open. On a store with guest checkout most visitors are anonymous, so the cookie is not optional.
- Make assignment sticky and cheap. Hash the visitor into a 0 to 999 bucket so percentages stay readable, and let the Transients API carry the lookup, since transients sit in the object cache when your host provides one.
<?php
/**
* Plugin Name: bbioon A/B assignment
*/
if ( ! defined( 'ABSPATH' ) ) {
exit;
}
function bbioon_visitor_id() : int {
if ( is_user_logged_in() ) {
return get_current_user_id();
}
$id = 0;
if ( isset( $_COOKIE['bbioon_vid'] ) && ctype_digit( (string) $_COOKIE['bbioon_vid'] ) ) {
$id = (int) $_COOKIE['bbioon_vid'];
} else {
$id = wp_rand( 100000000, 999999999 );
setcookie( 'bbioon_vid', (string) $id, time() + YEAR_IN_SECONDS, COOKIEPATH, COOKIE_DOMAIN );
}
return $id;
}
function bbioon_variation_for( string $test, array $variants = array( 'control', 'treatment' ) ) : string {
$key = 'bbioon_ab_' . md5( $test . '_' . bbioon_visitor_id() );
$cached = get_transient( $key );
if ( false !== $cached && in_array( $cached, $variants, true ) ) {
return $cached;
}
$bucket = (int) ( bbioon_visitor_id() % 1000 );
$share = (int) floor( 1000 / count( $variants ) );
$value = $variants[ min( (int) floor( $bucket / $share ), count( $variants ) - 1 ) ];
set_transient( $key, $value, DAY_IN_SECONDS );
return $value;
}WooCommerce core has run its own admin experiments through a small Experimental_Abtest class that grew out of the woocommerce-admin feature plugin: it asks a hosted service for the bucket, then caches the answer. Jetpack shipped the same pattern in jetpack-abtest before retiring the package. Skim both before writing code, because the problems they guard against, cache misses and assignments that move mid-test, are the only problems at this scale. (I would check aggressive full-page caching before trusting any of this, since a cached page can serve the wrong variation.)
- Log every assignment somewhere you can join it later. A small custom table, created with dbDelta on activation, holding test name, variation, visitor ID and timestamp, written once at assignment time. Revenue lives in WooCommerce orders, and orders carry the customer ID, so the join works if you logged the same key. Reconstructing buckets from analytics after the fact is where homemade experiments go to die, and the mistakes often start earlier, as anyone who has watched counts lie in WooCommerce order data already knows.
- Fix the decision rule before the test starts. The piece encodes a five-stage lifecycle, Propose, Design, Run, Analyse, Decide, into the tool’s own templates. A shared doc does the same job: what changed, which metric, how many orders, what result ships it. No peeking at interim numbers. This is the part I would keep even if no vendor ever arrives, because it is the part organizations skip.
What to fix before you talk to vendors
Four things, all cheap. Count traffic honestly, in orders rather than sessions for the same window, because people usually compare the wrong two numbers. Filter bots and your own team’s orders, both of which will happily flip a small test. Check consent and your cookie policy, since an assignment cookie follows the same rules as an analytics cookie. Name one owner for the experiment list, because the piece’s analyst who called herself a human microservice is what happens when nobody owns it.
None of that is wasted if you do end up buying. It becomes the requirements list, backed by three real experiments instead of anecdotes, and it sharpens the questions you ask a vendor. Picking between two capable platforms ends up looking like choosing between the two WooCommerce APIs: the tie breaks on trajectory and maintenance rather than feature counts.
One concrete step for today: drop the two functions above into a mu-plugin file, load your homepage twice, and open devtools to confirm that the bbioon_vid cookie kept the same value. Everything else depends on that cookie. And if you would rather have this wired properly on your own store, the assignment, the logging table and the join to order data, that is the kind of work I do, and one email is enough to start.
