The Chatbot Arena, the site that ranks large language models by public vote, asks each visitor which of two answers is better. Sean Moran’s walkthrough of the Bradley-Terry model on Towards Data Science explains the statistics behind that question, and the part that stuck with me is how far it gets on nothing but A-beats-B records, with no scale or rubric. Most ranking work in WordPress asks the opposite of people: give this product one to five stars, or give this answer a score out of ten.
The problem with averaging ratings
Ratings are calibrated against a scale that only exists in one person’s head. The article makes this point about annotators, and it transfers directly to store reviews and staff review queues: one reviewer’s 7 is another’s 9, some people compress everything into the top of the range, and the same person scores differently depending on the day. Averaging those numbers mixes disagreement about quality with disagreement about what the numbers mean.
A pairwise vote avoids that. “Is this product page better than that one” is a relative call, and people make relative calls consistently even when they’d argue about an absolute score. There’s a structural reason to like it too: on a store, ranking is usually the actual job, whether that’s ordering a “you might also like” block, ordering search results, or deciding which AI-written description to keep. The absolute score was a detour on the way to an ordering, and recommendation blocks only read an ordering.
What the Bradley-Terry model computes
Each item gets a latent strength, and the probability that item i beats item j is that strength divided by the sum of the two. Write the strength as an exponential, and the same probability becomes a logistic function of the difference between the two log-strengths. So the model is logistic regression on differences: the log-odds of a win are linear in the gap between the two scores.
The fitting loop is simple. The gradient of the log-likelihood for an item is its observed wins minus its expected wins under the current parameters. If an item wins more often than the model predicts, its score rises; if less, it falls. At the optimum, every item’s expected wins match its observed wins. And once you pin down the scale, the objective is concave, so there are no local maxima to get stuck in. That combination, a one-line learning signal and a single global optimum, is why the model has held up since the original Bradley and Terry paper from 1952.
Running it on a WordPress site
Use a custom table with one row per vote: winner ID, loser ID, a voter hash, a created timestamp. Resist the urge to store only counts per pair if you might ever care about voter quality, which I’ll come back to. The fit runs offline, from WP-CLI or a cron event, never on page load, and the resulting strengths go into an option that your template reads.
For the fit itself I’d use the MM update the article describes rather than gradient ascent, for two reasons: there’s no learning rate to tune, and strengths stay positive by construction. In PHP, on a contests array where each row is [winner, loser, count]:
function bbioon_fit_bradley_terry( array $contests, int $rounds = 200 ): array {
// Each row of $contests: [ winner_id, loser_id, vote_count ].
$strength = [];
foreach ( $contests as [ $w, $l ] ) {
$strength[ $w ] ??= 1.0;
$strength[ $l ] ??= 1.0;
}
for ( $r = 0; $r < $rounds; $r++ ) {
$next = $strength;
foreach ( $strength as $i => $pi ) {
$wins = 0.0;
$denom = 0.0;
foreach ( $contests as [ $w, $l, $n ] ) {
if ( $w === $i ) {
$wins += $n;
$denom += $n / ( $pi + $strength[ $l ] );
} elseif ( $l === $i ) {
$denom += $n / ( $pi + $strength[ $w ] );
}
}
// Items with no wins yet keep their previous strength.
if ( $denom > 0 && $wins > 0 ) {
$next[ $i ] = $wins / $denom;
}
}
$strength = $next;
}
// Pin the scale: geometric mean of 1.
$g = array_product( $strength ) ** ( 1 / count( $strength ) );
return array_map( fn( $pi ) => $pi / $g, $strength );
}The inner loop is items times contests per round, so it stays cheap while the vote table is small. If the table grows, run the fit outside the web server and store only the result.
The first deployment pattern I’d try is staff voting. Show two variants of a description, or two products in the same category, and let an admin screen record the pick. You get clean votes from people whose judgment you trust, and you learn whether the ranking is useful before building anything public.
If you want preference voting wired into a store, whether that’s staff picking between AI-generated descriptions or customers picking between products, that’s work I take on. Tell me the catalog size and where you’d collect the votes, and I can say whether Bradley-Terry earns its keep or something simpler covers it.
Where it breaks in practice
Four failure modes, all of which show up in a store context:
- Only relative strength is identified. Multiply every strength by a constant and every probability stays the same, so a strength of 3.4 means nothing on its own. Pin one item or normalize, and don’t surface raw scores as percentages anywhere.
- The comparison graph has to be connected. If a product was never compared, directly or through a chain, to the rest of the catalog, its position relative to them is not determined by the data. New products always start in that state, so they need a few bridge votes to place them.
- Lopsided matchups teach almost nothing. The curvature term is the product of the two win probabilities, which peaks when the contest is even. Pitting the best item against the worst is a wasted vote; pair items that sit close together in the current ranking instead.
- Voters differ in quality. The standard model treats every vote as an equally reliable coin flip, which is generous. CrowdBT adds a per-voter reliability parameter and fits it alongside the strengths with EM, so careless voters get downweighted without manual cleanup.
When head-to-head ranking is the wrong call
Head-to-head ranking is the wrong call when the number of pairs gets away from you. Votes are per pair, and pairs grow with the square of the catalog, so on a store with thousands of SKUs you sample matchups rather than compare everything. Sampling near the ranking boundary is the efficient choice anyway, because that’s where the curvature and the signal are concentrated.
It’s also wrong when you need an absolute answer. Bradley-Terry only knows relative strength, so “is this description factually wrong” or “how often does this page convert” are out of scope. For quality problems, refund and return rates are better signals than any amount of preference math.
And if preference depends on context, one strength per item is too coarse. The Arena paper describes this problem: easy prompts make every model look identical and the data fills with uninformative votes, so they use a contextual version where prompt difficulty and topic shift the outcome probability. That version is logistic regression on the difference of feature vectors, which any library can fit, and it buys you a cold-start trick: a new item’s strength can be estimated from its features before it has a single vote. On a store with steady new products and few voters, that extension is the piece I’d reach for first.
