I recently wrapped up a project for a high-traffic WooCommerce store with a broken internal search. Thousands of SKUs, and customers kept complaining that they could not find what they needed. The client was fixated on their Search Ranking Evaluation Metrics, Mean Reciprocal Rank (MRR) above all, and figured that as long as “something” relevant sat at the top they were winning. The site was fast, but the results were useless.
I spent a week optimizing their custom search engine before I accepted that MRR was lying to us. My first move was simple: tweak the weighting on product titles with a filter, on the assumption that a match in position one meant the job was done. Users were not stopping at the first result, though. They scanned, compared, and often skipped the first “relevant” item for something better further down. Measuring only the first hit hid all of that.
Why MAP and MRR fail as search ranking evaluation metrics
Mean Average Precision (MAP) and MRR both treat relevance as a yes or no question. Real relevance is a gradient, especially in heavy site search environments, where one product is a perfect match and the next one is merely acceptable. MRR does not make that distinction. It finds the first “yes” and stops.
MAP has the opposite problem: it leans on recall. It wants credit for finding every relevant item, including the ones on page ten that nobody scrolls to. I ran into the same thing while fetching search engine data, where optimizing against the wrong numbers burns the whole effort. Shoppers are not after the long tail. They want the best three options right now.
NDCG and ERR instead
Normalized Discounted Cumulative Gain (NDCG) and Expected Reciprocal Rank (ERR) are the two worth switching to, because they handle graded relevance. Rather than a flat relevant or not relevant, you score each item from 0 to 3. NDCG then discounts an item’s value the further down the list it sits, which is roughly how user attention drops off.
ERR goes further and models cascade behavior. It assumes the user reads from the top down with some probability of stopping at each item, based on how relevant that item is. The math is in this write-up on retrieval evaluation metrics if you want it, but in practice ERR scores what satisfied the user rather than theoretical precision.
/**
* A simple conceptual example of how to structure
* relevance labels for Search Ranking Evaluation Metrics.
*
* @param array $bbioon_search_results The raw results from WP_Query or ElasticSearch.
* @return array Processed results with relevance grades.
*/
function bbioon_get_search_relevance_labels( $bbioon_search_results ) {
$bbioon_graded_results = [];
foreach ( $bbioon_search_results as $index => $post_id ) {
// Grade 3: Exact match in title
// Grade 2: Match in tags/categories
// Grade 1: Match in description
// Grade 0: Not relevant
$relevance_grade = bbioon_calculate_grade( $post_id );
$bbioon_graded_results[] = [
'rank' => $index + 1,
'id' => $post_id,
'grade' => $relevance_grade
];
}
return $bbioon_graded_results;
}
// You would then pass this array into an NDCG calculation tool.
// Check out ACM's research on ERR for deeper integration:
// https://dl.acm.org/doi/10.1145/1645953.1646033
What to change in your store
Store search is closer to a recommendation problem than a binary lookup. Move your Search Ranking Evaluation Metrics over to position-aware models like NDCG and the numbers you are chasing start to match how your customers actually shop.
- Drop MRR, since it ignores everything after the first hit.
- MAP is the wrong target for UI work, because it pushes you to optimize products nobody sees.
- NDCG rewards you for putting the best items at the top.
This gets complicated fast. If you are tired of debugging someone else’s mess and you just want your site search to bring in conversions, send me a note. Odds are I have seen it before.
