Most writing about RAG tells you to fix retrieval: better chunking, better embeddings, add a re-ranker. Emmimal P Alexander on Towards Data Science argues that the quieter failure is money, and builds a RAG cost control layer: a semantic cache, a query router, a token budget and a circuit breaker, with claimed savings of up to 85.8% at 10,000 requests a day. I agree with the diagnosis. I would port almost none of the implementation to WordPress, because it assumes one long-lived Python process, and PHP-FPM gives you the opposite: many short-lived workers that share nothing in memory.
Read the numbers before repeating them. The 98.5% cache hit rate came from a run where 40% of the queries were pre-seeded into the cache; the author’s own cold-start estimate is 20 to 30%. The 85.8% figure is calculated from stated pricing, against a baseline where every request hits the most expensive model at 800 tokens, and it was not measured against a realistic mix. The direction is useful and worth acting on, but I would not put those numbers in a client proposal.
Where RAG cost control lives in WordPress
Python keeps the cache and the spend ledger in process memory. PHP forgets everything when the request ends, so the real design question is where state lives. There are three workable answers.
First, transients keyed on a normalized question. Lowercase, strip punctuation, collapse whitespace, hash it. This catches “What is your return policy?” and “what is your return policy” but not “can I send this back”, which is the same limitation the source admits for its TF-IDF cache: it matches shared tokens rather than meaning, so paraphrases miss. For a WooCommerce store where the same twenty questions dominate, normalization gets most of the repeat traffic for free.
Second, the object cache. Transients are a wrapper around it when the host has Redis or Memcached with the object-cache.php drop-in in place; without the drop-in they fall back to the wp_options table, so every cache hit is still a database read. With Redis you get sharing across workers and eviction without writing any of it, which is close to what the Python LRU provides. On any site running more than one PHP worker this is the same code and faster, so it is my default.
Third, a true semantic cache: embed every question, keep the vectors, return the nearest stored answer above a similarity threshold. That is GPTCache territory, and it costs an embedding call on every lookup that misses, plus somewhere to run the similarity math. Worth it when users rephrase instead of repeating; for a store support bot it is a sidecar project for a marginal hit rate. I wrote more about the cache you already have and about where else a RAG pipeline can cache.
<?php
function bbioon_answer_cached( string $question, callable $generate ) : string {
$key = 'bbioon_ans_' . md5( bbioon_normalize( $question ) );
$hit = get_transient( $key );
if ( false !== $hit ) {
return $hit;
}
$model = bbioon_is_factoid( $question ) ? 'small-model' : 'main-model';
$answer = $generate( $question, $model );
set_transient( $key, $answer, HOUR_IN_SECONDS );
return $answer;
}
function bbioon_normalize( string $q ) : string {
$q = strtolower( trim( $q ) );
$q = preg_replace( '/[^\p{L}\p{N}\s]/u', '', $q );
return trim( preg_replace( '/\s+/u', ' ', $q ) );
}
function bbioon_is_factoid( string $q ) : bool {
return (bool) preg_match( '/^(what is|what are|who is|define|list|how much|when does)\b/i', trim( $q ) );Routing is just preg_match
The router is the cheapest piece to port, mostly because there is little to port. The source scores queries on length, entity density and reasoning keywords, with a regex fast path for factoids, and maps three tiers to three models. All of that is string work. Its benchmark routed 81% of traffic to the cheap tier, and the expensive model was roughly 90 times the price per token, which is why routing tends to pay more than caching while the cache is still cold.
Keep the fallback behavior. The Python router degrades to the standard tier when a model is missing instead of throwing, and your wrapper should do the same, because sooner or later you will run this on an account where the big model is not enabled. A match statement with a default case is enough. Anything smarter, like learned thresholds, belongs behind a cron job that samples real queries and reports what your actual mix looks like.
The spend ledger has to outlive the request
The ledger and circuit breaker are the pieces that do not survive the port. In the source they live in memory, which the author flags even for Python: restart the process and your hourly and daily limits reset. In PHP they reset on every request, so an in-memory breaker protects nothing. The counter has to live in wp_options or Redis.
Options have a race: two workers read $4.80, both spend, both write $4.85, and you undercount. Redis INCR is atomic, so where an object cache exists, count there and mirror to an option for the admin screen. If you are stuck on wp_options only, accept the drift and set the threshold lower to compensate. I have not measured that drift on a busy multisite, so treat the sizing as a guess until you watch it for a week.
Breaker behavior matters more than storage. In the source’s own benchmark, a $0.001 hourly limit blocked all ten legitimate requests while the same code at $5.00 served all ten. The source’s rule is to set the limit at two to three times expected peak, not average. Defaulting to downgrading requests to the cheap model instead of blocking them is right for anything customer-facing, since a slightly worse answer beats a support ticket.
What I’d change first
For a single store on ordinary hosting, normalized transients plus a factoid router is an afternoon of work and most of the win. On a site with Redis and real traffic, move the same cache onto the object cache and add the INCR counter with a hard daily cap. Reserve the semantic cache for products where users rephrase heavily, and scope it as its own project.
If you have an AI feature on a WordPress or WooCommerce site and the provider bill keeps drifting up, that is the kind of work I take on: instrument the call paths, add the cache and routing, put a real cap underneath it.
Whatever you build, log tokens per request to a custom table for two weeks first. The whole premise of the source is that over-fetching and repeats stay invisible until someone reads the logs, and that measurement step costs nothing.
