$text = wp_strip_all_tags( do_blocks( $post->post_content ), true );This one line decides most of the quality of RAG content indexing on a WordPress site. do_blocks() renders the dynamic blocks, wp_strip_all_tags() drops the markup, the block comments and the contents of script and style tags, and what’s left is plain text an embedding model or a keyword search can use. Most indexing code I come across skips one of the two halves and ships post_content raw.
What got me writing this was Angela Shi’s article on Towards Data Science, which argues that RAG is not machine learning and that importing the ML toolkit (hyperparameter sweeps, train/test splits, explainability frameworks) solves the wrong problem. Her main example is a team that spent six months on Optuna sweeps and fine-tuning embeddings while the real bug sat in the parser, which treated scanned pages with degraded OCR layers as clean text. I agree with the framing, and for WordPress I’d take it one step further: the parser problem is the contents of post_content itself, which change shape every time a client edits a page with a different tool.
Where RAG content indexing breaks in WordPress
Shi splits the pipeline into four parts: parsing, question parsing, retrieval, generation. Only the last one is a model you consume as a service. The other three are engineering, and on our platform the first part hides three specific traps.
Block markup becomes corpus noise
post_content stores block comments like <!-- wp:paragraph -->, and some blocks carry inline JSON inside those comments. Strip tags too early and shortcodes survive unexpanded; skip do_blocks() and dynamic blocks contribute nothing, because a Latest Posts block stores attributes rather than the rendered post list. So the order is render first, strip second. And use wp_strip_all_tags() rather than PHP’s strip_tags(), because it also removes the contents of script and style tags instead of leaving them in the string. That’s the difference between a clean chunk and one with inline CSS glued to the prose. The function reference spells out that distinction.
Some pages keep their content outside post_content
Elementor saves the page layout as JSON in an _elementor_data meta row, and post_content on those pages is close to empty; their data structure docs say it plainly. If your indexer walks wp_posts, a large share of the site returns a title and almost nothing else. No retriever tweak fixes that, because the text was never indexed in the first place. WooCommerce is a gentler version of the same pattern: the SKU lives in postmeta and the wc_product_meta_lookup table, prices and attributes live in their own tables, and only the long description is prose.
Answers that live in meta fields
“What’s the SKU for this” has a single source of truth, and that source is a meta field. “Does this ship to Canada” works the same way: the answer is a shipping class. Shi’s routing argument applies here directly: a point lookup wants a small unit, a section question wants a whole section, and in store terms a SKU question wants a meta lookup rather than a chunk of description. Chunking can’t fix that, because the answer was never in the description text. It’s the same mismatch her insurance example describes, where no single chunk size serves both “what is the effective date” and “what are the exclusions”.
The cleaning step for the content you do have is short:
function bbioon_rag_clean_text( $post_id ) {
$post = get_post( $post_id );
if ( ! $post instanceof WP_Post ) {
return '';
}
// Render dynamic blocks, then drop markup and block comments.
$text = wp_strip_all_tags( do_blocks( $post->post_content ), true );
// Shortcodes survive tag stripping; remove the syntax, don't run it.
$text = strip_shortcodes( $text );
return trim( preg_replace( '/\s+/', ' ', $text ) );
}Two warnings before you ship something like this. strip_shortcodes() strips everything between an enclosing shortcode pair, including the content inside it, so a client shortcode that wraps real content will delete that content from your corpus. That’s long-standing core behavior, tracked in ticket 25644. And don’t substitute the full the_content filter stack for the manual pipeline, because plugin output (share buttons, related-post blocks, ad units) ends up indexed as if it were part of the article.
If you’re wiring a RAG or AI search feature into a WordPress or WooCommerce site and retrieval keeps missing pages you know are there, this is the kind of work I take on. Send me the indexing code and I’ll tell you where it’s losing content.
Trace the query before you tune anything
When an answer comes back wrong, Shi’s method is to trace the query end to end before changing anything. Either the right passage never reached the model, which puts the failure upstream in parsing, question expansion, retrieval or the corpus itself, or the passage did reach the model and the prompt or response schema is at fault. On a WordPress site that trace has four parts: the question after term expansion, the retrieved post IDs with their scores, the chunks that were sent, and whether the answer existed in the corpus at all. A custom table plus a logging hook in your query handler covers all four. You don’t need an observability platform for this.
For evaluation, don’t collapse everything into one accuracy number. Split results by question type, the way Shi splits by intent: product lookups, shipping or policy questions, category comparisons. One aggregate can average strong lookups against failing section questions and still look acceptable, and it tells you nothing about which fix to ship. Track a not-found rate too. A system that invents a SKU for a discontinued product is worse than one that says it found nothing.
Whatever the trace shows, the fix is configuration. Route SKU questions to meta, add a term-expansion table that maps the words users type to the words your products use (kettle versus small electrical appliance is Shi’s example, and every store has its own pair), and re-index with the cleaner above. Nothing in that list needs training. That’s the part of her argument I’d hold onto for our stack: WordPress is already a config-shaped system, so reaching for a chunk-size sweep or an embedding swap instead of a hook is the expensive mistake. I hit the same wall writing about semantic search for WooCommerce, where the honest stopping point was careful product data rather than a fancier model, and the same data-first order applies to what you check before switching on an AI product feature.
One thing worth doing today: count the pages whose real content isn’t in post_content.
wp db query "SELECT COUNT(*) AS builder_pages FROM wp_posts p JOIN wp_postmeta m ON m.post_id = p.ID WHERE p.post_type = 'page' AND p.post_status = 'publish' AND m.meta_key = '_elementor_data' AND CHAR_LENGTH(TRIM(p.post_content)) < 200"If that count is a large share of your published pages, the corpus has holes no embedding model will paper over, and that’s a parsing problem to fix before anyone argues about chunk size or top-k.
