Multimodal RAG has a default recipe now: dump every PDF page and JPEG into one shared vector space using massive multimodal embeddings. It wrecks retrieval accuracy. Anyone who has built a production chatbot that has to return a specific table or figure out of a technical manual knows the pattern. The model finds the text and then hallucinates the image, or it links a completely unrelated diagram from three chapters away.
I have spent the last 14 years wrestling with complex data structures in WordPress and WooCommerce. A race condition in a checkout hook, a massive transient cache creating a bottleneck: the answer is rarely “more data.” It is almost always better structure. The same holds here. You do not need a heavier model, you need to stop shredding your documents into a blind bag of words.
The retrieval alignment problem
Traditional multimodal RAG fails because it treats a document as flat chunks. Sliding-window chunking splits image captions across separate retrieval units. The LLM sees “Figure 5” in the text while the image metadata sits in a chunk that never got retrieved, because its similarity score landed 0.01 below the cutoff.
That is a plain misalignment. Meaning, visual meaning especially, lives inside a coherent semantic section rather than a character count. On how structure affects ordinary retrieval, there is my earlier dive into reliable RAG chunking strategies.
The proxy-pointer solution
Rather than forcing text and images into a shared vector space, which is expensive and often ambiguous, we use a proxy-pointer architecture. An LLM does not need to see an image to know it is relevant. It needs to know the image sits inside a specific, verified semantic section.
So the indexing pipeline gets refactored to treat the document as a hierarchical tree. Every node, meaning every section, holds both text and image “pointers.” Retrieve a section and the full context comes with it, local file paths for figures and tables included. A hard vision problem turns into filtering.
How the pipeline looks in production
- Parse the Markdown or XML into a skeleton tree, where each
node_idholds its own images. - Prepend the full structural path to the text before embedding, so every chunk carries its breadcrumb (
Manual > Section 4 > Maintenance). - Treat the retrieved text chunks as pointers and load the unbroken document section behind them.
<?php
/**
* Example: Structuring Sectional Pointers for WP-based RAG
* Instead of flat posts, we store sections with artifact metadata.
*/
function bbioon_register_rag_section_meta() {
register_post_meta( 'rag_section', '_section_artifacts', [
'type' => 'string',
'description' => 'JSON pointer to images/tables within this semantic node',
'single' => true,
'show_in_rest' => true,
]);
}
add_action( 'init', 'bbioon_register_rag_section_meta' );
// When the LLM retrieves a section, it pulls this context:
// { "node_id": "0045", "images": ["figures/pump_diagram_v2.png"], "text_snippet": "..." }
Why this beats multimodal embeddings
Multimodal embeddings optimize for similarity, not grounding. In vector space, a financial table for “Company A” looks nearly identical to a table for “Company B.” With no structural grounding, your multimodal RAG system will hallucinate data across documents. With the tree in place, the LLM judges relevance from the section’s full context instead.
I have watched this exact issue break enterprise search systems. Developers reach for another WP-CLI re-index or more expensive API calls, when the fix is architectural. There is more detail in my breakdown of 100% accuracy with Proxy-Pointer RAG.
If multimodal RAG is eating your dev hours, hand the work over. I have been wrestling with WordPress since the 4.x days.
Takeaway: precision over complexity
Multimodality is a retrieval alignment problem, not a vision problem. Instead of making your models see everything, give them the structure to find things. When accuracy matters more than a shiny demo, structural pointers are the safer bet. They cost less and they hold up in production.
