We need to talk about Multimodal RAG. For some reason, the standard advice has become dumping every PDF page and JPEG into a shared vector space using massive multimodal embeddings, and frankly, it’s killing retrieval accuracy. If you’ve ever tried to build a production-grade chatbot that needs to return specific tables or figures from technical manuals, you know the frustration: the model finds the text but hallucinations the image, or worse, links to a completely unrelated diagram from three chapters away.
I’ve spent the last 14 years wrestling with complex data structures in WordPress and WooCommerce. Whether it’s a race condition in a checkout Hook or a massive Transient cache causing a Bottleneck, the solution is rarely “more data”—it’s almost always “better structure.” The same applies here. You don’t need a heavier model; you need to stop shredding your documents into a blind bag-of-words.
The Retrieval Alignment Problem
Traditional Multimodal RAG fails because it treats documents as flat chunks. When you use sliding-window chunking, you inevitably split image captions across different retrieval units. The LLM might see “Figure 5” in the text, but the actual image metadata resides in a different chunk that wasn’t retrieved because its similarity score was 0.01 lower than the cutoff.
This is a classic misalignment. Meaning—especially visual meaning—lives within a coherent semantic section, not a character count. If you’re curious about how this structure affects basic retrieval, check out my previous dive into reliable RAG chunking strategies.
The Proxy-Pointer Solution: Structure is All You Need
Instead of forcing text and images into a shared vector space (which is expensive and often ambiguous), we use a Proxy-Pointer architecture. The premise is simple: an LLM doesn’t need to “see” the image to know it’s relevant; it needs to know the image exists within a specific, verified semantic section.
We refactor the indexing pipeline to treat the document as a hierarchical tree. Every node (section) in that tree acts as a container for both text and image “pointers.” When we retrieve a section, we retrieve the full context, including the local file paths for figures and tables. This turns a complex vision problem into a straightforward filtering task.
How the Pipeline Looks in Production
- Skeleton Tree Building: Parse Markdown or XML into a hierarchical tree where each
node_idcontains its respective images. - Breadcrumb Injection: Prepend the full structural path (e.g.,
Manual > Section 4 > Maintenance) to the text before embedding. - Pointer-Based Retrieval: Use the retrieved text chunks as pointers to load the unbroken document section.
<?php
/**
* Example: Structuring Sectional Pointers for WP-based RAG
* Instead of flat posts, we store sections with artifact metadata.
*/
function bbioon_register_rag_section_meta() {
register_post_meta( 'rag_section', '_section_artifacts', [
'type' => 'string',
'description' => 'JSON pointer to images/tables within this semantic node',
'single' => true,
'show_in_rest' => true,
]);
}
add_action( 'init', 'bbioon_register_rag_section_meta' );
// When the LLM retrieves a section, it pulls this context:
// { "node_id": "0045", "images": ["figures/pump_diagram_v2.png"], "text_snippet": "..." }
Why This Beats Multimodal Embeddings
Multimodal embeddings optimize for similarity, not grounding. In vector space, a financial table for “Company A” looks nearly identical to a table for “Company B.” Without structural grounding, your Multimodal RAG system will hallucinate data across documents. By using a structural tree, the LLM can rely on the section’s full context to judge relevance with surgical accuracy.
I’ve seen this exact issue break enterprise search systems. Developers try to fix it with more WP-CLI re-indexing or expensive API calls, but the fix is architectural. You can read more about achieving 100% accuracy with Proxy-Pointer RAG in my technical breakdown.
Look, if this Multimodal RAG stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress since the 4.x days.
Takeaway: Precision Over Complexity
Multimodality isn’t a vision problem—it’s a retrieval alignment problem. Stop trying to make your models “see” everything and start giving them the structure to “find” everything. If you’re building a system where accuracy matters more than a “shiny” demo, stick to structural pointers. It’s cheaper, it’s faster, and it actually works in production.
