Every “build an AI chatbot” tutorial lands on the same default: split the text into 512-token windows, add a small overlap, move on. It demos beautifully. Ship a RAG (Retrieval-Augmented Generation) system to a paying client on that basis and the cracks turn up in the answers nobody thought to test. RAG chunking strategies deserve more thought than a default.
I’ve watched this exact failure. A client asks about one exception in a refund policy. The document is indexed, the embedding model is a good one, the LLM is more than capable of the reasoning. The answer still comes back wrong, because the chunker cut between the general rule and the exception clause. Neither half meant much on its own, so retrieval was lost before the model ever saw the text.
Where fixed-size RAG chunking strategies break
Fixed-size chunking is the hello world of data ingestion. It’s fast, it respects your token limits, and it has no idea what the text says. Cut a semantic unit by character count and the context goes with it. Nothing crashes, which is the real trouble: you get answers that are 80% correct and not worth trusting.
On a WooCommerce store or an internal knowledge base, the remaining 20% is the part a customer notices. The way out is a chunking strategy that follows the structure of the document instead of ignoring it. If you’re weighing up the wider architecture too, I wrote about LLM Agent Memory Architecture and how to keep chunks useful across a long conversation.
Sentence windows instead of big blocks
Retrieve one sentence, then hand the model the window of sentences around it. The embedding match stays tight while the generator still gets enough text to work with. In a Python stack with LlamaIndex it looks like this:
from llama_index.core.node_parser import SentenceWindowNodeParser
# Index time: store the sentence + metadata window
parser = SentenceWindowNodeParser.from_defaults(
window_size=3,
window_metadata_key="window",
original_text_metadata_key="original_text"
)
# Query time: use a post-processor to swap the sentence for its window
# node_postprocessors=[MetadataReplacementPostProcessor(target_metadata_key="window")]Hierarchical chunking follows the document
In technical documentation and legal contracts the structure carries meaning. A heading, a table and a paragraph are not the same kind of data and shouldn’t be chunked as if they were. Hierarchical RAG chunking strategies index at the leaf level, usually paragraphs, then promote to the parent section when several siblings come back for the same query. That keeps fragments of one section from competing with each other for room in the context window.
Where PDFs and tables go wrong
Tables are the worst offender. Flatten one into plain text and the link between a header and its value is gone. A layout-aware parser such as PyMuPDF can rebuild the row and column relationships into sentences before anything gets embedded. A chunk that reads “Product: Widget A, Stock: 0” can answer a question. “Widget A 0” cannot.
Measuring performance with RAGAS
Guessing whether the chunking works is not a plan. RAGAS (Retrieval-Augmented Generation Assessment) puts numbers on the failures, and the one I check first is context recall. Low recall with high faithfulness means the model isn’t making things up, your retriever simply never handed it the right passage. I went through the whole process in my guide on LLM Agent Evaluation.
If chunking work is eating hours you don’t have, I can take it on. I’ve been working with WordPress since the 4.x days, and most of that job is connecting old data to newer AI infrastructure without breaking either one.
Before you ship it
Chunking isn’t a configuration detail. It’s the decision the rest of the stack inherits. A model can’t answer from context it never received, and a re-ranker can’t rescue a chunk that was never retrieved. Read your documents first, pick a strategy that matches how they’re actually written, and measure the result before it goes live.
