Most RAG pipelines that return garbage are not suffering from a chunk size problem. Tuning chunk sizes and bumping overlap is the standard advice, and it is a band-aid over the actual issue, which is context loss. Contextual Retrieval in RAG is what separates a pipeline that returns plausible text from one that returns the right passage.
I have watched developers burn hundreds of hours trying to fix a broken search by throwing more compute at it. The model is not the problem. Traditional RAG cuts documents into isolated chunks and strips out the glue that gave those chunks meaning, so the vector database fills up with homeless snippets.
The “broken mixture” problem
Take a technical manual with a section that reads “Heat the mixture slowly.” On its own, that chunk is useless. Tomato sauce, or lab-grade starch? A semantic search will happily return it for the query “heating instructions,” and the LLM then has nothing to go on but a guess. Accuracy drops because the meaning was cut off at the chunk boundary.
Contextual Retrieval in RAG deals with that directly. Rather than embedding the raw text, you situate each chunk inside its parent document before it reaches the index.
I have written about related mistakes in my post on not over-engineering your vector database. This is one of the few architectural shifts that pays for itself.
How contextual retrieval in RAG works
During ingestion, you send the chunk and its parent document to a cheaper, faster model and ask for a one-sentence summary that places the chunk in context. That summary gets prepended to the text before you build the embeddings and the BM25 index. The “mixture” chunk now carries the fact that it belongs to the Italian cookbook rather than the lab safety manual.
<!-- Situating a Chunk via LLM Prompt -->
<document>
{FULL_DOCUMENT_TEXT}
</document>
<chunk>
Heat the mixture slowly and stir occasionally.
</chunk>
Provide a brief context to situate this chunk within the overall document.
<!-- Result: "Instruction for simmering tomato sauce in the Italian Cookbook." -->Anthropic’s research puts the improvement at up to a 49% reduction in retrieval failure rates when the technique is combined with reranking. For anything running in production, that is a large change in how often the pipeline gets it right.
What it costs, with prompt caching
The first question I get is whether calling an LLM for every single chunk triples the ingestion bill. A year ago I would have said yes. With prompt caching it stops mattering much: you cache the full document once, and each chunk-situating call pays only for the incremental tokens.
All of this happens during ingestion, so none of it lands on the user’s query latency. The work goes in upfront and the search stays fast at query time. For more on that at scale, I have a longer piece on Agentic RAG Caching.
If this kind of retrieval work is eating your dev hours, I can take it off your plate. I have been working with WordPress and messy backend integrations since the 4.x days.
What to take away
Before you refactor your chunking strategy again, check whether the failures are about context rather than size. Contextual Retrieval in RAG gives your vector database a view of the whole document while it is looking at a single paragraph, and it is a cheap fix for a problem that usually gets treated as a tuning exercise. That is where I would start.
