Plenty of developers point a Large Language Model at a pile of proprietary data and expect it to work the rest out on its own. The bottleneck is almost never the model. It is the retrieval logic, and RAG chunk size sits right in the middle of it. Feed a vector database garbage fragments and you get garbage answers back.
I reviewed a project recently where retrieval kept failing on trivial distinctions. It could not separate “Project Alpha” from “Project Beta” because the text had been split mid-sentence. Once the RAG chunk size is wrong, a chunk no longer carries enough meaning for the vector search to do anything useful with it.
Small chunks vs. large chunks
A Retrieval-Augmented Generation pipeline does not search whole documents. It searches chunks. How big those chunks are, measured in characters, tokens or words, decides what the embedding model actually sees.
- Small chunks, say 80 characters, are very specific and lose almost all context. What comes back is often a sentence fragment with no usable meaning.
- Medium chunks around 220 characters look like the goldilocks zone, but they introduce a nastier problem: in testing, they return nearly identical cosine similarity scores for answers that are not remotely the same.
- Chunks of 500 characters and up hold their context and give stable rankings, at the cost of precision. You can pull back a wall of text that contains the answer plus a lot of noise.
If you are wiring AI into WordPress, maybe as a custom knowledge base for WooCommerce, you need a dependable way to split content. Here is the naive approach I keep running into, and the recursive version I would rather see.
Refactoring your text splitting logic
Do not reach for substr(). It knows nothing about word boundaries, so it cuts sentences in half and takes your accuracy down with them. Use a recursive splitter that respects whitespace or punctuation.
<?php
/**
* Naive Approach: The "Context Killer"
* This method often splits words in half, ruining the embedding.
*/
function bbioon_naive_split($text, $size) {
return str_split($text, $size);
}
/**
* Senior Approach: Recursive Character Splitting
* Respects boundaries to maintain RAG chunk size integrity.
*/
function bbioon_recursive_split($text, $max_size, $overlap = 50) {
$chunks = [];
$text_length = strlen($text);
$pointer = 0;
while ($pointer < $text_length) {
$chunk = substr($text, $pointer, $max_size);
// Find the last space to avoid cutting words
$last_space = strrpos($chunk, ' ');
if ($last_space !== false && ($pointer + $max_size) < $text_length) {
$chunk = substr($chunk, 0, $last_space);
}
$chunks[] = trim($chunk);
$pointer += (strlen($chunk) - $overlap);
}
return $chunks;
}Cosine similarity is not correctness
People read the similarity score as a confidence number. It is not one. It measures relative proximity in vector space, nothing more. When several chunks land within a hair of each other, 0.873 against 0.874, the system is guessing which one to return. That instability is why RAG chunk size is something you test rather than set once and forget. Larger chunks tend to spread those scores further apart, which makes the top result more trustworthy.
If you want more on this, I wrote up how to master your robust custom AI assistant, and Cohere’s documentation on chunking strategies is worth an hour of your time.
If RAG chunk size tuning is eating your dev hours, I can take it on. I have been working with WordPress since the 4.x days.
Test it on your own data
There is no perfect chunk size, only one that fits the shape of your data. Documentation built from bullet points can survive small chunks. Narrative writing needs 500 characters or more. Run the experiment against your own retrieval rankings: fix the splitter, try a few overlap values, and watch how far apart the similarity scores land.
