I honestly thought semantic search was the “holy grail” of retrieval until we launched our internal knowledge assistant. It looked perfect on paper, but in production, it was a mess. One of our platform engineers asked a simple question about a message-queue retry policy, and the system gave a confident, three-paragraph answer about exponential backoff that was completely wrong. The actual document she needed was sitting at position eleven—just outside the top ten we passed to the LLM. This is why hybrid search and re-ranking isn’t just an “optimization”; it’s a requirement for production-grade systems.
For most developers, the go-to approach is “just embed it and use cosine similarity.” But dense retrieval—converting text into high-dimensional vectors—has a massive blind spot. It’s great at conceptual queries like “how do we handle incidents?” but it fails miserably when someone searches for a specific technical term like “dead-letter queue threshold.” The embedding model averages out these specific terms into a single vector, losing the exact precision needed for technical lookups.
Why BM25 Still Matters in the Age of AI
Before we had neural retrieval, we had BM25 (Best Match 25). It’s a keyword-based approach that actually understands that rare terms matter more. If “dead-letter queue threshold” appears in only three documents, those documents get a massive score boost. Unlike dense vectors, BM25 doesn’t care about “semantic meaning”—it cares about exact matches, which is exactly what your engineers are looking for when they search for specific config keys or error codes.
The solution isn’t to pick one; it’s to use a hybrid approach. Specifically, we use hybrid search and re-ranking to combine the strengths of both. In systems like Weaviate or LlamaIndex, we use an alpha parameter to blend the results. An alpha of 1.0 is pure vector search, while 0.0 is pure BM25. For technical documentation, the “sweet spot” is usually around 0.5.
If you’re wondering why your current setup is missing the mark, you should check out my deep dive on why vector search misses exact IDs.
The Two-Stage Funnel: Re-Ranking with Cross-Encoders
Even with a hybrid retriever, you still face the “lost in the middle” problem. LLMs pay more attention to the start and end of their context window. If the most relevant chunk is ranked at #8 out of 10, the model might ignore it. This is where cross-encoders come in.
A bi-encoder (your standard embedding model) processes the query and document independently. A cross-encoder, however, looks at them together. It sees the direct interaction between the question and the answer, making it orders of magnitude more accurate—but also much slower. You can’t run it over a million documents, so you use it as a “second stage” to re-score the top 20 candidates retrieved by your hybrid search.
from llama_index.postprocessor.sbert_rerank import SentenceTransformerRerank
from llama_index.core.query_engine import RetrieverQueryEngine
# Stage 2: Cross-encoder re-ranker (ms-marco is a solid general choice)
reranker_postprocessor = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-6-v2",
top_n=5
)
# Assemble the query engine with the re-ranker
query_engine = RetrieverQueryEngine.from_args(
retriever=retriever,
node_postprocessors=[reranker_postprocessor]
)
# Now the LLM only sees the 5 most statistically relevant chunks
response = query_engine.query("What is the retry limit for the payment service?")Measuring Success with RAGAS
Don’t just guess if your search is better. You need to measure Context Precision and Context Recall. When we implemented hybrid search and re-ranking, our Context Recall (finding the right doc at all) jumped from 0.74 to 0.83. More importantly, our Context Precision (the relevancy of what we actually send to the LLM) hit 0.79. This directly translates to fewer hallucinations and more accurate “I don’t know” responses when the answer truly isn’t there.
Metadata filtering is another “gotcha.” If you have a runbook for a service decommissioned two years ago, it shouldn’t even be in the candidate pool. Use filters to narrow the retrieval space by department or document age before you even start the scoring process. It’s faster and significantly more reliable.
Look, if this hybrid search and re-ranking stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress since the 4.x days.
Stop Guessing, Start Tuning
Production RAG is a game of funnels. Start with a wide net using a hybrid retriever, narrow it down with metadata filters, and then use a cross-encoder to hand-deliver the best context to your LLM. If you’re still seeing “confident wrong answers,” your retrieval pipeline is likely the bottleneck, not your model choice. Refactor the search logic first, and you’ll save thousands in LLM tokens and debugging time. Ship it.
“}},excerpt:{raw:
