The way we evaluate RAG pipelines needs rethinking. The standard advice is to obsess over retrieval scores, hit 0.86 cosine similarity and call the job done. So we ship systems that look flawless on a dashboard and then answer wrong in production, confidently. I have watched this RAG conflict resolution gap sink projects with millions in funding, all because nobody accounted for one thing: the model ends up refereeing a dispute it was never trained to referee.
When retrieval succeeds and the answer still fails
Picture the good case. Your system retrieves the right documents. Retrieval worked. Except now the context window holds a June 2023 policy next to a November 2023 revision that contradicts it, and both score as highly relevant. The model reads both, takes the first one because of position bias, and reports a $4.2M revenue figure while the audited revision says $6.8M. Nothing was hallucinated. The RAG conflict resolution logic simply is not there.
I have written before about reliable RAG chunking strategies, and even flawless chunks do nothing for semantic collisions. When two authoritative sources disagree, most extractive QA models, minilm-uncased-squad2 among them, pick the span with the highest start and end logits. There is no output class for “these two documents disagree”.
Where the architecture falls short
The gap is architectural. We treat context assembly as string concatenation and move on. Research from Ye et al. (2026) found that LLMs given inconsistent retrieved context tend to answer wrong instead of abstaining. Two things drive that:
- Position bias. Spans that appear earlier in the context pick up marginally higher attention scores.
- Language strength. A flat declarative like “Revenue is $4M” outscores hedged or revised phrasing such as “Following restatement… is $6M”.
Building the conflict detection layer
The fix is a detector that sits between retrieval and generation and checks document pairs for contradictions before the LLM ever sees them. Two heuristics cover about 80% of the enterprise conflicts I run into: numerical contradiction and signal asymmetry.
# Numerical Contradiction Heuristic
def bbioon_detect_numeric_conflict(text_a, text_b):
nums_a = set(re.findall(r"\d+(?:\.\d+)?", text_a))
nums_b = set(re.findall(r"\d+(?:\.\d+)?", text_b))
# Filter out years and small integers to avoid noise
significant_a = {n for n in nums_a if float(n) > 2100 or float(n) < 1900}
significant_b = {n for n in nums_b if float(n) > 2100 or float(n) < 1900}
if significant_a and significant_b and not significant_a.intersection(significant_b):
return True # Non-overlapping claim values detected
return FalseDirectional tokens need handling too. One document saying “increased” while another says “decreased” about the same topic ID is a red flag. That in turn needs a cluster-aware resolution strategy, because keeping the most recent document globally throws away good data. Keep the most recent document per conflict cluster instead. When the accuracy bar is high, Proxy-Pointer RAG techniques help by mapping structures to scales.
Resolution via cluster-aware recency
Once a conflict is flagged, the pipeline builds a conflict graph and finds its connected components. Temporal conflicts such as policy updates and earnings restatements resolve to whichever document has the max ISO-8601 timestamp. Scoping it per cluster is what stops a conflict between two technical docs from quietly evicting an unrelated HR document that came back in the same retrieval.
# Cluster-aware resolution logic
def bbioon_resolve_conflicts(retrieved_docs, conflict_pairs):
# 1. Build adjacency list for conflict graph
# 2. Find connected components (clusters) using DFS
# 3. For each cluster, select the 'winner' based on timestamp
# 4. Return winners + all non-conflicting documents
pass If this kind of work is eating your dev hours, hand it over. I have been wrestling with WordPress and AI integrations since the early days, and I know where these ghost bugs like to hide.
Stop trusting retrieval scores
Retrieval itself is mostly a solved problem. Vector search is fast and well understood. Context assembly, which is where RAG conflict resolution lives, is still a blind spot for most developers. A pipeline with no contradiction check before generation is an expensive coin toss that sounds certain either way. Ship the detector, log what it flags, and stop shipping wrong answers.
