We need to talk about Context Engineering. For some reason, the standard advice in the AI space has become “just retrieve and stuff it into the prompt,” and it’s killing performance. I honestly thought I’d seen every way a data pipeline could break until I started building production-grade RAG systems. The reality? Most AI tutorials stop at the easy part.
If you’ve built a RAG (Retrieval-Augmented Generation) system that works perfectly for the first two questions but starts hallucinating or crashing by the fifth turn, you aren’t alone. It’s not your model, and it’s probably not your data. It’s your lack of a context layer. While RAG fetches the facts, Context Engineering is the architectural decision-making process that determines exactly what flows into that limited context window and in what form.
The Breaking Point: Why RAG Isn’t Enough
Most developers treat the LLM context window like an infinite bucket. It’s not. In 2025, Andrej Karpathy defined Context Engineering as the “delicate art and science of filling the context window with just the right information.” When your retrieved context is 6,000 characters but your budget is only 1,800, something has to give. If you don’t control that process, the model does—often by dropping the most relevant documents or losing the “middle” of the prompt.
I learned this the hard way while building a support agent for a high-traffic WooCommerce store. Turn one was fine. Turn twenty? The prompt overflowed, the wp_remote_post timed out, and the model forgot the customer’s name. The fix wasn’t a bigger model; it was a smarter pipeline.
Component 1: Memory with Exponential Decay
Conversational memory has two failure modes: forgetting too fast or accumulating noise until the system collapses. A simple sliding window is a blunt instrument. Instead, we use exponential decay. This mirrors human working memory—high-importance turns stay, while small talk fades.
In a WordPress environment, I treat these memory turns like transients with a score. We calculate an “effective score” based on recency, importance, and query relevance.
# Logic for Exponential Memory Decay
import math
import time
def bbioon_calculate_decay(importance, age_seconds, decay_rate=0.001):
# Recency = e^(-decay_rate * age)
recency = math.exp(-decay_rate * age_seconds)
return importance * recency
# Turn 1: "What is my order status?" (Importance: 2.5)
# 10 minutes later...
score = bbioon_calculate_decay(2.5, 600)
print(f"Effective Context Score: {score}")Component 2: Hybrid Retrieval and Re-Ranking
If you only use vector embeddings, you’ll miss exact keyword matches. If you only use keywords, you’ll miss semantic meaning. Context Engineering requires a hybrid approach. We blend TF-IDF scores with dense vector scores using a tunable alpha weight (usually around 0.65).
Furthermore, retrieval gives you candidates, but re-ranking decides the order. A simple heuristic like “boost documents with specific tags” is often faster and more effective than a neural cross-encoder for small-to-medium datasets. This prevents “retrieval noise” from crowding out the one useful document the model actually needs.
Component 3: The Token Budget Enforcer
This is the “Architect’s” secret weapon. You must reserve space in the prompt in a specific order. If you reserve in the wrong order, your documents will overflow before your conversation history is even accounted for.
- System Prompt: Fixed. Non-negotiable overhead.
- Conversation History: Reserved next to maintain coherence.
- Retrieved Docs: The variable. These get compressed or truncated to fit whatever is left.
For English prose, I use the 1 token ≈ 4 characters heuristic, but for production, you should use tiktoken to get exact counts. In a custom WordPress plugin, this happens right before the API call to OpenAI or Anthropic.
Component 4: Extractive Compression
When the budget is tight, don’t just truncate the tail of a document. Use extractive compression. We score every sentence across retrieved documents by token overlap with the query, then greedily select the top sentences until the budget is full. We then serve them back in their original order to preserve logical flow.
Look, if this Context Engineering stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress since the 4.x days and building LLM pipelines since GPT-3 was in private beta.
The Senior Dev Takeaway
RAG fetches the data, but Context Engineering decides if that data actually makes sense to the model. Without this layer, you’re just throwing strings at a black box and hoping for the best. Stop optimizing your prompts and start optimizing your context. Your latency—and your users—will thank you.
