Context Engineering is the part of the stack most AI advice skips. The standard suggestion is “just retrieve and stuff it into the prompt,” and it wrecks performance. I thought I had seen every way a data pipeline could break before I started building production-grade RAG systems. Most tutorials stop at the easy part.
A RAG (Retrieval-Augmented Generation) system that answers the first two questions well and starts hallucinating or crashing by the fifth is a common story. The model is usually fine. The data is usually fine. What is missing is a context layer. RAG fetches the facts; Context Engineering decides what gets into that limited context window, in what shape, and what gets left out.
The breaking point: why RAG isn’t enough
Plenty of developers treat the LLM context window as an infinite bucket. It is not. In 2025, Andrej Karpathy described Context Engineering as the “delicate art and science of filling the context window with just the right information.” When your retrieved context runs 6,000 characters against a budget of 1,800, something has to go. Leave that call to the model and it will make it for you, usually by dropping the most relevant documents or losing the middle of the prompt.
I learned this building a support agent for a high-traffic WooCommerce store. Turn one was fine. By turn twenty the prompt had overflowed, wp_remote_post had timed out, and the model had forgotten the customer’s name. A bigger model would not have saved it. The pipeline was the thing that needed fixing.
Component 1: memory with exponential decay
Conversational memory fails in two directions: it forgets too fast, or it piles up noise until the system falls over. A sliding window is too blunt for either. Exponential decay behaves more like working memory, where the turns that mattered stick around and the small talk fades out.
In a WordPress environment I treat these memory turns like transients with a score, where the effective score comes from recency, importance, and how relevant the turn is to the current query.
# Logic for Exponential Memory Decay
import math
import time
def bbioon_calculate_decay(importance, age_seconds, decay_rate=0.001):
# Recency = e^(-decay_rate * age)
recency = math.exp(-decay_rate * age_seconds)
return importance * recency
# Turn 1: "What is my order status?" (Importance: 2.5)
# 10 minutes later...
score = bbioon_calculate_decay(2.5, 600)
print(f"Effective Context Score: {score}")Component 2: hybrid retrieval and re-ranking
Vector embeddings on their own miss exact keyword matches. Keywords on their own miss meaning. So Context Engineering blends the two, weighing TF-IDF scores against dense vector scores with a tunable alpha that I usually land around 0.65.
Retrieval gives you candidates. Re-ranking decides the order. On small and medium datasets, a heuristic as plain as boosting documents with specific tags tends to beat a neural cross-encoder on both speed and results. It also keeps retrieval noise from crowding out the one document the model actually needed.
Component 3: the token budget enforcer
Reserve space in the prompt in a fixed order. Get that order wrong and your documents overflow before the conversation history has been counted at all.
- The system prompt is fixed overhead, so it gets reserved first.
- Conversation history comes next, since that is what keeps answers coherent.
- Retrieved documents are the variable, compressed or truncated to fit whatever room is left.
For English prose I estimate 4 characters per token, which is close enough for planning. In production, use tiktoken for exact counts. In a custom WordPress plugin, that count happens right before the API call to OpenAI or Anthropic.
Component 4: extractive compression
When the budget is tight, chopping the tail off a document is the lazy option. Extractive compression scores every sentence across the retrieved documents by token overlap with the query, then greedily takes the best ones until the budget fills. Serve them back in their original order so the argument still reads in sequence.
If Context Engineering is eating your dev hours, I can take it off your plate. I have been working with WordPress since the 4.x days and building LLM pipelines since GPT-3 was in private beta.
The takeaway
RAG fetches the data. Context Engineering decides whether that data makes any sense to the model. Skip the layer and you are throwing strings at a black box. Most of the hours people spend tuning prompt wording would pay off better spent on the context going into them, and the latency numbers will show it.
