Everyone is rushing to shove ChatGPT or Claude into their WordPress site, and almost nobody opens the billing dashboard until the invoice arrives. The advice going around is to keep adding context, which is also what makes the thing slow. If you are not using Prompt Caching, you are paying full price to recompute the same tokens on every single request.
What a long context actually costs
Large language models get expensive and slow once the context window fills up. Send a 10,000-token system prompt, or a big Retrieval-Augmented Generation context, and the model processes every one of those tokens from scratch. Your user sits watching a spinner while the API provider counts your money.
Prompt Caching stores the processed state of the prompt’s prefix. The model does not recompute your system instructions or the same document context on every call, it reuses what it already worked out. The numbers are worth the trouble: roughly 80% less latency and up to 90% off the input token bill.
How prompt caching works
LLM inference runs in two stages: pre-fill, where the model processes your prompt, and decoding, where it writes the response. Pre-fill is the compute-bound one. That is the stage where the model looks at every token you sent and works out how they relate to each other.
Most applications land near the Pareto split: 80% of the requests reuse 20% of the same data. System instructions, formatting rules, often a chunk of the recent conversation, all identical from one call to the next. Prompt caching lets the provider keep the pre-fill tensors for that identical prefix, so the next request carrying the same prefix skips pre-fill and goes straight to generation.
I have written before about advanced LLM optimization, and of everything on that list, caching is the one that moves the numbers most once you are in production.
Prefix stability
Caching works at the token level, from the start of the prompt onward. Change one character up front, a timestamp or a user ID, and you get a cache miss. So the static, heavy material has to sit first, and anything that varies goes after it.
<?php
/**
* Good vs Bad Prompt Construction for Caching
*/
function bbioon_get_llm_prompt($user_input) {
// BAD: Dynamic data at the start ruins the cache
$bad_prompt = "User: ID_123. Time: " . time() . "\n" . get_large_system_instructions() . "\nQuery: " . $user_input;
// GOOD: Static prefix is identical across all users/sessions
$system_instructions = get_large_system_instructions(); // 2000 tokens of static rules
$good_prompt = $system_instructions . "\nUser Query: " . $user_input;
return $good_prompt;
}Prompt caching in a WordPress plugin
If you are building a custom plugin or a WooCommerce integration, keep an eye on the token thresholds. OpenAI and Anthropic both want at least 1,024 tokens before caching kicks in. Below that, the bookkeeping costs more than the cache saves.
Build any real RAG pipeline, the kind I went through in the guide on vector search optimization, and you clear that threshold without trying. The providers part ways from there. With Claude you mark the breakpoints yourself, using the cache_control parameter in the API call. OpenAI matches your prefix against earlier requests on its own.
A practical strategy
- Move static data to the top: the 500-line “You are a helpful assistant…” block goes first.
- Limit dynamic variables: a date or a location belongs at the very end of the prompt.
- Batch similar requests: when you run a queue of background jobs, group them by system prompt so more of them land on the same cache.
If this kind of work is eating your dev hours, I can take it on. I have been building on WordPress since the 4.x days, and I have seen how messy an AI integration gets when the architecture was rushed.
The payoff
The model is not a black box you feed and hope about. Once you know how the attention mechanism chews through tokens, you can order a prompt so the expensive part gets reused instead of repeated, and the tool stops feeling sluggish. Prompt caching is cheap to add and it changes the cost profile of the whole integration, so do it before the invoice makes the decision for you.
