$body = json_decode( wp_remote_retrieve_body( $res ), true );
$cached = $body['usage']['prompt_tokens_details']['cached_tokens'] ?? 0;This is the number I care about on any job where several AI calls read the same material. It says how many tokens of the prompt the provider served from cache instead of processing again. Prompt caching is the hosted version of a trick someone just spent 1,500 lines of C++ doing by hand, and it’s the part you can control from a WordPress codebase.
The hand-rolled version is Prefill Once, Fan Out by Anubhab Banerjee on Towards Data Science. He ran into the ugly default of agentic pipelines: point two analytical agents at the same document through vanilla llama.cpp and each one runs its own prefill over the same bytes, because every llama_context starts empty. Agent two rebuilds, token by token, the exact state agent one finished building seconds earlier.
What the C++ fix was
Prefill is the dense pass that pushes every prompt token through every layer and fills the key/value cache. Decode is the cheap per-token part that follows. On his test rig, a GTX 1080 running Qwen2.5-7B Q4_K_M over a 3,501-token document, the prefill cost about 4.3 seconds while the branch decode averaged 77 ms. So nearly all of the second agent’s wait went to reading the document again.
SwarmKV prefills once, serializes the resulting KV state with llama_state_get_data into a host buffer, memcpys that buffer per branch, and each branch restores it with llama_state_seq_set_data before decoding. The two-agent pipeline ran 48.69 percent faster end to end, and the second branch’s activation latency dropped 98.09 percent, which is 8,685 ms of identical attention math nobody redid. The repo is on GitHub.
He is honest about the ceiling. The snapshot round trip goes through host memory, and every llama call runs under one global mutex, so the GPU work still lines up single file. The win is real but narrow: one long shared root, many short questions, and one model. Chat-shaped workloads with different histories per user get nothing from it.
Prompt caching on the API side
The same physics already runs on the other side of an HTTP call. OpenAI’s prompt caching is automatic on supported models, needs a cacheable prefix of at least 1,024 tokens, bills cached reads at a tenth of the uncached input rate on current models, and reports the number back as cached_tokens. Anthropic’s version is explicit: you mark blocks with cache_control, reads come in around a tenth of the base input price, and writes cost a bit more. The cache has a TTL measured in minutes unless you pay for longer, per the Claude pricing docs.
Neither hands you a KV snapshot as an object you can pass between contexts, which is the same graph-scoped gap the article complains about in vLLM and friends. Prefix reuse there is request-scoped or session-scoped rather than pipeline-scoped, so the orchestration is still your job.
So the fan-out you can afford from PHP is N requests that share an identical front half. The provider holds the computed state for that prefix and your per-agent question rides after it. I wrote about what that does to costs before, and there’s a longer take on caching in agentic RAG setups on this site too.
Order the prompt so the prefix survives
That puts the actual work in prompt construction, and this is where web code usually leaks. Build the prefix in one function so it cannot drift between jobs:
function bbioon_agent_prompt( string $document, string $question ): array {
return array(
array( 'role' => 'system', 'content' => bbioon_shared_instructions() ),
array( 'role' => 'user', 'content' => $document . "\n\n" . $question ),
);
}Static instructions first, the document next, the question last. Anything volatile (current date, username, request ID) goes after the prefix, or it invalidates the match every time. The match is on the exact prefix: one changed character near the front and you pay for a full re-read.
The article has a cousin of this bug at the llama.cpp level. If you restore a snapshot and then start the branch tokens at position zero instead of offsetting past the prefix, the rotary positions go wrong and the model emits confident nonsense. A drifted cache prefix fails quieter than that: the output still looks fine, and the bill is the only place it shows up.
Two habits worth stealing from the C++ side
- Fail fast on the budget. His step zero checks prefix plus branch plus headroom against n_ctx before allocating a byte. The PHP version: count tokens before the request and refuse to run a document that cannot fit alongside its longest branch, instead of letting the API truncate or error halfway through a paid call.
- Measure per call. He ships a timings key that reports zero for prefill and openly calls it an instrumentation bug. Store cached_tokens next to each job result, and you will spot the prompt edit that broke reuse on the day it happens rather than at the end of the month.
How I’d wire it into WordPress
Keep the document body in a transient or a custom table instead of rebuilding it per request. Fan the branches out as Action Scheduler jobs that all call the same prompt builder. Version the prefix on a hash of the document instead of the current time, so an unchanged document keeps its cache across runs. I’ve stopped assuming provider TTLs survive anything scheduled more than a few minutes out, so keep the gap between the shared call and the branch calls short.
The check I run after deploying something like this: fire the same document at two branches a few seconds apart and compare cached_tokens. If the second call reports zero, the prefix drifted, and that is the bug to chase. If you are wiring AI features into a WordPress or WooCommerce site and the token bill is climbing faster than the features justify, I’m happy to look at how your prompts are built and where the reuse is leaking.
