We need to talk about disaggregated LLM inference. The standard advice in the industry has become dumping everything into a monolithic vLLM stack, and it is quietly killing your margins. I have watched teams spend six figures on H100 clusters and then work out that they get meaningful work out of maybe 30% of their compute capacity. The mess comes from a misunderstanding of how transformers actually process data.
Every inference request is really two workloads. The prefill phase reads the prompt in parallel, which is compute-bound and hammers the tensor cores. The decode phase then generates tokens one at a time, which is memory-bound and leaves those expensive tensor cores idle while the memory bus saturates. Running both on one GPU is like using a Ferrari as a delivery van. It works, and you pay a massive premium for performance you are not using during 90% of the trip.
Prefill and decode want different hardware
In a monolithic deployment the scheduler interleaves prefill and decode requests in the same batch, so a large prompt arriving stalls the active decode requests. That is where the jitter comes from, the inter-token latency (ITL) spikes that make a streaming response feel broken. You also end up overprovisioning hardware for prefill peaks, and utilization craters to 20-30% through the long tail of generation.
Disaggregated LLM inference splits those phases into separate hardware pools, which is what lets you right-size the infrastructure. Prefill tasks go on GPUs with high FLOPS, decode tasks on GPUs with a lot of memory bandwidth. Anyone who has fought KV cache VRAM bottlenecks already knows that managing this state is the hardest part of scaling.
How disaggregated serving is put together
Splitting the path is not free. The KV-cache has to travel from the prefill worker to the decode worker over the network. A 4,096-token prompt on Llama 70B produces about 1.34 GB of KV-cache. If the network cannot keep up, transfer latency eats every gain you made, so you want RDMA (Remote Direct Memory Access), or at minimum a 100 Gbps link. Without it you have traded a compute bottleneck for a network one.
# Calculating KV-cache size to see if your network can handle it
def bbioon_calc_kv_cache(layers, heads, dim, seq_len, bytes_per_val=2):
# Standard formula: layers * heads * dim * 2 (K and V) * bytes
per_token = layers * heads * dim * 2 * bytes_per_val
total_gb = (per_token * seq_len) / 1e9
return total_gb
# Llama-3 70B example
print(f"70B KV Cache: {bbioon_calc_kv_cache(80, 8, 128, 4096):.2f} GB")Frameworks like NVIDIA Dynamo and vLLM support this natively now. They use KV-aware routers that track which decode worker holds which cache, which enables prefix caching and cuts redundant computation. I went further into the reasoning behind choices like this in my post on LLM architecture truths.
Setting up vLLM for disaggregated inference
On vLLM, a basic disaggregated setup means spinning up producer (prefill) and consumer (decode) instances. In production they belong on different node types, each tuned for its own bottleneck: maximum FLOPS on the prefill nodes, HBM capacity on the decode nodes.
# Start the prefill (producer) worker
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B \
--kv-transfer-config '{
"kv_connector": "PyNcclConnector",
"kv_role": "kv_producer"
}'
# Start the decode (consumer) worker
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B \
--kv-transfer-config '{
"kv_connector": "PyNcclConnector",
"kv_role": "kv_consumer"
}'When should you disaggregate?
Do not over-engineer this without the traffic to justify it. I have seen devs reach for vLLM disaggregated serving on small internal bots and end up slower than they started. With short prompts, under 512 tokens, or a low GPU count, the scheduling overhead and transfer time will cost you more than they save. Serving thousands of concurrent users with a GPU bill that has become a board-level concern is a different story, and there this is the most effective cost reduction available in 2026.
If this disaggregated LLM inference work is eating your dev hours, hand it to me. I have been wrestling with WordPress since the 4.x days, and I have built enough high-scale infrastructure to know where the bottlenecks hide.
Where this leaves your GPU bill
Stop treating LLM inference as a monolithic task. Prefill and decode have opposite hardware needs, and separating them removes inter-token jitter while getting more out of every GPU you rent. Research from UCSD’s Hao AI Lab makes the case that this architecture is where low-latency serving is headed. Skip the planning now and you will overspend as you scale.
