We need to talk about disaggregated LLM inference. For some reason, the standard advice in the industry has become dumping everything into a monolithic vLLM stack, and it’s killing your margins. I’ve seen teams spend six figures on H100 clusters only to realize they are getting meaningful work out of maybe 30% of their compute capacity. It’s a mess, and it stems from a fundamental misunderstanding of how transformers actually process data.
Every inference request is a tale of two workloads. First, you have the prefill phase, where the model reads the prompt in parallel. This is compute-bound and hits the tensor cores hard. Then, you have the decode phase, where it generates tokens one by one. This is memory-bound and leaves those expensive tensor cores sitting idle while the memory bus saturates. Running both on the same GPU is like trying to use a Ferrari as a delivery van; it works, but you’re paying a massive premium for performance you aren’t using during the 90% of the trip.
The Performance Mismatch: Prefill vs. Decode
In a standard monolithic deployment, the scheduler interleaves prefill and decode requests in the same batch. Consequently, when a large prompt enters the system, active decode requests stall. This creates that annoying “jitter” or inter-token latency (ITL) spikes that make streaming responses feel broken. Furthermore, you’re overprovisioning hardware to handle prefill peaks, which then crater to 20-30% utilization during the long tail of generation.
This is where disaggregated LLM inference changes the game. By splitting these phases into separate hardware pools, you can right-size your infrastructure. You put your prefill tasks on GPUs with high FLOPS and your decode tasks on GPUs with massive memory bandwidth. If you’ve ever dealt with KV cache VRAM bottlenecks, you know that managing this state is the hardest part of scaling.
The Architecture of Disaggregated Serving
Splitting the path isn’t free. You have to move the KV-cache from the prefill worker to the decode worker over the network. For a Llama 70B model, a 4,096-token prompt generates about 1.34 GB of KV-cache. If your network isn’t up to the task, the transfer latency will eat all your gains. Specifically, you need RDMA (Remote Direct Memory Access) or at least a 100 Gbps link to make this viable. Without it, you’re just trading compute bottlenecks for network bottlenecks.
# Calculating KV-cache size to see if your network can handle it
def bbioon_calc_kv_cache(layers, heads, dim, seq_len, bytes_per_val=2):
# Standard formula: layers * heads * dim * 2 (K and V) * bytes
per_token = layers * heads * dim * 2 * bytes_per_val
total_gb = (per_token * seq_len) / 1e9
return total_gb
# Llama-3 70B example
print(f"70B KV Cache: {bbioon_calc_kv_cache(80, 8, 128, 4096):.2f} GB")Modern frameworks like NVIDIA Dynamo and vLLM now support this natively. They use “KV-aware” routers that track which decode worker holds which cache, enabling prefix caching and reducing redundant computations. For a deep dive into why these architectural choices matter, check out my post on LLM architecture truths.
Setting Up vLLM for Disaggregated Inference
If you’re running vLLM, a basic disaggregated setup involves spinning up producer (prefill) and consumer (decode) instances. In production, these should be on different node types optimized for their specific bottlenecks. Therefore, your prefill nodes should have maximum FLOPS, while decode nodes focus on HBM capacity.
# Start the prefill (producer) worker
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B \
--kv-transfer-config '{
"kv_connector": "PyNcclConnector",
"kv_role": "kv_producer"
}'
# Start the decode (consumer) worker
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B \
--kv-transfer-config '{
"kv_connector": "PyNcclConnector",
"kv_role": "kv_consumer"
}'When Should You Disaggregate?
Don’t over-engineer this if you don’t have the traffic. I’ve seen devs jump into vLLM disaggregated serving for small internal bots, and it actually slows things down. If your prompts are short (under 512 tokens) or your GPU count is low, the scheduling overhead and transfer time will kill your performance. However, if you are serving thousands of concurrent users and your GPU bill is becoming a board-level concern, this is the most effective cost-reduction strategy available in 2026.
Look, if this disaggregated LLM inference stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress since the 4.x days, and I’ve built enough high-scale infrastructure to know where the real bottlenecks hide.
The Core Takeaway
Stop treating LLM inference as a monolithic task. Prefill and decode have opposite hardware needs. By adopting a disaggregated approach, you remove inter-token jitter and maximize your GPU ROI. Research from UCSD’s Hao AI Lab proves that this architecture is the future of low-latency serving. If you aren’t planning for it now, you’re going to overspend as you scale. Ship it, but ship it efficiently.
