cu_seqlens = torch.tensor([0, 200, 344, 408], dtype=torch.int32)Three documents of 200, 144 and 64 tokens pass through a transformer as one flat stream, and that array of cumulative offsets is what keeps them separate. The attention kernel reads cu_seqlens, knows where each document starts, and runs exact attention over all three without any pad tokens. WarpGroup-Backend, a C++17 pipeline described on Towards Data Science by Anubhab Banerjee, is built around that tensor, and the project exists because LLM padding overhead is bigger than most people assume.
Default batching pads every short sequence up to the longest one in the batch, and the GPU multiplies those zeros along with the real tokens. The repo’s benchmark on an H100 with Qwen2.5-7B over 400 mixed-length PDFs reports 48.41 percent padding, which falls to 0.55 percent after packing. That gap is where the 2.08x throughput gain comes from. On a GTX 1080 with a small SmolLM2 model the gain was 5.89x, which says the tax bites hardest on cheap hardware, the only kind most of us can justify buying.
Fixed counts versus measured budgets
Padded batching is what sample code hands you. Every row has the same shape, the code stays readable, and the waste scales with how uneven your lengths are. It never reaches zero though: the same benchmark measured 36.2 percent padding on a corpus with a tidy, uniform length distribution.
Varlen packing flips the order of operations. Sort sequences longest first, first-fit them into bins sized to actual free VRAM, round each length up to a multiple of 16 for the Tensor Core tiles, then hand one pinned-memory buffer to FlashAttention’s flash_attn_varlen_func. Free VRAM is not a number you can compute, so Phase 0 of the pipeline feeds synthetic forwards until CUDA throws out of memory, then backs off 10 percent. My one change: persist the measured capacity per model and host. Probing is a startup cost, and there is no reason to pay it again on every run. And the repo assumes NVIDIA CUDA, which limits where you can deploy it.
There is also a correctness trap. Once documents share one stream, position_ids must restart at zero for each document, or rotary embeddings drift quietly. The source handles it in a model wrapper, and it is the kind of bug that survives a week of staring at throughput numbers.
The budget-first instinct should sound familiar to anyone who ships WooCommerce work. Action Scheduler claims 25 actions, keeps processing until it reaches 90 percent of available memory or 30 seconds, then loops back in a fresh request. The count sets how much it claims, and the memory or time limit is what stops it. I wrote about that loop before, and it is the same instinct as the OOM probe, with friendlier failure modes.
Where LLM padding overhead meets WooCommerce
For a store pushing AI work to a hosted API, the GPU math barely applies, because requests leave one at a time and you pay per token anyway. What transfers is the scheduling shape: run catalog-scale jobs, like regenerating 4,000 product descriptions or embedding every product for semantic search, as budget-bounded batches instead of one long request. Shared hosting will kill the long version halfway through, with no resume.
Self-hosting turns it back into a hardware problem. If you run an embedding or classification model on one GPU, the padding numbers apply directly, and if you already run a cross-encoder reranker for search, the same job shape shows up at inference time. There, varlen packing is worth the setup cost, and a single consumer card is where the win is largest.
Size the batch for the weakest environment the job will run on, cache anything you had to measure, and make the loop resumable per batch. If you want a hand wiring a catalog-wide description or embedding job into that loop, that is work I take on. Either way, probe the budget on the host that runs the job and store the result, because a limit measured on another machine won’t hold.
