How .item() stalls your PyTorch token generation loop
A blocking .item() call in an autoregressive decode loop leaves the GPU idle every step. Two CUDA streams in a ping-pong pattern hide that sync, and StaticCache with torch.compile keeps the computation graph fixed.