How Infini-attention keeps long context from eating memory
Longer context windows cost server memory, not just compute. Infini-attention replaces the growing KV cache with a fixed-size compressive memory matrix updated by a Delta Rule, which cut memory use by 114x in Google’s tests.