LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

arXiv:2607.24555v3 Announce Type: replace-cross Abstract: Serving large language models at long context is bottlenecked by the key-value (KV) cache, which is read at every decode step. We find that attention keys are approximately low-rank within…

Read the full story at arXiv cs.AI ↗