Continuum Attention
On this page
arXiv:2610.20586v2 Announce Type: replace-cross Abstract: Conventional causal attention aggregates values at discrete memory positions. We introduce Continuum Attention (CA), in which neighboring key and value vectors define a continuous, piecewise linear memory trajectory and each query retrieves information by normalized integration over its visible prefix. The segment integrals have closed forms in exponential moments, yielding analytically differentiable attention without numerical quadrature. The resulting operator preserves causal masking, normalized nonnegative effective weights, and the standard Transformer decoder structure. Controlled byte level experiments favor exact continuous integration over matched softmax attention. We additionally evaluate data efficiency, out of domain language modeling, zero-shot multiple choice prediction, robustness, and multilingual generalization. In multilingual evaluation, CA consistently outperforms all baselines across every evaluated language, achieving an average 5.8% reduction in bits per byte (BPB) relative to the strongest baseline.
Why It Matters
Continuum Attention replaces the discrete key-value memory positions of standard causal attention with a continuous, piecewise linear memory trajectory, letting each query integrate over its visible prefix in closed form. The paper reports consistent gains in multilingual byte-level language modeling, including a 5.8% reduction in bits per byte across languages. It is a fundamental architectural proposal for the attention mechanism at the heart of Transformers. See Large Language Models, Algorithms Underlying AI, Transformers, and Training Pipelines.
Sources
Back to Recent News · Sub-article index.