Attention Is Not What You Need (2025)

The work asks whether strong sequence modeling and reasoning require explicit self-attention. It frames multi-head attention as a highly expressive way to lift hidden vectors into a space of pairwise interactions, but argues that the resulting computations are difficult to describe through a compact set of interpretable invariants. As an alternative, it introduces an attention-free architecture built around Grassmann flows. Its Causal Grassmann layer reduces token representations, encodes neighboring token pairs as two-dimensional subspaces using Plücker coordinates, then uses gated mixing to return those geometric features to the hidden states. Across multiple local window sizes, information moves through controlled changes in low-rank subspaces rather than through an attention matrix.

Experiments suggest this approach can be competitive, though the results vary by task. Grassmann-only language models with 13-18 million parameters reach WikiText-2 validation perplexities roughly 10-15% higher than size-matched Transformers, indicating a remaining performance gap. For SNLI natural-language inference, a Grassmann-Plücker classification head added to DistilBERT slightly surpasses a Transformer head: validation accuracy is 0.8550 versus 0.8545, and test accuracy is 0.8538 versus 0.8511. The work also analyzes the computational cost of Grassmann mixing, finding linear scaling with sequence length when rank is fixed. Its broader claim is that manifold-based architectures may provide a more structured, geometrically interpretable alternative to attention, not that they already match Transformers across settings.