Hacker News with Generative AI: Attention Mechanisms

Laser: Attention with Exponential Transformation (arxiv.org)
Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention.
Transformers, Machine Learning, Attention Mechanisms, Deep Learning
13 points by PaulHoule 51 days ago | 0 comments
Differential Transformer (arxiv.org)
Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise.
Machine Learning, Attention Mechanisms
562 points by weirdcat 102 days ago | 177 comments
New attention mechanisms that outperform standard multi-head attention (arxiv.org)
Deep Learning, Attention Mechanisms
233 points by snats 234 days ago | 49 comments
Ring Attention Explained – Unlocking Near Infinite Context Window (coconut-mode.com)
Machine Learning, Attention Mechanisms
10 points by simonguozirui 281 days ago | 2 comments