Hacker News with Generative AI: Transformers

Three things everyone should know about Vision Transformers (arxiv.org)
After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks such as image classification, detection, segmentation, and video analysis.
Computer Vision, Transformers, Artificial Intelligence, Deep Learning
69 points by reqo 2 days ago | 16 comments
Sparsely-Gated Mixture of Experts (MoE) (thegreenplace.net)
In transformer models, the attention block is typically followed by a feed forward layer (FF), which is a simple fully-connected NN with a hidden layer and nonlinearity.
Transformers, Deep Learning, Machine Learning, Computer Science
15 points by mfrw 6 days ago | 0 comments
The Solid-State Shift: Reinventing the Transformer for Modern Grids (powermag.com)
Transformers have been the backbone of power grids for over a century, but today’s demands for renewable energy, electric vehicles, and smarter grids are exposing their limits.
Energy, Grids, Transformers, Renewable Energy, Electric Vehicles
58 points by JumpCrisscross 14 days ago | 24 comments
Former Tesla exec Drew Baglino's new startup is rethinking the transformer (techcrunch.com)
Former Tesla executive Drew Baglino has a new startup developing solid-state transformers for the electric grid, Axios reported.
Tesla, Startups, Electric Grid, Renewable Energy, Transformers
10 points by plun9 18 days ago | 2 comments
Real-Time Introspective Compression for Transformers (github.com/Dicklesworthstone)
This article proposes a novel approach to address both problems simultaneously.
Transformers, Compression, Artificial Intelligence, Machine Learning
14 points by eigenvalue 24 days ago | 11 comments
How DeepSeek Rewrote the Transformer [video] (youtube.com)
Transformers, Deep Learning, Computer Vision, Artificial Intelligence, Video
5 points by chetangole 29 days ago | 0 comments
VGGT: Visual Geometry Grounded Transformer (github.com/facebookresearch)
Computer Vision, Transformers, Artificial Intelligence, Deep Learning, Facebook Research
190 points by xnx 32 days ago | 42 comments
Transformers as Support Vector Machines (2023) (arxiv.org)
Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP.
Transformers, Machine Learning, Computer Science, Artificial Intelligence
3 points by TaurenHunter 40 days ago | 0 comments
Transformers Without Normalization (jiachenzhu.github.io)
This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique.
Transformers, Deep Learning, Machine Learning
260 points by hellollm 42 days ago | 32 comments
Understanding Transformers (beyond the Math) – kalomaze's kalomazing blog (bearblog.dev)
Maybe you don't want to attempt the conventional approaches for understanding the transformer architecture for language models. If you're anything like me, an informal approach is what you'd prefer - one that helps you reason about what's happening with these models in the abstract, without requiring mastery on the technical level to begin with.
Machine Learning, Deep Learning, Language Models, Transformers
8 points by bilsbie 48 days ago | 2 comments
RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals (arxiv.org)
Transformers have achieved great success in effectively processing sequential data such as text.
Transformers, Machine Learning, Deep Learning, Computer Science
3 points by PaulHoule 50 days ago | 0 comments
Some thoughts on autoregressive models (wonderfall.dev)
Most generative AI models nowadays are autoregressive. That means they’re following the concept of next token prediction, and the transformer architecture is the current implementation that has been used for years now thanks to its computational efficiency.
Generative AI, Artificial Intelligence, Machine Learning, Transformers
79 points by Wonderfall 54 days ago | 58 comments
Go-attention: A full attention mechanism and transformer in pure Go (github.com/takara-ai)
From the Frontier Research Team at takara.ai we present the first pure Go implementation of attention mechanisms and transformer layers, designed for high performance and ease of use.
Go, Attention Mechanism, Transformers, Machine Learning
168 points by PaulHoule 54 days ago | 85 comments
Show HN: A Transformer model that preserves logical equivalence (huggingface.co)
Running
Transformers, Artificial Intelligence
9 points by snowkylin 55 days ago | 0 comments
Goku Flow Based Video Generative Foundation Models (github.com/Saiyan-World)
Goku is a new family of joint image-and-video generation models based on rectified flow Transformers.
Generative AI, Video Generation, Deep Learning, Transformers
34 points by lastdong 74 days ago | 11 comments
How has DeepSeek improved the Transformer architecture? (epoch.ai)
DeepSeek has recently released DeepSeek v3, which is currently state-of-the-art in benchmark performance among open-weight models, alongside a technical report describing in some detail the training of the model. Impressively, they’ve achieved this SOTA performance by only using 2.8 million H800 hours of training hardware time—equivalent to about 4e24 FLOP if we assume 40% MFU. This is about ten times less training compute than the similarly performing Llama 3.1 405B.
Transformers, Generative AI, Machine Learning
258 points by superasn 88 days ago | 68 comments
RWKV Language Model (rwkv.com)
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose".
RNNs, GPT, Transformers
183 points by simonpure 116 days ago | 52 comments
A transformer supply crisis bottlenecks energy projects (ieee.org)
A transformer supply crisis bottlenecks energy projects
Energy, Supply Chain, Infrastructure, Technology, Transformers
115 points by TaurenHunter 134 days ago | 108 comments
Laser: Attention with Exponential Transformation (arxiv.org)
Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention.
Transformers, Machine Learning, Attention Mechanisms, Deep Learning
13 points by PaulHoule 149 days ago | 0 comments
You could have designed state of the art positional encoding (fleetwood.dev)
This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is expected.
Transformers, Machine Learning, Computer Science, Deep Learning
216 points by Philpax 160 days ago | 46 comments
Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (rccchoudhury.github.io)
We present Run-Length Tokenization (RLT), a simple and efficient approach to speed up video transformers by removing redundant tokens from the input.
Computer Vision, Machine Learning, Video Processing, Deep Learning, Transformers
75 points by jasondavies 161 days ago | 16 comments
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)
Deep Learning, Machine Learning, Artificial Intelligence, Transformers
174 points by og_kalu 176 days ago | 33 comments
Oasis: A Universe in a Transformer (decart.ai)
We're excited to announce Oasis, the first playable, realtime, open-world AI model — it's an interactive video game, but generated end-to-end by a transformer on a frame-by-frame basis.
Artificial Intelligence, Video Games, Transformers
11 points by EvgeniyZh 176 days ago | 1 comments
Transformers Utilization in Chart Understanding: A Review of Advances and Future (arxiv.org)
In recent years, interest in vision-language tasks has grown, especially those involving chart interactions.
Transformers, Machine Learning, Computer Vision, Data Visualization
39 points by sandwichsphinx 186 days ago | 2 comments
New Transformer architecture modifications from Nvidia researchers (twitter.com)
Transformers, Artificial Intelligence, Computer Science, Research, Nvidia
4 points by amichail 189 days ago | 0 comments
Trap – Transformers in APL (github.com/BobMcDear)
trap is an implementation of autoregressive transformers - namely, GPT2 - in APL. In addition to containing the complete definition of GPT, it also supports backpropagation and training with Adam, achieving parity with the PyTorch reference code.
Transformers, APL, Language Models, Machine Learning, Open Source
95 points by tlack 201 days ago | 30 comments
Transformers 2.0: What Ilya and Sam Might Have Missed (ycombinator.com)
Transformers 2.0: What Ilya and Sam Might Have Missed
Transformers, Artificial Intelligence
5 points by metehan777 202 days ago | 1 comments
Transformers for Ruby (github.com/ankane)
Transformers, Ruby, Software, New Releases
366 points by felipemesquita 249 days ago | 52 comments
Transformers in music recommendation (research.google)
Transformers, Music Recommendation, Machine Learning
211 points by panarky 250 days ago | 125 comments
Synthesizing Abstract Transformers for Reduced-Product Domains (arxiv.org)
Transformers, Machine Learning, Artificial Intelligence, Computer Science
28 points by luu 252 days ago | 0 comments