Hacker News with Generative AI: Transformers

Real-Time Introspective Compression for Transformers (github.com/Dicklesworthstone)
This article proposes a novel approach to address both problems simultaneously.

Transformers, Compression, Artificial Intelligence, Machine Learning

8 points by eigenvalue 8 hours ago | 11 comments

How DeepSeek Rewrote the Transformer [video] (youtube.com)

Transformers, Deep Learning, Computer Vision, Artificial Intelligence, Video

5 points by chetangole 5 days ago | 0 comments

VGGT: Visual Geometry Grounded Transformer (github.com/facebookresearch)

Computer Vision, Transformers, Artificial Intelligence, Deep Learning, Facebook Research

190 points by xnx 9 days ago | 42 comments

Transformers as Support Vector Machines (2023) (arxiv.org)
Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP.

Transformers, Machine Learning, Computer Science, Artificial Intelligence

3 points by TaurenHunter 16 days ago | 0 comments

Transformers Without Normalization (jiachenzhu.github.io)
This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique.

Transformers, Deep Learning, Machine Learning

260 points by hellollm 19 days ago | 32 comments

Understanding Transformers (beyond the Math) – kalomaze's kalomazing blog (bearblog.dev)
Maybe you don't want to attempt the conventional approaches for understanding the transformer architecture for language models. If you're anything like me, an informal approach is what you'd prefer - one that helps you reason about what's happening with these models in the abstract, without requiring mastery on the technical level to begin with.

Machine Learning, Deep Learning, Language Models, Transformers

8 points by bilsbie 24 days ago | 2 comments

RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals (arxiv.org)
Transformers have achieved great success in effectively processing sequential data such as text.

Transformers, Machine Learning, Deep Learning, Computer Science

3 points by PaulHoule 26 days ago | 0 comments

Some thoughts on autoregressive models (wonderfall.dev)
Most generative AI models nowadays are autoregressive. That means they’re following the concept of next token prediction, and the transformer architecture is the current implementation that has been used for years now thanks to its computational efficiency.

Generative AI, Artificial Intelligence, Machine Learning, Transformers

79 points by Wonderfall 30 days ago | 58 comments

Go-attention: A full attention mechanism and transformer in pure Go (github.com/takara-ai)
From the Frontier Research Team at takara.ai we present the first pure Go implementation of attention mechanisms and transformer layers, designed for high performance and ease of use.

Go, Attention Mechanism, Transformers, Machine Learning

168 points by PaulHoule 30 days ago | 85 comments

Show HN: A Transformer model that preserves logical equivalence (huggingface.co)
Running

Transformers, Artificial Intelligence

9 points by snowkylin 31 days ago | 0 comments

Goku Flow Based Video Generative Foundation Models (github.com/Saiyan-World)
Goku is a new family of joint image-and-video generation models based on rectified flow Transformers.

Generative AI, Video Generation, Deep Learning, Transformers

34 points by lastdong 50 days ago | 11 comments

How has DeepSeek improved the Transformer architecture? (epoch.ai)
DeepSeek has recently released DeepSeek v3, which is currently state-of-the-art in benchmark performance among open-weight models, alongside a technical report describing in some detail the training of the model. Impressively, they’ve achieved this SOTA performance by only using 2.8 million H800 hours of training hardware time—equivalent to about 4e24 FLOP if we assume 40% MFU. This is about ten times less training compute than the similarly performing Llama 3.1 405B.

Transformers, Generative AI, Machine Learning

258 points by superasn 64 days ago | 68 comments

RWKV Language Model (rwkv.com)
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose".

RNNs, GPT, Transformers

183 points by simonpure 93 days ago | 52 comments

A transformer supply crisis bottlenecks energy projects (ieee.org)
A transformer supply crisis bottlenecks energy projects

Energy, Supply Chain, Infrastructure, Technology, Transformers

115 points by TaurenHunter 111 days ago | 108 comments

Laser: Attention with Exponential Transformation (arxiv.org)
Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention.

Transformers, Machine Learning, Attention Mechanisms, Deep Learning

13 points by PaulHoule 125 days ago | 0 comments

You could have designed state of the art positional encoding (fleetwood.dev)
This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is expected.

Transformers, Machine Learning, Computer Science, Deep Learning

216 points by Philpax 136 days ago | 46 comments

Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (rccchoudhury.github.io)
We present Run-Length Tokenization (RLT), a simple and efficient approach to speed up video transformers by removing redundant tokens from the input.

Computer Vision, Machine Learning, Video Processing, Deep Learning, Transformers

75 points by jasondavies 138 days ago | 16 comments

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)

Deep Learning, Machine Learning, Artificial Intelligence, Transformers

174 points by og_kalu 152 days ago | 33 comments

Oasis: A Universe in a Transformer (decart.ai)
We're excited to announce Oasis, the first playable, realtime, open-world AI model — it's an interactive video game, but generated end-to-end by a transformer on a frame-by-frame basis.

Artificial Intelligence, Video Games, Transformers

11 points by EvgeniyZh 153 days ago | 1 comments

Transformers Utilization in Chart Understanding: A Review of Advances and Future (arxiv.org)
In recent years, interest in vision-language tasks has grown, especially those involving chart interactions.

Transformers, Machine Learning, Computer Vision, Data Visualization

39 points by sandwichsphinx 163 days ago | 2 comments

New Transformer architecture modifications from Nvidia researchers (twitter.com)

Transformers, Artificial Intelligence, Computer Science, Research, Nvidia

4 points by amichail 165 days ago | 0 comments

Trap – Transformers in APL (github.com/BobMcDear)
trap is an implementation of autoregressive transformers - namely, GPT2 - in APL. In addition to containing the complete definition of GPT, it also supports backpropagation and training with Adam, achieving parity with the PyTorch reference code.

Transformers, APL, Language Models, Machine Learning, Open Source

95 points by tlack 177 days ago | 30 comments

Transformers 2.0: What Ilya and Sam Might Have Missed (ycombinator.com)
Transformers 2.0: What Ilya and Sam Might Have Missed

Transformers, Artificial Intelligence

5 points by metehan777 179 days ago | 1 comments

Transformers for Ruby (github.com/ankane)

Transformers, Ruby, Software, New Releases

366 points by felipemesquita 226 days ago | 52 comments

Transformers in music recommendation (research.google)

Transformers, Music Recommendation, Machine Learning

211 points by panarky 226 days ago | 125 comments

Synthesizing Abstract Transformers for Reduced-Product Domains (arxiv.org)