Hacker News with Generative AI: Transformers

Goku Flow Based Video Generative Foundation Models (github.com/Saiyan-World)
Goku is a new family of joint image-and-video generation models based on rectified flow Transformers.

Generative AI, Video Generation, Deep Learning, Transformers

34 points by lastdong 10 days ago | 11 comments

How has DeepSeek improved the Transformer architecture? (epoch.ai)
DeepSeek has recently released DeepSeek v3, which is currently state-of-the-art in benchmark performance among open-weight models, alongside a technical report describing in some detail the training of the model. Impressively, they’ve achieved this SOTA performance by only using 2.8 million H800 hours of training hardware time—equivalent to about 4e24 FLOP if we assume 40% MFU. This is about ten times less training compute than the similarly performing Llama 3.1 405B.

Transformers, Generative AI, Machine Learning

258 points by superasn 24 days ago | 68 comments

RWKV Language Model (rwkv.com)
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose".

RNNs, GPT, Transformers

183 points by simonpure 53 days ago | 52 comments

A transformer supply crisis bottlenecks energy projects (ieee.org)
A transformer supply crisis bottlenecks energy projects

Energy, Supply Chain, Infrastructure, Technology, Transformers

115 points by TaurenHunter 71 days ago | 108 comments

Laser: Attention with Exponential Transformation (arxiv.org)
Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention.

Transformers, Machine Learning, Attention Mechanisms, Deep Learning

13 points by PaulHoule 85 days ago | 0 comments

You could have designed state of the art positional encoding (fleetwood.dev)
This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is expected.

Transformers, Machine Learning, Computer Science, Deep Learning

216 points by Philpax 96 days ago | 46 comments

Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (rccchoudhury.github.io)
We present Run-Length Tokenization (RLT), a simple and efficient approach to speed up video transformers by removing redundant tokens from the input.

Computer Vision, Machine Learning, Video Processing, Deep Learning, Transformers

75 points by jasondavies 98 days ago | 16 comments

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)

Deep Learning, Machine Learning, Artificial Intelligence, Transformers

174 points by og_kalu 112 days ago | 33 comments

Oasis: A Universe in a Transformer (decart.ai)
We're excited to announce Oasis, the first playable, realtime, open-world AI model — it's an interactive video game, but generated end-to-end by a transformer on a frame-by-frame basis.

Artificial Intelligence, Video Games, Transformers

11 points by EvgeniyZh 113 days ago | 1 comments

Transformers Utilization in Chart Understanding: A Review of Advances and Future (arxiv.org)
In recent years, interest in vision-language tasks has grown, especially those involving chart interactions.

Transformers, Machine Learning, Computer Vision, Data Visualization

39 points by sandwichsphinx 123 days ago | 2 comments

New Transformer architecture modifications from Nvidia researchers (twitter.com)

Transformers, Artificial Intelligence, Computer Science, Research, Nvidia

4 points by amichail 125 days ago | 0 comments

Trap – Transformers in APL (github.com/BobMcDear)
trap is an implementation of autoregressive transformers - namely, GPT2 - in APL. In addition to containing the complete definition of GPT, it also supports backpropagation and training with Adam, achieving parity with the PyTorch reference code.

Transformers, APL, Language Models, Machine Learning, Open Source

95 points by tlack 137 days ago | 30 comments

Transformers 2.0: What Ilya and Sam Might Have Missed (ycombinator.com)
Transformers 2.0: What Ilya and Sam Might Have Missed

Transformers, Artificial Intelligence

5 points by metehan777 139 days ago | 1 comments

Transformers for Ruby (github.com/ankane)

Transformers, Ruby, Software, New Releases

366 points by felipemesquita 185 days ago | 52 comments

Transformers in music recommendation (research.google)

Transformers, Music Recommendation, Machine Learning

211 points by panarky 186 days ago | 125 comments

Synthesizing Abstract Transformers for Reduced-Product Domains (arxiv.org)