Hacker News with Generative AI: Transformers

RWKV Language Model (rwkv.com)
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose".

RNNs, GPT, Transformers

183 points by simonpure 18 days ago | 52 comments

A transformer supply crisis bottlenecks energy projects (ieee.org)
A transformer supply crisis bottlenecks energy projects

Energy, Supply Chain, Infrastructure, Technology, Transformers

115 points by TaurenHunter 36 days ago | 108 comments

Laser: Attention with Exponential Transformation (arxiv.org)
Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention.

Transformers, Machine Learning, Attention Mechanisms, Deep Learning

13 points by PaulHoule 50 days ago | 0 comments

You could have designed state of the art positional encoding (fleetwood.dev)
This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is expected.

Transformers, Machine Learning, Computer Science, Deep Learning

216 points by Philpax 61 days ago | 46 comments

Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (rccchoudhury.github.io)
We present Run-Length Tokenization (RLT), a simple and efficient approach to speed up video transformers by removing redundant tokens from the input.

Computer Vision, Machine Learning, Video Processing, Deep Learning, Transformers

75 points by jasondavies 63 days ago | 16 comments

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (arxiv.org)

Deep Learning, Machine Learning, Artificial Intelligence, Transformers

174 points by og_kalu 78 days ago | 33 comments

Oasis: A Universe in a Transformer (decart.ai)
We're excited to announce Oasis, the first playable, realtime, open-world AI model — it's an interactive video game, but generated end-to-end by a transformer on a frame-by-frame basis.

Artificial Intelligence, Video Games, Transformers

11 points by EvgeniyZh 78 days ago | 1 comments

Transformers Utilization in Chart Understanding: A Review of Advances and Future (arxiv.org)
In recent years, interest in vision-language tasks has grown, especially those involving chart interactions.

Transformers, Machine Learning, Computer Vision, Data Visualization

39 points by sandwichsphinx 88 days ago | 2 comments

New Transformer architecture modifications from Nvidia researchers (twitter.com)

Transformers, Artificial Intelligence, Computer Science, Research, Nvidia

4 points by amichail 91 days ago | 0 comments

Trap – Transformers in APL (github.com/BobMcDear)
trap is an implementation of autoregressive transformers - namely, GPT2 - in APL. In addition to containing the complete definition of GPT, it also supports backpropagation and training with Adam, achieving parity with the PyTorch reference code.

Transformers, APL, Language Models, Machine Learning, Open Source

95 points by tlack 102 days ago | 30 comments

Transformers 2.0: What Ilya and Sam Might Have Missed (ycombinator.com)
Transformers 2.0: What Ilya and Sam Might Have Missed

Transformers, Artificial Intelligence

5 points by metehan777 104 days ago | 1 comments

Transformers for Ruby (github.com/ankane)

Transformers, Ruby, Software, New Releases

366 points by felipemesquita 151 days ago | 52 comments

Transformers in music recommendation (research.google)

Transformers, Music Recommendation, Machine Learning

211 points by panarky 151 days ago | 125 comments

Synthesizing Abstract Transformers for Reduced-Product Domains (arxiv.org)