Hacker News with Generative AI: Inference

Accelerated AI Inference via Dynamic Execution Methods (arxiv.org)
In this paper, we focus on Dynamic Execution techniques that optimize the computation flow based on input.

Artificial Intelligence, Machine Learning, Computer Science, Optimization, Inference

63 points by PaulHoule 8 days ago | 3 comments

Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference (cerebras.ai)
Frontier AI now runs at instant speed. Last week we ran a customer workload on Llama 3.1 405B at 969 tokens/s – a new record for Meta’s frontier model. Llama 3.1 405B on Cerebras is by far the fastest frontier model in the world – 12x faster than GPT-4o and 18x faster than Claude 3.5 Sonnet. In addition, we achieved the highest performance at 128K context length and shortest time-to-first-token latency, as measured by Artificial Analysis.

Artificial Intelligence, Speed, Performance, Inference

427 points by benchmarkist 15 days ago | 156 comments

GDDR7 Memory Supercharges AI Inference (semiengineering.com)
High bandwidth and low latency are paramount for AI-powered edge and endpoints.

AI, Hardware, Memory, Inference, Edge Computing

68 points by PaulHoule 36 days ago | 34 comments

Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s (cerebras.ai)
Today we’re announcing the biggest update to Cerebras Inference since launch. Cerebras Inference now runs Llama 3.1-70B at an astounding 2,100 tokens per second – a 3x performance boost over the prior release.

Artificial Intelligence, Hardware, Performance, Inference

147 points by campers 40 days ago | 84 comments

AMD GPU Inference (github.com/slashml)
This project provides a Docker-based inference engine for running Large Language Models (LLMs) on AMD GPUs. It's designed to work with models from Hugging Face, with a focus on the LLaMA model family.

Generative AI, AMD GPUs, Docker, Inference

270 points by fazkan 62 days ago | 99 comments

Cerebras Inference: AI at Instant Speed (cerebras.ai)

AI, Inference, Hardware, Performance

174 points by meetpateltech 98 days ago | 71 comments

Cerebras launches inference for Llama 3.1; benchmarked at 1846 tokens/s on 8B (twitter.com)

Generative AI, Inference, Benchmarks, Hardware

95 points by _micah_h 98 days ago | 42 comments

Cerebras Launches the Fastest AI Inference (cerebras.ai)

AI, Hardware, Inference

13 points by cs-fan-101 98 days ago | 1 comments

Inference is free and instant (fume.substack.com)