Hacker News with Generative AI: Quantization

Pushing the Limits of LLM Quantization via the Linearity Theorem (arxiv.org)
Quantizing large language models has become a standard way to reduce their memory and computational costs.
Machine Learning, Artificial Intelligence, Optimization, Quantization
95 points by felineflock 6 days ago | 2 comments
SplitQuantV2: Enhancing Low-Bit Quantization of LLMs Without GPUs (arxiv.org)
The quantization of large language models (LLMs) is crucial for deploying them on devices with limited computational resources.
Quantization, Computer Science, Artificial Intelligence
34 points by PaulHoule 31 days ago | 9 comments
16-Bit to 1-Bit: Visual KV Cache Quantization for Efficient Multimodal LLMs (arxiv.org)
Multimodal Large Language Models (MLLMs) have achieved remarkable success across various applications, yet their computational overhead during deployment remains a critical challenge.
Computer Vision, Artificial Intelligence, Optimization, Quantization
87 points by PaulHoule 52 days ago | 1 comments
SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup (hanlab.mit.edu)
A new post-training training quantization paradigm for diffusion models, which quantize both the weights and activations of FLUX.1 to 4 bits, achieving 3.5× memory and 8.7× latency reduction on a 16GB laptop 4090 GPU.
Machine Learning, Deep Learning, Computer Vision, Hardware, Quantization
179 points by lmxyy 168 days ago | 65 comments
VPTQ: Extreme low-bit Quantization for real LLMs (github.com/microsoft)
Vector Post-Training Quantization (VPTQ) is a novel Post-Training Quantization method that leverages Vector Quantization to high accuracy on LLMs at an extremely low bit-width (<2-bit). VPTQ can compress 70B, even the 405B model, to 1-2 bits without retraining and maintain high accuracy.
Machine Learning, Quantization, Artificial Intelligence
20 points by OpenSourceRonin 187 days ago | 1 comments
EfficientQAT: LLM Quantization, gets a 2-bit llama2-70B outperform regular 13B (reddit.com)
Quantization, Performance Optimization, AI
21 points by jackbravo 282 days ago | 0 comments
Towards Optimal LLM Quantization (picovoice.ai)
Optimization, AI, Quantization
18 points by bejager 331 days ago | 8 comments
On-Device LLM Inference Powered by X-Bit Quantization (github.com/Picovoice)
Machine Learning, Artificial Intelligence, Quantization
15 points by dynamix 332 days ago | 0 comments