Hacker News with Generative AI: Quantization

SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup (hanlab.mit.edu)
A new post-training training quantization paradigm for diffusion models, which quantize both the weights and activations of FLUX.1 to 4 bits, achieving 3.5× memory and 8.7× latency reduction on a 16GB laptop 4090 GPU.
Machine Learning, Deep Learning, Computer Vision, Hardware, Quantization
179 points by lmxyy 12 days ago | 65 comments
VPTQ: Extreme low-bit Quantization for real LLMs (github.com/microsoft)
Vector Post-Training Quantization (VPTQ) is a novel Post-Training Quantization method that leverages Vector Quantization to high accuracy on LLMs at an extremely low bit-width (<2-bit). VPTQ can compress 70B, even the 405B model, to 1-2 bits without retraining and maintain high accuracy.
Machine Learning, Quantization, Artificial Intelligence
20 points by OpenSourceRonin 31 days ago | 1 comments
EfficientQAT: LLM Quantization, gets a 2-bit llama2-70B outperform regular 13B (reddit.com)
Quantization, Performance Optimization, AI
21 points by jackbravo 127 days ago | 0 comments
Towards Optimal LLM Quantization (picovoice.ai)
Optimization, AI, Quantization
18 points by bejager 175 days ago | 8 comments
On-Device LLM Inference Powered by X-Bit Quantization (github.com/Picovoice)
Machine Learning, Artificial Intelligence, Quantization
15 points by dynamix 176 days ago | 0 comments