Hacker News with Generative AI: Quantization

SplitQuantV2: Enhancing Low-Bit Quantization of LLMs Without GPUs (arxiv.org)
The quantization of large language models (LLMs) is crucial for deploying them on devices with limited computational resources.
Quantization, Computer Science, Artificial Intelligence
34 points by PaulHoule 8 days ago | 9 comments
16-Bit to 1-Bit: Visual KV Cache Quantization for Efficient Multimodal LLMs (arxiv.org)
Multimodal Large Language Models (MLLMs) have achieved remarkable success across various applications, yet their computational overhead during deployment remains a critical challenge.
Computer Vision, Artificial Intelligence, Optimization, Quantization
87 points by PaulHoule 29 days ago | 1 comments
SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup (hanlab.mit.edu)
A new post-training training quantization paradigm for diffusion models, which quantize both the weights and activations of FLUX.1 to 4 bits, achieving 3.5× memory and 8.7× latency reduction on a 16GB laptop 4090 GPU.
Machine Learning, Deep Learning, Computer Vision, Hardware, Quantization
179 points by lmxyy 145 days ago | 65 comments
VPTQ: Extreme low-bit Quantization for real LLMs (github.com/microsoft)
Vector Post-Training Quantization (VPTQ) is a novel Post-Training Quantization method that leverages Vector Quantization to high accuracy on LLMs at an extremely low bit-width (<2-bit). VPTQ can compress 70B, even the 405B model, to 1-2 bits without retraining and maintain high accuracy.
Machine Learning, Quantization, Artificial Intelligence
20 points by OpenSourceRonin 164 days ago | 1 comments
EfficientQAT: LLM Quantization, gets a 2-bit llama2-70B outperform regular 13B (reddit.com)
Quantization, Performance Optimization, AI
21 points by jackbravo 259 days ago | 0 comments
Towards Optimal LLM Quantization (picovoice.ai)
Optimization, AI, Quantization
18 points by bejager 308 days ago | 8 comments
On-Device LLM Inference Powered by X-Bit Quantization (github.com/Picovoice)
Machine Learning, Artificial Intelligence, Quantization
15 points by dynamix 309 days ago | 0 comments