Hacker News with Generative AI: Compression

CMU research shows compression alone may unlock AI puzzle-solving abilities (arstechnica.com)
A pair of Carnegie Mellon University researchers recently discovered hints that the process of compressing information can solve complex reasoning tasks without pre-training on a large number of examples.

Artificial Intelligence, Research, Computer Science, Compression

5 points by PaulHoule 14 days ago | 0 comments

MinLZ: Efficient and fast Snappy/LZ4 style compressor in Go (Apache 2.0) (github.com/minio)
MinLZ is a LZ77-type compressor with a fixed byte-aligned encoding, in the similar class to Snappy and LZ4.

Compression, Open Source, Software, Performance

18 points by klauspost 16 days ago | 4 comments

zlib-ng: zlib replacement with optimizations for "next generation" systems (github.com/zlib-ng)
zlib replacement with optimizations for "next generation" systems.

Software, Compression, Optimization, GitHub

55 points by tosh 17 days ago | 14 comments

Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (arxiv.org)
To address this challenge, we present RocketKV, a training-free KV cache compression strategy designed specifically to reduce both memory bandwidth and capacity demand of KV cache during the decode phase.

AI, Computer Science, Compression

4 points by PaulHoule 24 days ago | 0 comments

Spark Texture Compression 1.2 (ludicon.com)
I’m excited to announce that Spark 1.2 is finally out!

New Releases, Software, Compression

33 points by luu 27 days ago | 8 comments

What if we just didn't decompress it? (spiraldb.com)
Vortex is unique in the way it evaluates filter and projection expressions by supporting full compute push-down, in many cases avoiding decompression entirely.

Database, Compression, Optimization, Performance

12 points by gatesn 28 days ago | 2 comments

An Experimental Study of Bitmap Compression vs. Inverted List Compression (dl.acm.org)
Bitmap compression has been studied extensively in the database area and many efficient compression schemes were proposed, e.g., BBC, WAH, EWAH, and Roaring. Inverted list compression is also a well-studied topic in the information retrieval community and many inverted list compression algorithms were developed as well, e.g., VB, PforDelta, GroupVB, Simple8b, and SIMDPforDelta.

Database, Information Retrieval, Compression, Algorithms, Experimental Studies

32 points by westurner 33 days ago | 6 comments

Zlib-Rs Is Not Only Safer but Now Outperforming Zlib C Implementations (phoronix.com)
Zlib-rs as a Rust programming language implementation of the Zlib file format for better safety is now beginning to outperform the C implementations of the widely-used Zlib.

Rust, Compression, Performance, Software, C

63 points by mrpotato 36 days ago | 17 comments

Lzbench compression benchmark (morotti.github.io)
lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors.

Compression, Benchmarking, Software

23 points by todsacerdoti 50 days ago | 6 comments

Bzip3: A spiritual successor to BZip2 (github.com/kspalaiologos)
A better, faster and stronger spiritual successor to BZip2. Features higher compression ratios and better performance thanks to a order-0 context mixing entropy coder, a fast Burrows-Wheeler transform code making use of suffix arrays and a RLE with Lempel Ziv+Prediction pass based on LZ77-style string matching and PPM-style context modeling.

Compression, Software, Programming, Algorithms, New Releases

355 points by tosh 60 days ago | 176 comments

Deflate Decompression in C++23 (garymm.org)
In this post I describe some things I learned while working on Starflate, an implementation of Deflate decompression in C++23 that I wrote with my friend Oliver Lee.

C++, Programming, Compression, Software

3 points by spearman 60 days ago | 0 comments

Query Engines: Gatekeepers of the Parquet File Format (duckdb.org)
TL;DR: Mainstream query engines do not support reading newer Parquet encodings, forcing systems like DuckDB to default to writing older encodings, thereby sacrificing compression.

Database Systems, Data Formats, Performance, Compression, Query Engines

32 points by tosh 63 days ago | 1 comments

Show HN: Plik – a tiny FUSE filesystem with compression and deduplication (sytes.net)
File system in user space (FUSE) with compression and deduplication. Written in a single C file (1K LOC). Uses OpenSSL for hashing, zlib for compression and SQLite for data storage.

Operating Systems, File Systems, Compression, Deduplication

5 points by zeymejbdv 64 days ago | 0 comments

Fabrice Bellard's Ts_SMS: Short Message Compression Using LLM (bellard.org)
ts_sms: Short Message Compression using Large Language Models

Compression, Language Models, Artificial Intelligence, Software

10 points by BiteCode_dev 93 days ago | 4 comments

Short Message Compression Using LLMs (bellard.org)

Compression

261 points by chunkles 97 days ago | 116 comments

Linux EFI Zboot Abandoning "Compression Library Museum", Focusing on Gzip, ZSTD (phoronix.com)
The Linux kernel EFI Zboot code for carrying the Linux kernel image for EFI systems in compressed form is doing away with its "compression library museum" of offering Gzip, LZ4, LZMA, LZO, XZ, and Zstd compression options to instead just focus on Gzip and Zstd compression support.

Linux Kernel, EFI, Compression, Performance Optimization

70 points by rbanffy 115 days ago | 19 comments

BC7 optimal solid-color blocks (wordpress.com)
That’s right, it’s another texture compression blog post! I’ll keep it short. By “solid-color block”, I mean a 4×4 block of pixels that all have the same color. ASTC has a dedicated encoding for these (“void-extent blocks”), BC7 does not. Therefore we have an 8-bit RGBA input color and want to figure out how to best encode that color with the encoding options we have.

Graphics, Compression, Image Processing, Computer Graphics

14 points by luu 149 days ago | 1 comments

PeaZip 10.0.0 Released (peazip.github.io)
PeaZip 10.0.0 comes with a revamped GUI, providing more icon sizes, updated Themes and compression pre-sets, and better organized menus.

New Releases, Software, GUI, Compression

41 points by thunderbong 158 days ago | 15 comments

RRR: A Succinct Rank/Select Index for Bit Vectors (2011) (alexbowe.com)
This blog post will give an overview of a static bitsequence data structure known as RRR, which answers arbitrary length rank queries in $\mathcal{O}(1)$ time, and provides implicit compression.

Data Structures, Algorithms, Computer Science, Compression

26 points by Tomte 164 days ago | 0 comments

Maximum best compression for binary data: xz (LZMA) vs. ZSTD vs. 7z vs. bzip2 (dwaves.de)
once upon a time, compressing massive amounts of binary was required.

Compression, File Formats, Software, Performance

8 points by delduca 167 days ago | 4 comments

RFC9659: Window Sizing for Zstandard Content Encoding on the Web (rfc-editor.org)

Web Standards, Compression, Networking, RFC, Zstandard

8 points by mfiguiere 187 days ago | 0 comments

SQLite Transparent Compression (github.com/phiresky)
Extension for sqlite that provides transparent dictionary-based row-level compression for sqlite. This basically allows you to compress entries in a sqlite database almost as well as if you were compressing the whole DB file, but while retaining random access.

Database, Compression, SQLite, Software, Open Source

5 points by edweis 197 days ago | 2 comments

An SVE backend for astcenc (Adaptive Scalable Texture Compression Encoder) (solidpixel.github.io)

Compression, Image Processing, Software, Backend, ASTC

9 points by matt_d 239 days ago | 0 comments

The Evolution of Extreme LLM Compression: From Quip to AQLM with PV-Tuning (medium.com)

Compression, AI

10 points by annaerma 240 days ago | 0 comments

AQLM and PV-Tuning: methods that compress LLMs by 8 times, retain 95% quality (github.com/Vahe1994)

Compression, AI, Machine Learning

10 points by annaerma 253 days ago | 0 comments

Borg 2.0 beta (deduplicating backup program with compression and encryption) (borgbackup.org)