Hacker News with Generative AI: Compression

What if we just didn't decompress it? (spiraldb.com)
Vortex is unique in the way it evaluates filter and projection expressions by supporting full compute push-down, in many cases avoiding decompression entirely.

Database, Compression, Optimization, Performance

12 points by gatesn 1 day ago | 2 comments

An Experimental Study of Bitmap Compression vs. Inverted List Compression (dl.acm.org)
Bitmap compression has been studied extensively in the database area and many efficient compression schemes were proposed, e.g., BBC, WAH, EWAH, and Roaring. Inverted list compression is also a well-studied topic in the information retrieval community and many inverted list compression algorithms were developed as well, e.g., VB, PforDelta, GroupVB, Simple8b, and SIMDPforDelta.

Database, Information Retrieval, Compression, Algorithms, Experimental Studies

31 points by westurner 6 days ago | 6 comments

Zlib-Rs Is Not Only Safer but Now Outperforming Zlib C Implementations (phoronix.com)
Zlib-rs as a Rust programming language implementation of the Zlib file format for better safety is now beginning to outperform the C implementations of the widely-used Zlib.

Rust, Compression, Performance, Software, C

63 points by mrpotato 10 days ago | 17 comments

Lzbench compression benchmark (morotti.github.io)
lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors.

Compression, Benchmarking, Software

23 points by todsacerdoti 23 days ago | 6 comments

Bzip3: A spiritual successor to BZip2 (github.com/kspalaiologos)
A better, faster and stronger spiritual successor to BZip2. Features higher compression ratios and better performance thanks to a order-0 context mixing entropy coder, a fast Burrows-Wheeler transform code making use of suffix arrays and a RLE with Lempel Ziv+Prediction pass based on LZ77-style string matching and PPM-style context modeling.

Compression, Software, Programming, Algorithms, New Releases

355 points by tosh 33 days ago | 176 comments

Deflate Decompression in C++23 (garymm.org)
In this post I describe some things I learned while working on Starflate, an implementation of Deflate decompression in C++23 that I wrote with my friend Oliver Lee.

C++, Programming, Compression, Software

3 points by spearman 34 days ago | 0 comments

Query Engines: Gatekeepers of the Parquet File Format (duckdb.org)
TL;DR: Mainstream query engines do not support reading newer Parquet encodings, forcing systems like DuckDB to default to writing older encodings, thereby sacrificing compression.

Database Systems, Data Formats, Performance, Compression, Query Engines

32 points by tosh 36 days ago | 1 comments

Show HN: Plik – a tiny FUSE filesystem with compression and deduplication (sytes.net)
File system in user space (FUSE) with compression and deduplication. Written in a single C file (1K LOC). Uses OpenSSL for hashing, zlib for compression and SQLite for data storage.

Operating Systems, File Systems, Compression, Deduplication

5 points by zeymejbdv 37 days ago | 0 comments

Fabrice Bellard's Ts_SMS: Short Message Compression Using LLM (bellard.org)
ts_sms: Short Message Compression using Large Language Models

Compression, Language Models, Artificial Intelligence, Software

10 points by BiteCode_dev 67 days ago | 4 comments

Short Message Compression Using LLMs (bellard.org)

Compression

261 points by chunkles 70 days ago | 116 comments

Linux EFI Zboot Abandoning "Compression Library Museum", Focusing on Gzip, ZSTD (phoronix.com)
The Linux kernel EFI Zboot code for carrying the Linux kernel image for EFI systems in compressed form is doing away with its "compression library museum" of offering Gzip, LZ4, LZMA, LZO, XZ, and Zstd compression options to instead just focus on Gzip and Zstd compression support.

Linux Kernel, EFI, Compression, Performance Optimization

70 points by rbanffy 88 days ago | 19 comments

BC7 optimal solid-color blocks (wordpress.com)
That’s right, it’s another texture compression blog post! I’ll keep it short. By “solid-color block”, I mean a 4×4 block of pixels that all have the same color. ASTC has a dedicated encoding for these (“void-extent blocks”), BC7 does not. Therefore we have an 8-bit RGBA input color and want to figure out how to best encode that color with the encoding options we have.

Graphics, Compression, Image Processing, Computer Graphics

14 points by luu 123 days ago | 1 comments

PeaZip 10.0.0 Released (peazip.github.io)
PeaZip 10.0.0 comes with a revamped GUI, providing more icon sizes, updated Themes and compression pre-sets, and better organized menus.

New Releases, Software, GUI, Compression

41 points by thunderbong 132 days ago | 15 comments

RRR: A Succinct Rank/Select Index for Bit Vectors (2011) (alexbowe.com)
This blog post will give an overview of a static bitsequence data structure known as RRR, which answers arbitrary length rank queries in $\mathcal{O}(1)$ time, and provides implicit compression.

Data Structures, Algorithms, Computer Science, Compression

26 points by Tomte 138 days ago | 0 comments

Maximum best compression for binary data: xz (LZMA) vs. ZSTD vs. 7z vs. bzip2 (dwaves.de)
once upon a time, compressing massive amounts of binary was required.

Compression, File Formats, Software, Performance

8 points by delduca 141 days ago | 4 comments

RFC9659: Window Sizing for Zstandard Content Encoding on the Web (rfc-editor.org)

Web Standards, Compression, Networking, RFC, Zstandard

8 points by mfiguiere 160 days ago | 0 comments

SQLite Transparent Compression (github.com/phiresky)
Extension for sqlite that provides transparent dictionary-based row-level compression for sqlite. This basically allows you to compress entries in a sqlite database almost as well as if you were compressing the whole DB file, but while retaining random access.

Database, Compression, SQLite, Software, Open Source

5 points by edweis 171 days ago | 2 comments

An SVE backend for astcenc (Adaptive Scalable Texture Compression Encoder) (solidpixel.github.io)

Compression, Image Processing, Software, Backend, ASTC

9 points by matt_d 212 days ago | 0 comments

The Evolution of Extreme LLM Compression: From Quip to AQLM with PV-Tuning (medium.com)

Compression, AI

10 points by annaerma 214 days ago | 0 comments

AQLM and PV-Tuning: methods that compress LLMs by 8 times, retain 95% quality (github.com/Vahe1994)

Compression, AI, Machine Learning

10 points by annaerma 227 days ago | 0 comments

Borg 2.0 beta (deduplicating backup program with compression and encryption) (borgbackup.org)