Hacker News with Generative AI: Data Compression

Burrows–Wheeler Transform (wikipedia.org)
The Burrows–Wheeler transform (BWT, also called block-sorting compression) rearranges a character string into runs of similar characters.

Data Compression, Algorithms, Computer Science

19 points by tosh 20 days ago | 2 comments

Reduce your LLM agent costs by 90% with structure-preserving HTML compression (github.com/emmetify)
Cut your LLM processing costs by up to 90% by transforming verbose HTML into efficient Emmet notation, without losing structural integrity.

Cost Optimization, HTML, Web Development, Data Compression

13 points by maledorak 24 days ago | 10 comments

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search (arxiv.org)
Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine.

Machine Learning, Data Compression

151 points by fzliu 30 days ago | 6 comments

NNCP: Lossless Data Compression with Neural Networks (bellard.org)
NNCP is an experiment to build a practical lossless data compressor with neural networks.

Neural Networks, Data Compression, Machine Learning

10 points by ksec 53 days ago | 0 comments

Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data (kevinslin.com)
On a rainy March day in 1538, Thomas Howard, the Duke of Norfolk, found himself confined within the cold stone walls of his grand estate. His fingers reluctantly penned a letter to sell his cherished lands to settle longstanding debts.

Log Aggregation, Data Compression, Software, Technology

140 points by benshumaker 78 days ago | 84 comments

Huffman Coding (wikipedia.org)
In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression.

Computer Science, Information Theory, Data Compression

5 points by tosh 86 days ago | 3 comments

The $5000 Compression Challenge (2001) (patrickcraig.co.uk)
Mike Goldman makes another offer: I will attach a prize of $5,000 to anyone who successfully meets this challenge. First, the contestant will tell me HOW LONG of a data file to generate. Second, I will generate the data file, and send it to the contestant.

Data Compression, Challenges, Prizes

181 points by ekiauhce 90 days ago | 185 comments

Binary vector embeddings are so cool (emschwartz.me)
Vector embeddings by themselves are pretty neat. Binary quantized vector embeddings are extra impressive. In short, they can retain 95+% retrieval accuracy with 32x compression and ~25x retrieval speedup. Let's get into how this works and why it's so crazy.

Machine Learning, Computer Science, Data Compression

87 points by emschwartz 102 days ago | 11 comments

Consistently faster and smaller compressed bitmaps with Roaring (2016) (arxiv.org)
Compressed bitmap indexes are used in databases and search engines.

Databases, Search Engines, Data Compression, Performance Optimization, Algorithms

54 points by hambandit 112 days ago | 8 comments

Show HN: Vortex – a high-performance columnar file format (github.com/spiraldb)
Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

Databases, Data Storage, Performance, Data Compression, Open Source

249 points by gatesn 130 days ago | 61 comments

Parallel PNG Proposal (2021) (github.com/DavidBuchanan314)
This is a proof-of-concept implementation of a parallel-decodable PNG format, based on ideas from https://github.com/brion/mtpng

Image Processing, Computer Vision, Data Compression, Open Source

43 points by networked 154 days ago | 3 comments

Large Text Compression Benchmark (mattmahoney.net)
This competition ranks lossless data compression programs by the compressed size (including the size of the decompression program) of the first 109 bytes of the XML text dump of the English version of Wikipedia on Mar. 3, 2006.

Data Compression, Benchmarking, Computer Science, Wikipedia

97 points by redeux 156 days ago | 55 comments

Compression Dictionary Transport (ietf.org)
This document specifies a mechanism for dictionary-based compression in the Hypertext Transfer Protocol (HTTP). By utilizing this technique, clients and servers can reduce the size of transmitted data, leading to improved performance and reduced bandwidth consumption. This document extends existing HTTP compression methods and provides guidelines for the delivery and use of compression dictionaries within the HTTP protocol.

Networking, HTTP, Data Compression

70 points by tosh 159 days ago | 50 comments

Faster Inverse BWT (2021) (blogspot.com)

Data Compression, Algorithms, Computer Science

65 points by fanf2 183 days ago | 2 comments

Compressing data with sample points and polynomial interpolation (johndcook.com)

Data Compression, Mathematics, Algorithms

63 points by speckx 184 days ago | 11 comments