Hacker News with Generative AI: Data Compression

NNCP: Lossless Data Compression with Neural Networks (bellard.org)
NNCP is an experiment to build a practical lossless data compressor with neural networks.

Neural Networks, Data Compression, Machine Learning

10 points by ksec 18 days ago | 0 comments

Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data (kevinslin.com)
On a rainy March day in 1538, Thomas Howard, the Duke of Norfolk, found himself confined within the cold stone walls of his grand estate. His fingers reluctantly penned a letter to sell his cherished lands to settle longstanding debts.

Log Aggregation, Data Compression, Software, Technology

140 points by benshumaker 43 days ago | 84 comments

Huffman Coding (wikipedia.org)
In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression.

Computer Science, Information Theory, Data Compression

5 points by tosh 52 days ago | 3 comments

The $5000 Compression Challenge (2001) (patrickcraig.co.uk)
Mike Goldman makes another offer: I will attach a prize of $5,000 to anyone who successfully meets this challenge. First, the contestant will tell me HOW LONG of a data file to generate. Second, I will generate the data file, and send it to the contestant.

Data Compression, Challenges, Prizes

181 points by ekiauhce 55 days ago | 185 comments

Binary vector embeddings are so cool (emschwartz.me)
Vector embeddings by themselves are pretty neat. Binary quantized vector embeddings are extra impressive. In short, they can retain 95+% retrieval accuracy with 32x compression and ~25x retrieval speedup. Let's get into how this works and why it's so crazy.

Machine Learning, Computer Science, Data Compression

87 points by emschwartz 68 days ago | 11 comments

Consistently faster and smaller compressed bitmaps with Roaring (2016) (arxiv.org)
Compressed bitmap indexes are used in databases and search engines.

Databases, Search Engines, Data Compression, Performance Optimization, Algorithms

54 points by hambandit 77 days ago | 8 comments

Show HN: Vortex – a high-performance columnar file format (github.com/spiraldb)
Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

Databases, Data Storage, Performance, Data Compression, Open Source

249 points by gatesn 96 days ago | 61 comments

Parallel PNG Proposal (2021) (github.com/DavidBuchanan314)
This is a proof-of-concept implementation of a parallel-decodable PNG format, based on ideas from https://github.com/brion/mtpng

Image Processing, Computer Vision, Data Compression, Open Source

43 points by networked 120 days ago | 3 comments

Large Text Compression Benchmark (mattmahoney.net)
This competition ranks lossless data compression programs by the compressed size (including the size of the decompression program) of the first 109 bytes of the XML text dump of the English version of Wikipedia on Mar. 3, 2006.

Data Compression, Benchmarking, Computer Science, Wikipedia

97 points by redeux 122 days ago | 55 comments

Compression Dictionary Transport (ietf.org)
This document specifies a mechanism for dictionary-based compression in the Hypertext Transfer Protocol (HTTP). By utilizing this technique, clients and servers can reduce the size of transmitted data, leading to improved performance and reduced bandwidth consumption. This document extends existing HTTP compression methods and provides guidelines for the delivery and use of compression dictionaries within the HTTP protocol.

Networking, HTTP, Data Compression

70 points by tosh 125 days ago | 50 comments

Faster Inverse BWT (2021) (blogspot.com)

Data Compression, Algorithms, Computer Science

65 points by fanf2 149 days ago | 2 comments

Compressing data with sample points and polynomial interpolation (johndcook.com)

Data Compression, Mathematics, Algorithms

63 points by speckx 150 days ago | 11 comments