Data Compression Nerds Hate This One Trick [video]
(media.ccc.de)
How one guy in his bedroom (kind of) beat all of PNG's combined multi-decade effort in one year, and why that's strange.
How one guy in his bedroom (kind of) beat all of PNG's combined multi-decade effort in one year, and why that's strange.
Show HN: BSE – Semantic Zip Engine for Text, Image and Audio
(ycombinator.com)
We built BSE (Bramble Semantic Engine) – a semantic compressor that transforms natural inputs into low-dimensional structured representations.
We built BSE (Bramble Semantic Engine) – a semantic compressor that transforms natural inputs into low-dimensional structured representations.
Compress Better, Compute Bigger
(ironarray.io)
Have you ever experienced the frustration of not being able to analyze a dataset because it's too large to fit in memory? Or perhaps you've encountered the memory wall, where computation is hindered by slow memory access? These are common challenges in data science and high-performance computing. The developers of Blosc and Blosc2 have consistently focused on achieving compression and decompression speeds that approach or even exceed memory bandwidth limits.
Have you ever experienced the frustration of not being able to analyze a dataset because it's too large to fit in memory? Or perhaps you've encountered the memory wall, where computation is hindered by slow memory access? These are common challenges in data science and high-performance computing. The developers of Blosc and Blosc2 have consistently focused on achieving compression and decompression speeds that approach or even exceed memory bandwidth limits.
Burrows–Wheeler Transform
(wikipedia.org)
The Burrows–Wheeler transform (BWT, also called block-sorting compression) rearranges a character string into runs of similar characters.
The Burrows–Wheeler transform (BWT, also called block-sorting compression) rearranges a character string into runs of similar characters.
Reduce your LLM agent costs by 90% with structure-preserving HTML compression
(github.com/emmetify)
Cut your LLM processing costs by up to 90% by transforming verbose HTML into efficient Emmet notation, without losing structural integrity.
Cut your LLM processing costs by up to 90% by transforming verbose HTML into efficient Emmet notation, without losing structural integrity.
Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search
(arxiv.org)
Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine.
Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine.
NNCP: Lossless Data Compression with Neural Networks
(bellard.org)
NNCP is an experiment to build a practical lossless data compressor with neural networks.
NNCP is an experiment to build a practical lossless data compressor with neural networks.
Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data
(kevinslin.com)
On a rainy March day in 1538, Thomas Howard, the Duke of Norfolk, found himself confined within the cold stone walls of his grand estate. His fingers reluctantly penned a letter to sell his cherished lands to settle longstanding debts.
On a rainy March day in 1538, Thomas Howard, the Duke of Norfolk, found himself confined within the cold stone walls of his grand estate. His fingers reluctantly penned a letter to sell his cherished lands to settle longstanding debts.
Huffman Coding
(wikipedia.org)
In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression.
In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression.
The $5000 Compression Challenge (2001)
(patrickcraig.co.uk)
Mike Goldman makes another offer: I will attach a prize of $5,000 to anyone who successfully meets this challenge. First, the contestant will tell me HOW LONG of a data file to generate. Second, I will generate the data file, and send it to the contestant.
Mike Goldman makes another offer: I will attach a prize of $5,000 to anyone who successfully meets this challenge. First, the contestant will tell me HOW LONG of a data file to generate. Second, I will generate the data file, and send it to the contestant.
Binary vector embeddings are so cool
(emschwartz.me)
Vector embeddings by themselves are pretty neat. Binary quantized vector embeddings are extra impressive. In short, they can retain 95+% retrieval accuracy with 32x compression and ~25x retrieval speedup. Let's get into how this works and why it's so crazy.
Vector embeddings by themselves are pretty neat. Binary quantized vector embeddings are extra impressive. In short, they can retain 95+% retrieval accuracy with 32x compression and ~25x retrieval speedup. Let's get into how this works and why it's so crazy.
Consistently faster and smaller compressed bitmaps with Roaring (2016)
(arxiv.org)
Compressed bitmap indexes are used in databases and search engines.
Compressed bitmap indexes are used in databases and search engines.
Show HN: Vortex – a high-performance columnar file format
(github.com/spiraldb)
Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.
Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.
Parallel PNG Proposal (2021)
(github.com/DavidBuchanan314)
This is a proof-of-concept implementation of a parallel-decodable PNG format, based on ideas from https://github.com/brion/mtpng
This is a proof-of-concept implementation of a parallel-decodable PNG format, based on ideas from https://github.com/brion/mtpng
Large Text Compression Benchmark
(mattmahoney.net)
This competition ranks lossless data compression programs by the compressed size (including the size of the decompression program) of the first 109 bytes of the XML text dump of the English version of Wikipedia on Mar. 3, 2006.
This competition ranks lossless data compression programs by the compressed size (including the size of the decompression program) of the first 109 bytes of the XML text dump of the English version of Wikipedia on Mar. 3, 2006.
Compression Dictionary Transport
(ietf.org)
This document specifies a mechanism for dictionary-based compression in the Hypertext Transfer Protocol (HTTP). By utilizing this technique, clients and servers can reduce the size of transmitted data, leading to improved performance and reduced bandwidth consumption. This document extends existing HTTP compression methods and provides guidelines for the delivery and use of compression dictionaries within the HTTP protocol.
This document specifies a mechanism for dictionary-based compression in the Hypertext Transfer Protocol (HTTP). By utilizing this technique, clients and servers can reduce the size of transmitted data, leading to improved performance and reduced bandwidth consumption. This document extends existing HTTP compression methods and provides guidelines for the delivery and use of compression dictionaries within the HTTP protocol.
