Hacker News with Generative AI: Data Formats

A love letter to the CSV format (github.com/medialab)
Every month or so, a new blog article declaring the near demise of CSV in favor of some "obviously superior" format (parquet, newline-delimited JSON, MessagePack records etc.) find its ways to the reader's eyes. Sadly those articles often offer a very narrow and biased comparison and often fail to understand what makes CSV a seemingly unkillable staple of data serialization.

Data Formats, CSV, Programming, Comparisons

708 points by Yomguithereal 7 days ago | 688 comments

Begrudgingly Choosing CBOR over MessagePack (taylor.town)
In my pursuit of a sharable programming language, I need a suitable serialization format that's (1) performant to send/store and (2) easy to understand/implement.

Programming Languages, Data Formats, Performance

55 points by jjgreen 31 days ago | 78 comments

Reverse Engineering Apple's typedstream Format (chrissardegna.com)
imessage-exporter’s goal is to provide the most comprehensive representation of iMessage data available. Message data is stored in a legacy format that appears to be a stream that represents objects.

Reverse Engineering, Apple, Data Formats, iMessage

127 points by css 58 days ago | 25 comments

Sparrow, a modern C++ implementation of the Apache Arrow columnar format (medium.com)
We are thrilled to introduce Sparrow, a new library designed to simplify the integration of Apache Arrow’s columnar format into C++ applications.

C++, Apache Arrow, Data Formats, Libraries, Performance

116 points by SylvainCorlay 60 days ago | 25 comments

Query Engines: Gatekeepers of the Parquet File Format (duckdb.org)
TL;DR: Mainstream query engines do not support reading newer Parquet encodings, forcing systems like DuckDB to default to writing older encodings, thereby sacrificing compression.

Database Systems, Data Formats, Performance, Compression, Query Engines

32 points by tosh 63 days ago | 1 comments

Preserves: An Expressive Data Language (preserves.dev)
This repository contains a definition and various implementations of Preserves, a data model with associated serialization formats in many ways comparable to JSON, XML, S-expressions, CBOR, ASN.1 BER, and so on.

Data Formats, Programming Languages, Software, Data Models, Serialization

78 points by mpweiher 63 days ago | 45 comments

Cramming Scrapscript into Msgpack (taylor.town)
msgpack is a lovely little serialization format. As a JSON replacement, it saves bandwidth while preserving native language features (e.g. tuples, records, objects, dates).

Serialization, Programming Languages, Data Formats

12 points by surprisetalk 97 days ago | 1 comments

Nobody gets fired for picking JSON, but maybe they should? (mcyoung.xyz)
JSON is extremely popular but deeply flawed. This article discusses the details of JSON’s design, how it’s used (and misused), and how seemingly helpful “human readability” features cause headaches instead. Crucially, you rarely find JSON-based tools (except dedicated tools like jq) that can safely handle arbitrary JSON documents without a schema—common corner cases can lead to data corruption!

Programming, Data Formats, JSON

44 points by maximilianburke 112 days ago | 57 comments

Nobody Gets Fired for Picking JSON, but Maybe They Should? (mcyoung.xyz)
JSON is extremely popular but deeply flawed. This article discusses the details of JSON’s design, how it’s used (and misused), and how seemingly helpful “human readability” features cause headaches instead. Crucially, you rarely find JSON-based tools (except dedicated tools like jq) that can safely handle arbitrary JSON documents without a schema—common corner cases can lead to data corruption!

Data Formats, JSON, Software Development, Programming Languages

10 points by usdogu 112 days ago | 1 comments

Pg_parquet – Postgres to Parquet Interoperability (i-programmer.info)

Databases, Data Formats, Postgres, Parquet

4 points by plaur782 124 days ago | 0 comments

ASCII Delimited Text – Not CSV or Tab Delimited Text (wordpress.com)
Unfortunately a quick google search on “ASCII Delimited Text” shows that IBM and Oracle failed to read the ASCII specification and both define ASCII Delimited Text as a CSV format. ASCII Delimited Text should use the record separators defined as ASCII 28-31.

Data Formats, Programming, Text Processing, Technical Standards

114 points by ejstronge 143 days ago | 117 comments

New better alterative to XML, JSON and YAML (xenondata.org)
Xenon is the best way to represent information: Terse. Readable multiple line indented text. Native support for arrays Native support for a graph structure, elements may have multiple parents. Native support for a types used in serialization. Unambiguous choice of data structure. Efficient to write by hand. Can be implemented to be blazingly fast or using a mode-less tokenizer/parser. The xenon document is named.

Data Formats, Programming Languages, Software Development

24 points by GeneThomas 149 days ago | 142 comments

JSON Patch (zuplo.com)
JSON Patch is a standardized format defined in RFC 6902 for describing how to modify a JSON document.

JSON, API, Web Development, Data Formats

300 points by DataOverload 166 days ago | 160 comments

JSON is usually the least bad option for machine-readable output formats (utoronto.ca)

JSON, Data Formats, Programming, Software Engineering

20 points by luu 217 days ago | 14 comments

CSVs Are Kinda Bad. DSVs Are Kinda Good (matthodges.com)

Data Formats, Programming, Software Development

107 points by hieronymusN 231 days ago | 127 comments

Recommended Formats Statement (loc.gov)