Hacker News with Generative AI: Data Formats

Reverse Engineering Apple's typedstream Format (chrissardegna.com)
imessage-exporter’s goal is to provide the most comprehensive representation of iMessage data available. Message data is stored in a legacy format that appears to be a stream that represents objects.

Reverse Engineering, Apple, Data Formats, iMessage

127 points by css 18 days ago | 25 comments

Sparrow, a modern C++ implementation of the Apache Arrow columnar format (medium.com)
We are thrilled to introduce Sparrow, a new library designed to simplify the integration of Apache Arrow’s columnar format into C++ applications.

C++, Apache Arrow, Data Formats, Libraries, Performance

116 points by SylvainCorlay 21 days ago | 25 comments

Query Engines: Gatekeepers of the Parquet File Format (duckdb.org)
TL;DR: Mainstream query engines do not support reading newer Parquet encodings, forcing systems like DuckDB to default to writing older encodings, thereby sacrificing compression.

Database Systems, Data Formats, Performance, Compression, Query Engines

32 points by tosh 23 days ago | 1 comments

Preserves: An Expressive Data Language (preserves.dev)
This repository contains a definition and various implementations of Preserves, a data model with associated serialization formats in many ways comparable to JSON, XML, S-expressions, CBOR, ASN.1 BER, and so on.

Data Formats, Programming Languages, Software, Data Models, Serialization

78 points by mpweiher 23 days ago | 45 comments

Cramming Scrapscript into Msgpack (taylor.town)
msgpack is a lovely little serialization format. As a JSON replacement, it saves bandwidth while preserving native language features (e.g. tuples, records, objects, dates).

Serialization, Programming Languages, Data Formats

12 points by surprisetalk 57 days ago | 1 comments

Nobody gets fired for picking JSON, but maybe they should? (mcyoung.xyz)
JSON is extremely popular but deeply flawed. This article discusses the details of JSON’s design, how it’s used (and misused), and how seemingly helpful “human readability” features cause headaches instead. Crucially, you rarely find JSON-based tools (except dedicated tools like jq) that can safely handle arbitrary JSON documents without a schema—common corner cases can lead to data corruption!

Programming, Data Formats, JSON

44 points by maximilianburke 73 days ago | 57 comments

Nobody Gets Fired for Picking JSON, but Maybe They Should? (mcyoung.xyz)
JSON is extremely popular but deeply flawed. This article discusses the details of JSON’s design, how it’s used (and misused), and how seemingly helpful “human readability” features cause headaches instead. Crucially, you rarely find JSON-based tools (except dedicated tools like jq) that can safely handle arbitrary JSON documents without a schema—common corner cases can lead to data corruption!

Data Formats, JSON, Software Development, Programming Languages

10 points by usdogu 73 days ago | 1 comments

Pg_parquet – Postgres to Parquet Interoperability (i-programmer.info)

Databases, Data Formats, Postgres, Parquet

4 points by plaur782 85 days ago | 0 comments

ASCII Delimited Text – Not CSV or Tab Delimited Text (wordpress.com)
Unfortunately a quick google search on “ASCII Delimited Text” shows that IBM and Oracle failed to read the ASCII specification and both define ASCII Delimited Text as a CSV format. ASCII Delimited Text should use the record separators defined as ASCII 28-31.

Data Formats, Programming, Text Processing, Technical Standards

114 points by ejstronge 103 days ago | 117 comments

New better alterative to XML, JSON and YAML (xenondata.org)
Xenon is the best way to represent information: Terse. Readable multiple line indented text. Native support for arrays Native support for a graph structure, elements may have multiple parents. Native support for a types used in serialization. Unambiguous choice of data structure. Efficient to write by hand. Can be implemented to be blazingly fast or using a mode-less tokenizer/parser. The xenon document is named.

Data Formats, Programming Languages, Software Development

24 points by GeneThomas 110 days ago | 142 comments

JSON Patch (zuplo.com)
JSON Patch is a standardized format defined in RFC 6902 for describing how to modify a JSON document.

JSON, API, Web Development, Data Formats

300 points by DataOverload 127 days ago | 160 comments

JSON is usually the least bad option for machine-readable output formats (utoronto.ca)

JSON, Data Formats, Programming, Software Engineering

20 points by luu 178 days ago | 14 comments

CSVs Are Kinda Bad. DSVs Are Kinda Good (matthodges.com)

Data Formats, Programming, Software Development

107 points by hieronymusN 191 days ago | 127 comments

Recommended Formats Statement (loc.gov)