Hacker News with Generative AI: Database Systems

Query Engines: Gatekeepers of the Parquet File Format (duckdb.org)
TL;DR: Mainstream query engines do not support reading newer Parquet encodings, forcing systems like DuckDB to default to writing older encodings, thereby sacrificing compression.
Database Systems, Data Formats, Performance, Compression, Query Engines
32 points by tosh 64 days ago | 1 comments
Scalable OLTP in the Cloud: What's the Big Deal? (blogspot.com)
This paper is from Pat Helland, the apostate philosopher of database systems, overall a superb person, and a good friend of mine. The paper appeared this week at CIDR'24. (Check out the program for other interesting papers). The motivating question behind this work is: "What are the asymptotic limits to scale for cloud OLTP (OnLine Transaction Processing) systems?" Pat says that the CIDR 2023 paper "Is Scalable OLTP in the Cloud a Solved Problem?" prompted this question.
Database Systems, Cloud Computing, Scalability
93 points by SchwKatze 66 days ago | 49 comments
Nulls: Revisiting null representation in modern columnar formats (dl.acm.org)
Nulls are common in real-world data sets, yet recent research on columnar formats and encodings rarely address Null representations.
Data Representation, Database Systems, Data Management, Columnar Formats, Null Values
54 points by tosh 154 days ago | 6 comments
Designing a Query Execution Engine (trychroma.com)
Distributed Chroma is a multi-tenant system. Query and Compactor nodes serve queries and build indexes for multiple tenants. By leveraging multi-tenancy we can maximize utilization of nodes in our system, resulting in lower costs for our users. However, building with multi-tenancy in mind presents the challenge of how to optimally structure, dispatch, and schedule work such that resources are fairly used across all tenants.
Database Systems, Query Optimization, Software Architecture, Distributed Systems
41 points by jeffchuber 154 days ago | 0 comments
A Deep Dive into German Strings (cedardb.com)
“Strings are Everywhere”! At least according to a 2018 DBTest Paper from the Hyper team at Tableau. In fact, strings make up nearly half of the data processed at Tableau. This high prevalence undoubtedly applies to many other companies as well, as the paper’s dataset consists of data analyzed by Tableau’s users. The string-heavy nature of the data makes string processing one of the most important tasks of a database system.
Database Systems, String Processing, Data Analysis, German
10 points by tosh 162 days ago | 6 comments
The Untold Story of SQLite (2021) (corecursive.com)
Database Systems, History, Software
80 points by usernamesdf 230 days ago | 25 comments
Umbra: A Disk-Based System with In-Memory Performance [pdf] (cidrdb.org)
Database Systems, Performance, Disk-Based Systems
55 points by itunpredictable 336 days ago | 24 comments