Hacker News with Generative AI: Data Management

Multiply Went from Datomic to XTDB to Rama (redplanetlabs.com)
"With databases, the conversation always started with ‘what are we able to do?’. I rarely find myself asking what Rama is able to support, and rather ‘how?’. The requirements of the application dictate how we utilise the platform, not the other way around. Rama as a tool allows us to think product first, while still delivering highly optimised and scalable features for specific use cases, something that would not have been possible without a much larger team.”

Databases, Software, Technology, Data Management, Company Specific

15 points by winkywooster 2 days ago | 0 comments

Understanding Smallpond and 3FS (definite.app)
I didn't have "DeepSeek releases distributed DuckDB" on my 2025 bingo card.

Databases, Data Management, Technology, Open Source

261 points by mritchie712 4 days ago | 47 comments

Segment for LLM Traces? Seeking Feedback on an Open Source LLM Log Router (ycombinator.com)
I’m considering starting a new open source project and wanted to see if anyone else thinks the idea could be useful. The concept is simple: an open source LLM log router that works like Segment—but specifically for LLM logs.

Open Source, Generative AI, Data Management

11 points by patethegreat 13 days ago | 2 comments

Augmenting NLQ with language knowledge bases like web search for ChatGPT (hyperarc.com)
The rise of warehouses like Snowflake and CDPs like Segment broke down data silos, joining your CRM to your marketing automation, support tickets, and more. This connected view of your business enabled more accurate and actionable insights in traditional BI.

Artificial Intelligence, Data Management, Business Intelligence

9 points by zzsf 13 days ago | 0 comments

Where are all the rewrite rules? (philipzucker.com)
I think a thing that’d be nice is to have a databank of rewrite rules. Here’s some of the ones I know about.

Software Engineering, Programming Languages, Data Management

59 points by todsacerdoti 16 days ago | 29 comments

Modern CSV: Multi-Platform CSV File Editor and Viewer (moderncsv.com)
Modern CSV is a powerful CSV file editor/viewer application for Windows, Mac, and Linux. Professionals at all levels of technical proficiency use it to analyze data, check files for uploading to databases, modify configuration files, maintain customer lists, and more. We designed it to compensate for the deficiencies of spreadsheet programs in handling CSV/TSV/DSV/etc. files. We strive to create a user experience our customers describe as “blissful”.

Data Analysis, Software, Cross-Platform, Data Management, File Editing

5 points by disqard 17 days ago | 1 comments

Will AI Agents Revolutionize How We Query and Use Data? (ycombinator.com)
Snowflake just announced AI Data Agents in Cortex, a new way to automate and streamline data workflows with AI.

Artificial Intelligence, Data Management, Automation, Cloud Computing, Snowflake

8 points by charliessnow 21 days ago | 7 comments

A new approach to data handling between systems/for AI (github.com/dev-formata-io)
Stof is efficient, governable, and accessible data that is much simpler to use, offering fine-grained control and sandboxed manipulation between computer systems without the need for additional application code, servers, and dependencies.

Data Management, AI, Software, Open Source

23 points by AmeliaWampler 22 days ago | 6 comments

Apache Iceberg now supports geospatial data types natively (wherobots.com)
Geospatial solutions were thought of as “special”, because what modernized the data ecosystem of today, left geospatial data mostly behind. This changes today. Thanks to the efforts of the Apache Iceberg and Parquet communities, we are excited to share that both Iceberg and Parquet now support geometry and geography (collectively the GEO) data types.

Geospatial Data, Apache Iceberg, Open Source, Data Management, Parquet

104 points by Mosarwat 23 days ago | 21 comments

Bulk inserts on ClickHouse: How to avoid overstuffing your instance (runportcullis.co)
As we hit the midway point of the second month in 2025, a lot of you might be starting to really dig in on new data initiatives and planning key infrastructure changes to your company’s data stack.

Data Management, Databases, ClickHouse

42 points by mooreds 24 days ago | 12 comments

PostgreSQL Best Practices (speakdatascience.com)
PostgreSQL (Postgres) is one of the most powerful and popular relational database management systems available today. Whether you’re a database administrator, developer, or DevOps engineer, following best practices ensures optimal performance, security, and maintainability of your database systems.

Databases, PostgreSQL, Best Practices, Data Management

72 points by lemonwaterlime 25 days ago | 45 comments

Over 700M events/second: How Cloudflare makes sense of too much data (cloudflare.com)
Cloudflare's network provides an enormous array of services to our customers. We collect and deliver associated data to customers in the form of event logs and aggregated analytics. As of December 2024, our data pipeline is ingesting up to 706M events per second generated by Cloudflare's services, and that represents 100x growth since our 2018 data pipeline blog post.

Cloud Computing, Data Management, Performance, Scale, Cloudflare

13 points by caust1c 30 days ago | 1 comments

Datawave: Open source data fusion across structured and unstructured datasets (code.nsa.gov)
DataWave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data.

Open Source, Data Management, Java, Big Data

5 points by teleforce 40 days ago | 0 comments

Apple Passwords is hostile to backups (lapcatsoftware.com)
In my view, a useful backup system must be (1) chronological, (2) granular, and (3) redundant. A chronological backup system includes multiple historical snapshots of your data, allowing you to recover not only the latest version of your data but also past data that has been deleted or edited. A granular backup system allows you to selectively recover specific fragments of data from your backup without disturbing, deleting, or corrupting the rest of your current data.

Apple, Security, Backups, Data Management

6 points by ksec 41 days ago | 1 comments

Data Branching for Batch Job Systems (isaacjordan.me)
Data is being increasingly treated like code has been treated for decades. For many use-cases it isn't enough to know "What is the current value?" but also "What was the value previously?", "Who last changed the value?", and "Why did they change the value?"

Data Management, Software Development, Programming, Software Engineering, Databases

44 points by thunderbong 44 days ago | 6 comments

Storage is cheap, but not thinking about logging is expensive (counting-stuff.com)
The bad habits of data over-collection run deep.

Data Storage, Logging, Best Practices, Data Management

11 points by todsacerdoti 44 days ago | 0 comments

InfluxDB 3 Open Source Now in Public Alpha Under MIT/Apache 2 License (influxdata.com)
Today we’re excited to announce the alpha release of InfluxDB 3 Core (download), the new open source product in the InfluxDB 3 product line along with InfluxDB 3 Enterprise (download), a commercial version that builds on Core’s foundation.

Open Source, Databases, Software, Data Management

31 points by otoolep 51 days ago | 8 comments

ZFS 2.3 released with ZFS raidz expansion (github.com/openzfs)
We are excited to announce the release of OpenZFS 2.3.0.

New Releases, Software, Storage, Data Management, Open Source

383 points by scrp 52 days ago | 320 comments

Gmvault: Backup and restore your Gmail account (ycombinator.com)
Gmvault: Backup and restore your Gmail account

Gmail, Backup, Security, Data Management

4 points by Openai2 56 days ago | 1 comments

Parquet and ORC's many shortfalls for machine learning, and what to do about it? (starburst.io)
At the turn of the century (around a quarter of a decade ago), over 99% of the data management industry used row-oriented storage to store data for all workloads involving structured data — including transactional and analytical workloads.

Machine Learning, Data Storage, Data Management, Big Data

6 points by abadid 58 days ago | 1 comments

Using watermarks to coordinate change data capture in Postgres (sequinstream.com)
In change data capture, consistency is paramount. A single missing or duplicate message can cascade into time-consuming bugs and erode trust in your entire system. The moment you find a record missing in the destination, you have to wonder: is this the only one? How many others are there?

Databases, Data Management, PostgreSQL

102 points by misonic 63 days ago | 0 comments

Managing Data Corruption in the Cloud (mongodb.com)
MongoDB has been named a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMSs) for the third consecutive year.

Data Management, Cloud Computing, Databases

14 points by LittleCat38 74 days ago | 1 comments

Tell HN: Deduplicating a 10.4 TiB game preservation archive (WIP) (ycombinator.com)

Data Management, Game Preservation, Optimization

30 points by DrFrugal 79 days ago | 53 comments

Is stuff online worth saving? (rubenerd.com)
Related to my post about self-hosted bookmarking tools (thanks to everyone for the suggestions!), I’ve been exporting my bookmarks from various sites so I can eventually aggregate them into once place. It’s a lot of work, and it might mostly be for naught.

Web Development, Data Management, Personal Productivity

158 points by Brajeshwar 80 days ago | 165 comments

Ask questions of SQLite databases and CSV/JSON files in your terminal (simonwillison.net)
I built a new plugin for my sqlite-utils CLI tool that lets you ask human-language questions directly of SQLite databases and CSV/JSON files on your computer.

Databases, Data Analysis, Data Management, Programming

118 points by redbell 102 days ago | 49 comments

DELETEs Are Difficult (boringsql.com)
Your database is ticking along nicely - until a simple DELETE brings it to its knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT operations, we often overlook the hidden complexities of DELETE. Yet, removing unnecessary data is just as critical. Outdated or irrelevant data can bloat your database, degrade performance, and make maintenance a nightmare. Worse, retaining some types of data without valid justification might even lead to compliance issues.

Databases, Data Management, Performance Optimization

139 points by radimm 102 days ago | 115 comments

NASA SC24: NASA-GPT: Searching the Entire NASA Technical Reports Server Using AI (nasa.gov)
Researchers at NASA’s Ames Research Center in Silicon Valley are using artificial intelligence (AI) tools to create a data-sharing resource for the agency’s scientific and engineering staff.

Artificial Intelligence, NASA, Science, Research, Data Management

13 points by rbanffy 103 days ago | 1 comments

Snowflake opens chat-driven access to enterprise and third-party data (theregister.com)
Snowflake is set to preview a new platform it claims will help organizations build chatbots that can serve up data from its own analytics systems and those external to the cloud data platform vendor.

Data Management, Cloud Computing, Artificial Intelligence, Chatbots

6 points by rntn 113 days ago | 0 comments

Netflix's Distributed Counter Abstraction (netflixtechblog.com)
In our previous blog post, we introduced Netflix’s TimeSeries Abstraction, a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction. This counting service, built on top of the TimeSeries Abstraction, enables distributed counting at scale while maintaining similar low latency performance. As with all our abstractions, we use our Data Gateway Control Plane to shard, configure, and deploy this service globally.

Netflix, Distributed Systems, Data Management

133 points by benocodes 113 days ago | 71 comments

Utilizing HubSpot's APIs to Break Down Data Silos (accelant.com)
I’ve seen this movie countless times: Business Has Tons of Useful Data That’s Scattered Across Systems and Isn’t Being Used for Much (or anything).

Data Management, APIs, Business Intelligence, Marketing Automation

9 points by mooreds 114 days ago | 3 comments