Hacker News with Generative AI: Data Management

Palantir suggests 'common operating system' for UK govt data (theregister.com)
In a witness statement to the UK COVID-19 Inquiry [PDF], an ongoing independent public inquiry into the nation's response to the pandemic (in which around 208,000 people died), Louis Mosley, executive veep of Palantir Technologies UK, said the government should invest in a "common operating system" for its data, encompassing departments such as the Department for Work and Pensions and local authorities.
Government, Data Management, UK, Technology, Pandemic
49 points by rntn 8 days ago | 76 comments
Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s) (ycombinator.com)
I’ve spent the past year and a half constructing a Retrieval Augmented Generation (RAG) chatbot for a Fortune 500 manufacturing company, integrating over 50 million records across a dozen databases.
Chatbots, Fortune 500, Data Management
19 points by tylersuard 13 days ago | 15 comments
Time-Series vs. Streaming Databases: Key Differences and Use Cases (risingwave.com)
Databases, Time Series, Streaming Data, Data Management
23 points by Sheldon_fun 16 days ago | 1 comments
Ask HN: How do you manage and version control small structured data? (ycombinator.com)
So I work in a heavily regulated field and often come across the need to document all kinds of semi-structured data like requirements, risks, test-cases, etc.
Data Management, Version Control, Software Engineering, Regulatory Compliance, Ask HN
5 points by Igin 22 days ago | 6 comments
New Zealand's $16B health dept managed finances with single Excel spreadsheet (theregister.com)
The body that runs New Zealand’s public health system uses a single Excel spreadsheet as the primary source of data to consolidate and manage its finances, which aren’t in great shape perhaps due to the sheet’s shortcomings.
New Zealand, Healthcare, Finance, Data Management, Technology
164 points by indy 22 days ago | 162 comments
$16B health dept managed finances with single Excel sheet. It hasn't gone well (theregister.com)
The body that runs New Zealand’s public health system uses a single Excel spreadsheet as the primary source of data to consolidate and manage its finances, which aren’t in great shape perhaps due to the sheet’s shortcomings.
New Zealand, Public Health, Finance, Data Management, Excel
7 points by rntn 23 days ago | 1 comments
We built a modern data stack from scratch and reduced our bill by 70% (jchandra.com)
Building and managing a data platform that is both scalable and cost-effective is a challenge many organizations face. We managed an extensive data lake with a lean data team and reduced our Infra Cost by 70%.
Data Management, Cost Reduction, Cloud Infrastructure
83 points by jchandra 23 days ago | 59 comments
Multiply Went from Datomic to XTDB to Rama (redplanetlabs.com)
"With databases, the conversation always started with ‘what are we able to do?’. I rarely find myself asking what Rama is able to support, and rather ‘how?’. The requirements of the application dictate how we utilise the platform, not the other way around. Rama as a tool allows us to think product first, while still delivering highly optimised and scalable features for specific use cases, something that would not have been possible without a much larger team.”
Databases, Software, Technology, Data Management, Company Specific
16 points by winkywooster 28 days ago | 0 comments
Understanding Smallpond and 3FS (definite.app)
I didn't have "DeepSeek releases distributed DuckDB" on my 2025 bingo card.
Databases, Data Management, Technology, Open Source
262 points by mritchie712 31 days ago | 47 comments
Segment for LLM Traces? Seeking Feedback on an Open Source LLM Log Router (ycombinator.com)
I’m considering starting a new open source project and wanted to see if anyone else thinks the idea could be useful. The concept is simple: an open source LLM log router that works like Segment—but specifically for LLM logs.
Open Source, Generative AI, Data Management
11 points by patethegreat 39 days ago | 2 comments
Augmenting NLQ with language knowledge bases like web search for ChatGPT (hyperarc.com)
The rise of warehouses like Snowflake and CDPs like Segment broke down data silos, joining your CRM to your marketing automation, support tickets, and more. This connected view of your business enabled more accurate and actionable insights in traditional BI.
Artificial Intelligence, Data Management, Business Intelligence
9 points by zzsf 39 days ago | 0 comments
Where are all the rewrite rules? (philipzucker.com)
I think a thing that’d be nice is to have a databank of rewrite rules. Here’s some of the ones I know about.
Software Engineering, Programming Languages, Data Management
59 points by todsacerdoti 42 days ago | 29 comments
Modern CSV: Multi-Platform CSV File Editor and Viewer (moderncsv.com)
Modern CSV is a powerful CSV file editor/viewer application for Windows, Mac, and Linux. Professionals at all levels of technical proficiency use it to analyze data, check files for uploading to databases, modify configuration files, maintain customer lists, and more. We designed it to compensate for the deficiencies of spreadsheet programs in handling CSV/TSV/DSV/etc. files. We strive to create a user experience our customers describe as “blissful”. 
Data Analysis, Software, Cross-Platform, Data Management, File Editing
5 points by disqard 43 days ago | 1 comments
Will AI Agents Revolutionize How We Query and Use Data? (ycombinator.com)
Snowflake just announced AI Data Agents in Cortex, a new way to automate and streamline data workflows with AI.
Artificial Intelligence, Data Management, Automation, Cloud Computing, Snowflake
8 points by charliessnow 48 days ago | 7 comments
A new approach to data handling between systems/for AI (github.com/dev-formata-io)
Stof is efficient, governable, and accessible data that is much simpler to use, offering fine-grained control and sandboxed manipulation between computer systems without the need for additional application code, servers, and dependencies.
Data Management, AI, Software, Open Source
23 points by AmeliaWampler 48 days ago | 6 comments
Apache Iceberg now supports geospatial data types natively (wherobots.com)
Geospatial solutions were thought of as “special”, because what modernized the data ecosystem of today, left geospatial data mostly behind. This changes today. Thanks to the efforts of the Apache Iceberg and Parquet communities, we are excited to share that both Iceberg and Parquet now support geometry and geography (collectively the GEO) data types.
Geospatial Data, Apache Iceberg, Open Source, Data Management, Parquet
104 points by Mosarwat 49 days ago | 21 comments
Bulk inserts on ClickHouse: How to avoid overstuffing your instance (runportcullis.co)
As we hit the midway point of the second month in 2025, a lot of you might be starting to really dig in on new data initiatives and planning key infrastructure changes to your company’s data stack.
Data Management, Databases, ClickHouse
42 points by mooreds 50 days ago | 12 comments
PostgreSQL Best Practices (speakdatascience.com)
PostgreSQL (Postgres) is one of the most powerful and popular relational database management systems available today. Whether you’re a database administrator, developer, or DevOps engineer, following best practices ensures optimal performance, security, and maintainability of your database systems.
Databases, PostgreSQL, Best Practices, Data Management
72 points by lemonwaterlime 51 days ago | 45 comments
Over 700M events/second: How Cloudflare makes sense of too much data (cloudflare.com)
Cloudflare's network provides an enormous array of services to our customers. We collect and deliver associated data to customers in the form of event logs and aggregated analytics. As of December 2024, our data pipeline is ingesting up to 706M events per second generated by Cloudflare's services, and that represents 100x growth since our 2018 data pipeline blog post.
Cloud Computing, Data Management, Performance, Scale, Cloudflare
13 points by caust1c 56 days ago | 1 comments
Datawave: Open source data fusion across structured and unstructured datasets (code.nsa.gov)
DataWave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data.
Open Source, Data Management, Java, Big Data
5 points by teleforce 66 days ago | 0 comments
Apple Passwords is hostile to backups (lapcatsoftware.com)
In my view, a useful backup system must be (1) chronological, (2) granular, and (3) redundant. A chronological backup system includes multiple historical snapshots of your data, allowing you to recover not only the latest version of your data but also past data that has been deleted or edited. A granular backup system allows you to selectively recover specific fragments of data from your backup without disturbing, deleting, or corrupting the rest of your current data.
Apple, Security, Backups, Data Management
6 points by ksec 68 days ago | 1 comments
Data Branching for Batch Job Systems (isaacjordan.me)
Data is being increasingly treated like code has been treated for decades. For many use-cases it isn't enough to know "What is the current value?" but also "What was the value previously?", "Who last changed the value?", and "Why did they change the value?"
Data Management, Software Development, Programming, Software Engineering, Databases
44 points by thunderbong 70 days ago | 6 comments
Storage is cheap, but not thinking about logging is expensive (counting-stuff.com)
The bad habits of data over-collection run deep.
Data Storage, Logging, Best Practices, Data Management
11 points by todsacerdoti 71 days ago | 0 comments
InfluxDB 3 Open Source Now in Public Alpha Under MIT/Apache 2 License (influxdata.com)
Today we’re excited to announce the alpha release of InfluxDB 3 Core (download), the new open source product in the InfluxDB 3 product line along with InfluxDB 3 Enterprise (download), a commercial version that builds on Core’s foundation.
Open Source, Databases, Software, Data Management
31 points by otoolep 77 days ago | 8 comments
ZFS 2.3 released with ZFS raidz expansion (github.com/openzfs)
We are excited to announce the release of OpenZFS 2.3.0.
New Releases, Software, Storage, Data Management, Open Source
383 points by scrp 78 days ago | 320 comments
Gmvault: Backup and restore your Gmail account (ycombinator.com)
Gmvault: Backup and restore your Gmail account
Gmail, Backup, Security, Data Management
4 points by Openai2 82 days ago | 1 comments
Parquet and ORC's many shortfalls for machine learning, and what to do about it? (starburst.io)
At the turn of the century (around a quarter of a decade ago), over 99% of the data management industry used row-oriented storage to store data for all workloads involving structured data — including transactional and analytical workloads.
Machine Learning, Data Storage, Data Management, Big Data
6 points by abadid 84 days ago | 1 comments
Using watermarks to coordinate change data capture in Postgres (sequinstream.com)
In change data capture, consistency is paramount. A single missing or duplicate message can cascade into time-consuming bugs and erode trust in your entire system. The moment you find a record missing in the destination, you have to wonder: is this the only one? How many others are there?
Databases, Data Management, PostgreSQL
102 points by misonic 89 days ago | 0 comments
Managing Data Corruption in the Cloud (mongodb.com)
MongoDB has been named a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMSs) for the third consecutive year.
Data Management, Cloud Computing, Databases
14 points by LittleCat38 100 days ago | 1 comments
Tell HN: Deduplicating a 10.4 TiB game preservation archive (WIP) (ycombinator.com)
Data Management, Game Preservation, Optimization
30 points by DrFrugal 105 days ago | 53 comments