Hacker News with Generative AI: Data Management

Modern CSV: Multi-Platform CSV File Editor and Viewer (moderncsv.com)
Modern CSV is a powerful CSV file editor/viewer application for Windows, Mac, and Linux. Professionals at all levels of technical proficiency use it to analyze data, check files for uploading to databases, modify configuration files, maintain customer lists, and more. We designed it to compensate for the deficiencies of spreadsheet programs in handling CSV/TSV/DSV/etc. files. We strive to create a user experience our customers describe as “blissful”.

Data Analysis, Software, Cross-Platform, Data Management, File Editing

5 points by disqard 2 days ago | 1 comments

Will AI Agents Revolutionize How We Query and Use Data? (ycombinator.com)
Snowflake just announced AI Data Agents in Cortex, a new way to automate and streamline data workflows with AI.

Artificial Intelligence, Data Management, Automation, Cloud Computing, Snowflake

8 points by charliessnow 7 days ago | 7 comments

A new approach to data handling between systems/for AI (github.com/dev-formata-io)
Stof is efficient, governable, and accessible data that is much simpler to use, offering fine-grained control and sandboxed manipulation between computer systems without the need for additional application code, servers, and dependencies.

Data Management, AI, Software, Open Source

22 points by AmeliaWampler 8 days ago | 6 comments

Apache Iceberg now supports geospatial data types natively (wherobots.com)
Geospatial solutions were thought of as “special”, because what modernized the data ecosystem of today, left geospatial data mostly behind. This changes today. Thanks to the efforts of the Apache Iceberg and Parquet communities, we are excited to share that both Iceberg and Parquet now support geometry and geography (collectively the GEO) data types.

Geospatial Data, Apache Iceberg, Open Source, Data Management, Parquet

103 points by Mosarwat 9 days ago | 21 comments

Bulk inserts on ClickHouse: How to avoid overstuffing your instance (runportcullis.co)
As we hit the midway point of the second month in 2025, a lot of you might be starting to really dig in on new data initiatives and planning key infrastructure changes to your company’s data stack.

Data Management, Databases, ClickHouse

42 points by mooreds 9 days ago | 12 comments

PostgreSQL Best Practices (speakdatascience.com)
PostgreSQL (Postgres) is one of the most powerful and popular relational database management systems available today. Whether you’re a database administrator, developer, or DevOps engineer, following best practices ensures optimal performance, security, and maintainability of your database systems.

Databases, PostgreSQL, Best Practices, Data Management

72 points by lemonwaterlime 11 days ago | 45 comments

Over 700M events/second: How Cloudflare makes sense of too much data (cloudflare.com)
Cloudflare's network provides an enormous array of services to our customers. We collect and deliver associated data to customers in the form of event logs and aggregated analytics. As of December 2024, our data pipeline is ingesting up to 706M events per second generated by Cloudflare's services, and that represents 100x growth since our 2018 data pipeline blog post.

Cloud Computing, Data Management, Performance, Scale, Cloudflare

13 points by caust1c 16 days ago | 1 comments

Datawave: Open source data fusion across structured and unstructured datasets (code.nsa.gov)
DataWave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data.

Open Source, Data Management, Java, Big Data

5 points by teleforce 26 days ago | 0 comments

Apple Passwords is hostile to backups (lapcatsoftware.com)
In my view, a useful backup system must be (1) chronological, (2) granular, and (3) redundant. A chronological backup system includes multiple historical snapshots of your data, allowing you to recover not only the latest version of your data but also past data that has been deleted or edited. A granular backup system allows you to selectively recover specific fragments of data from your backup without disturbing, deleting, or corrupting the rest of your current data.

Apple, Security, Backups, Data Management

6 points by ksec 27 days ago | 1 comments

Data Branching for Batch Job Systems (isaacjordan.me)
Data is being increasingly treated like code has been treated for decades. For many use-cases it isn't enough to know "What is the current value?" but also "What was the value previously?", "Who last changed the value?", and "Why did they change the value?"

Data Management, Software Development, Programming, Software Engineering, Databases

44 points by thunderbong 29 days ago | 6 comments

Storage is cheap, but not thinking about logging is expensive (counting-stuff.com)
The bad habits of data over-collection run deep.

Data Storage, Logging, Best Practices, Data Management

11 points by todsacerdoti 30 days ago | 0 comments

InfluxDB 3 Open Source Now in Public Alpha Under MIT/Apache 2 License (influxdata.com)
Today we’re excited to announce the alpha release of InfluxDB 3 Core (download), the new open source product in the InfluxDB 3 product line along with InfluxDB 3 Enterprise (download), a commercial version that builds on Core’s foundation.

Open Source, Databases, Software, Data Management

31 points by otoolep 37 days ago | 8 comments

ZFS 2.3 released with ZFS raidz expansion (github.com/openzfs)
We are excited to announce the release of OpenZFS 2.3.0.

New Releases, Software, Storage, Data Management, Open Source

383 points by scrp 38 days ago | 320 comments

Gmvault: Backup and restore your Gmail account (ycombinator.com)
Gmvault: Backup and restore your Gmail account

Gmail, Backup, Security, Data Management

4 points by Openai2 42 days ago | 1 comments

Parquet and ORC's many shortfalls for machine learning, and what to do about it? (starburst.io)
At the turn of the century (around a quarter of a decade ago), over 99% of the data management industry used row-oriented storage to store data for all workloads involving structured data — including transactional and analytical workloads.

Machine Learning, Data Storage, Data Management, Big Data

6 points by abadid 44 days ago | 1 comments

Using watermarks to coordinate change data capture in Postgres (sequinstream.com)
In change data capture, consistency is paramount. A single missing or duplicate message can cascade into time-consuming bugs and erode trust in your entire system. The moment you find a record missing in the destination, you have to wonder: is this the only one? How many others are there?

Databases, Data Management, PostgreSQL

102 points by misonic 49 days ago | 0 comments

Managing Data Corruption in the Cloud (mongodb.com)
MongoDB has been named a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMSs) for the third consecutive year.

Data Management, Cloud Computing, Databases

14 points by LittleCat38 59 days ago | 1 comments

Tell HN: Deduplicating a 10.4 TiB game preservation archive (WIP) (ycombinator.com)

Data Management, Game Preservation, Optimization

30 points by DrFrugal 65 days ago | 53 comments

Is stuff online worth saving? (rubenerd.com)
Related to my post about self-hosted bookmarking tools (thanks to everyone for the suggestions!), I’ve been exporting my bookmarks from various sites so I can eventually aggregate them into once place. It’s a lot of work, and it might mostly be for naught.

Web Development, Data Management, Personal Productivity

158 points by Brajeshwar 65 days ago | 165 comments

Ask questions of SQLite databases and CSV/JSON files in your terminal (simonwillison.net)
I built a new plugin for my sqlite-utils CLI tool that lets you ask human-language questions directly of SQLite databases and CSV/JSON files on your computer.

Databases, Data Analysis, Data Management, Programming

118 points by redbell 87 days ago | 49 comments

DELETEs Are Difficult (boringsql.com)
Your database is ticking along nicely - until a simple DELETE brings it to its knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT operations, we often overlook the hidden complexities of DELETE. Yet, removing unnecessary data is just as critical. Outdated or irrelevant data can bloat your database, degrade performance, and make maintenance a nightmare. Worse, retaining some types of data without valid justification might even lead to compliance issues.

Databases, Data Management, Performance Optimization

139 points by radimm 88 days ago | 115 comments

NASA SC24: NASA-GPT: Searching the Entire NASA Technical Reports Server Using AI (nasa.gov)
Researchers at NASA’s Ames Research Center in Silicon Valley are using artificial intelligence (AI) tools to create a data-sharing resource for the agency’s scientific and engineering staff.

Artificial Intelligence, NASA, Science, Research, Data Management

13 points by rbanffy 89 days ago | 1 comments

Snowflake opens chat-driven access to enterprise and third-party data (theregister.com)
Snowflake is set to preview a new platform it claims will help organizations build chatbots that can serve up data from its own analytics systems and those external to the cloud data platform vendor.

Data Management, Cloud Computing, Artificial Intelligence, Chatbots

6 points by rntn 99 days ago | 0 comments

Netflix's Distributed Counter Abstraction (netflixtechblog.com)
In our previous blog post, we introduced Netflix’s TimeSeries Abstraction, a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction. This counting service, built on top of the TimeSeries Abstraction, enables distributed counting at scale while maintaining similar low latency performance. As with all our abstractions, we use our Data Gateway Control Plane to shard, configure, and deploy this service globally.

Netflix, Distributed Systems, Data Management

133 points by benocodes 99 days ago | 71 comments

Utilizing HubSpot's APIs to Break Down Data Silos (accelant.com)
I’ve seen this movie countless times: Business Has Tons of Useful Data That’s Scattered Across Systems and Isn’t Being Used for Much (or anything).

Data Management, APIs, Business Intelligence, Marketing Automation

9 points by mooreds 100 days ago | 3 comments

Evolving a NoSQL Database Schema (karmanivero.us)
In a NoSQL environment, Entity Manager organizes the physical distribution of data to support efficient query operations.

NoSQL, Databases, Data Management, Optimization

83 points by smitty1e 102 days ago | 20 comments

Get me out of data hell (mataroa.blog)
It is 9:59 AM in Melbourne, 9th October, 2024. Sunlight filters through my windows, illuminating swirling motes of dust across my living room. There is a cup of tea in my hand. I take a sip and savor it.

Data Management, Personal Stories, Technology

698 points by pavel_lishin 112 days ago | 300 comments

Make It Ephemeral: Software Should Decay and Lose Data (pocoo.org)
Most software that exists today does not forget. Creating software that remembers is easy, but designing software that deliberately “forgets” is a bit more complex. By “forgetting,” I don't mean losing data because it wasn’t saved or losing it randomly due to bugs. I'm referring to making a deliberate design decision to discard data at a later time. This ability to forget can be an incredibly benefitial property for many applications. Most importantly software that forgets enables different user experiences.

Software Design, User Experience, Data Management, Ephemerality

31 points by BerislavLopac 112 days ago | 40 comments

Nulls: Revisiting null representation in modern columnar formats (dl.acm.org)
Nulls are common in real-world data sets, yet recent research on columnar formats and encodings rarely address Null representations.

Data Representation, Database Systems, Data Management, Columnar Formats, Null Values

54 points by tosh 113 days ago | 6 comments

A brief history of Notion's data catalog (notion.so)
Over the past few years, the number of data assets and systems Notion uses has skyrocketed.

Notion, Data Management, Software, History, Data Catalog

10 points by alex_hirner 119 days ago | 0 comments