Hacker News with Generative AI: Data Management

Snowflake opens chat-driven access to enterprise and third-party data (theregister.com)
Snowflake is set to preview a new platform it claims will help organizations build chatbots that can serve up data from its own analytics systems and those external to the cloud data platform vendor.

Data Management, Cloud Computing, Artificial Intelligence, Chatbots

6 points by rntn 8 days ago | 0 comments

Netflix's Distributed Counter Abstraction (netflixtechblog.com)
In our previous blog post, we introduced Netflix’s TimeSeries Abstraction, a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction. This counting service, built on top of the TimeSeries Abstraction, enables distributed counting at scale while maintaining similar low latency performance. As with all our abstractions, we use our Data Gateway Control Plane to shard, configure, and deploy this service globally.

Netflix, Distributed Systems, Data Management

133 points by benocodes 8 days ago | 71 comments

Utilizing HubSpot's APIs to Break Down Data Silos (accelant.com)
I’ve seen this movie countless times: Business Has Tons of Useful Data That’s Scattered Across Systems and Isn’t Being Used for Much (or anything).

Data Management, APIs, Business Intelligence, Marketing Automation

9 points by mooreds 9 days ago | 3 comments

Evolving a NoSQL Database Schema (karmanivero.us)
In a NoSQL environment, Entity Manager organizes the physical distribution of data to support efficient query operations.

NoSQL, Databases, Data Management, Optimization

83 points by smitty1e 10 days ago | 20 comments

Get me out of data hell (mataroa.blog)
It is 9:59 AM in Melbourne, 9th October, 2024. Sunlight filters through my windows, illuminating swirling motes of dust across my living room. There is a cup of tea in my hand. I take a sip and savor it.

Data Management, Personal Stories, Technology

698 points by pavel_lishin 21 days ago | 300 comments

Make It Ephemeral: Software Should Decay and Lose Data (pocoo.org)
Most software that exists today does not forget. Creating software that remembers is easy, but designing software that deliberately “forgets” is a bit more complex. By “forgetting,” I don't mean losing data because it wasn’t saved or losing it randomly due to bugs. I'm referring to making a deliberate design decision to discard data at a later time. This ability to forget can be an incredibly benefitial property for many applications. Most importantly software that forgets enables different user experiences.

Software Design, User Experience, Data Management, Ephemerality

31 points by BerislavLopac 21 days ago | 40 comments

Nulls: Revisiting null representation in modern columnar formats (dl.acm.org)
Nulls are common in real-world data sets, yet recent research on columnar formats and encodings rarely address Null representations.

Data Representation, Database Systems, Data Management, Columnar Formats, Null Values

54 points by tosh 22 days ago | 6 comments

A brief history of Notion's data catalog (notion.so)
Over the past few years, the number of data assets and systems Notion uses has skyrocketed.

Notion, Data Management, Software, History, Data Catalog

10 points by alex_hirner 28 days ago | 0 comments

A new JSON data type for ClickHouse (clickhouse.com)
JSON has become the lingua franca for handling semi-structured and unstructured data in modern data systems. Whether it’s in logging and observability scenarios, real-time data streaming, mobile app storage, or machine learning pipelines, JSON’s flexible structure makes it the go-to format for capturing and transmitting data across distributed systems.

Data Management, JSON, ClickHouse

382 points by markhneedham 30 days ago | 122 comments

Profile generation and data management with dbm (deepnote.com)

Data Management, Databases, Machine Learning

13 points by Katerinahyn 30 days ago | 3 comments

Ask HN: How big of a problem is unstructured data for companies? (ycombinator.com)
I read somewhere that 90% of companies have data like documents, PDFs, videos, images, audio clips, and other content that are unstructured that will be a big obstacle for ai.

Data Management, Artificial Intelligence, Business, Big Data, Unstructured Data

16 points by dark7 38 days ago | 16 comments

Restic: Backups done right (restic.net)
Restic is a modern backup program that can back up your files:

Backup Software, Data Management, Security

231 points by fanf2 39 days ago | 107 comments

Secure Custom Fields by WordPress.org (wordpress.org)
Secure Custom Fields (SCF) turns WordPress sites into a fully-fledged content management system by giving you all the tools to do more with your data.

WordPress, Security, Data Management

172 points by ValentineC 40 days ago | 191 comments

How CERN serves 1EB of data via FUSE [video] (kernel-recipes.org)
CERN, the European Organization for Nuclear Research, generates vast amounts of data from experiments at the Large Hadron Collider (LHC).

CERN, Data Management, Physics, Video

252 points by pabs3 50 days ago | 115 comments

Ask HN: What do you use to backup your VMs? (ycombinator.com)
How do you backup VMs with installed Postgres, MariaDB instances and local files?

Virtual Machines, Backup, Databases, Data Management

74 points by bykof 57 days ago | 57 comments

Insights after 11 years with Datomic [video] (youtube.com)

Database, Data Management, Software, Video, Interviews

205 points by eduction 57 days ago | 106 comments

Datomic and Content Addressable Techniques (latacora.com)
Latacora collects and analyzes data about services our clients use. You may have read about our approach to building security tooling, but the tl;dr is we make requests to all the (configuration metadata) read-only APIs available to us and store the results in S3. We leverage the data to understand our clients’ infrastructure and identify security issues and misconfigurations. We retain the files (“snapshots”) to support future IR/forensics efforts.

Data Management, Security, Cloud Storage, Infrastructure

166 points by lvh 66 days ago | 40 comments

Rearchiving 2M hours of digital radio, a comprehensive process (digitalpreservation-blog.nb.no)

Digital Preservation, Archives, Radio, Data Management

144 points by todsacerdoti 83 days ago | 36 comments

Facebook uses 10k Blu-ray discs to store 'cold' data (2014) (pcworld.com)

Data Storage, Facebook, Technology, Data Management

19 points by padthai 84 days ago | 9 comments

Don't Believe the Big Database Hype, Stonebraker Warns (datanami.com)

Databases, Data Management, Industry News, Opinion

5 points by turtledragonfly 92 days ago | 1 comments

Amazon S3 now supports conditional writes (amazon.com)