Hacker News with Generative AI: Data Storage

Reducing Cloud Spend: Migrating Logs from CloudWatch to Iceberg with Postgres (crunchydata.com)
As a database service provider, we store a number of logs internally to audit and oversee what is happening within our systems.

Cloud Computing, Cost Optimization, Data Storage, Databases, Logging

11 points by rubiquity 6 days ago | 0 comments

Scoping a Local-First Image Archive (scottishstoater.com)
For years, I’ve been thinking about how we store and access our digital files, especially photos.

Data Storage, Photography, Personal Projects

31 points by stog 13 days ago | 13 comments

Preview: Amazon S3 Tables and Lakehouse in DuckDB (duckdb.org)
TL;DR: We are happy to announce a new preview feature that adds support for Apache Iceberg REST Catalogs, enabling DuckDB users to connect to Amazon S3 Tables and Amazon SageMaker Lakehouse with ease.

Database, Data Storage, Cloud Computing, Preview

177 points by hn1986 15 days ago | 47 comments

The real failure rate of EBS (planetscale.com)
PlanetScale has deployed millions of Amazon Elastic Block Store (EBS) volumes across the world. We create and destroy tens of thousands of them every day as we stand up databases for customers, take backups, and test our systems end-to-end. Through this experience, we have an unique viewpoint into the failure rate and mechanisms of EBS, and have spent a lot of time working on how to mitigate them.

Cloud Computing, Databases, Reliability, Data Storage

113 points by QuinnyPig 15 days ago | 30 comments

Archival Storage (dshr.org)
I'm honored to appear in what I believe is the final series of these seminars. Most of my previous appearances have focused on debunking some conventional wisdom, and this one is no exception. My parting gift to you is to stop you wasting time and resources on yet another seductive but impractical idea — that the solution to storing archival data is quasi-immortal media. As usual, you don't have to take notes.

Data Storage, Archival Data, Technology, Digital Preservation

362 points by rbanffy 15 days ago | 193 comments

Theory crafting a system for 1000 simultaneous micro SD card ingests (level1techs.com)

Hardware, Data Storage, Theory Crafting

62 points by daemontus 19 days ago | 36 comments

Ask HN: What do you think of BDXL (100GB disks)? (ycombinator.com)
I still have a need to archive data and I'm thinking about getting a BDXL writer and some disks. Is this a dumb thing to do in 2025?

Data Storage, Technology, Hardware, Archiving

9 points by criddell 21 days ago | 20 comments

Put a data center on the moon? (ieee.org)
Lonestar Data Holdings is sending a test mission, aiming to safeguard valuable data

Space Exploration, Data Storage, Technology, Moon

76 points by pseudolus 34 days ago | 170 comments

Hard Drive Graveyard (benjdd.com)
Hard drive graveyard

Hardware, Technology, Data Storage, Data Recovery

7 points by bddicken 35 days ago | 0 comments

What 5 Megabytes of Data Looked Like in 1966 (62,500 punched cards) (vintag.es)
In 1966, computing was in its infancy, and the concept of data storage and processing looked drastically different from today’s instant access to vast amounts of information.

History, Technology, Data Storage

18 points by dxs 42 days ago | 10 comments

Are SSDs more reliable than hard drives? (2021) (backblaze.com)
Solid-state drives (SSDs) continue to become more and more a part of the data storage landscape. And while our SSD 101 series has covered topics like upgrading, troubleshooting, and recycling your SSDs, we’d like to test one of the more popular declarations from SSD proponents: that SSDs fail much less often than our old friend, the hard disk drive (HDD).

Data Storage, Hardware, Reliability, SSDs, HDDs

77 points by fanf2 42 days ago | 69 comments

12 years of Backblaze data center storage drives, visualized (benjdd.com)
1 small node -> 100 drives

Data Storage, Visualization, Hardware, Reliability

147 points by bddicken 42 days ago | 47 comments

Backblaze Drive Stats for 2024 (backblaze.com)
As of December 31, 2024, we had 305,180 drives under management. Of that number, there were 4,060 boot drives and 301,120 data drives. This report will focus on those data drives as we review the Q4 2024 annualized failure rates (AFR), the 2024 failure rates, and the lifetime failure rates for the drive models in service as of the end of 2024.

Hardware, Data Storage, Cloud Storage, Statistics

646 points by TangerineDream 50 days ago | 205 comments

Seagate's HDD scandal deepens clues point at Chinese Chia mining farms (tomshardware.com)

Hard Drives, Data Storage, Cryptocurrency, China

15 points by snvzz 52 days ago | 0 comments

Cloudflare R2 Incident on February 6, 2025 (cloudflare.com)
Multiple Cloudflare services, including our R2 object storage, were unavailable for 59 minutes on Thursday, February 6th. This caused all operations against R2 to fail for the duration of the incident, and caused a number of other Cloudflare services that depend on R2 — including Stream, Images, Cache Reserve, Vectorize and Log Delivery — to suffer significant failures.

Cloud Computing, Security, Web Services, Data Storage, Outage

40 points by ko_pivot 54 days ago | 13 comments

For privacy: Change of our refund policy from 30 to 14 days (mullvad.net)
As part of our ongoing commitment to storing less user data and protect your privacy, we’re updating our refund policy.

Privacy, Refund Policy, Data Storage

26 points by raybb 61 days ago | 10 comments

Husky: Efficient Compaction at Datadog Scale (datadoghq.com)
In a previous blog post, we introduced our Husky event store system. Husky is a distributed storage system that is layered over object storage (e.g., Amazon S3, Google Cloud Storage, Azure Blob Storage, etc.), with the query system acting as a cache over this storage. We also did a deep dive into Husky’s ingestion pipelines that we built to handle the scale of our customer data. In this post, we’ll cover how we designed Husky’s underlying data storage layer.

Data Storage, Distributed Systems, Cloud Storage, Databases, Event Streaming

30 points by gttalbot 62 days ago | 2 comments

Apache Accumulo 4.0 Feature Preview (apache.org)
Apache Accumulo® is a sorted, distributed key/value store that provides robust, scalable data storage and retrieval.

Apache Accumulo, Databases, Big Data, Data Storage, Key/Value Stores

14 points by teleforce 67 days ago | 0 comments

Storage is cheap, but not thinking about logging is expensive (counting-stuff.com)
The bad habits of data over-collection run deep.

Data Storage, Logging, Best Practices, Data Management

11 points by todsacerdoti 70 days ago | 0 comments

Seagate smashes largest HDD world record with 36TB hard drive (techradar.com)

Hardware, Storage, Data Storage, World Records

25 points by airhangerf15 71 days ago | 11 comments

Parquet and ORC's many shortfalls for machine learning, and what to do about it? (starburst.io)
At the turn of the century (around a quarter of a decade ago), over 99% of the data management industry used row-oriented storage to store data for all workloads involving structured data — including transactional and analytical workloads.

Machine Learning, Data Storage, Data Management, Big Data

6 points by abadid 84 days ago | 1 comments

37signals Dev – Monitoring 10 Petabytes of Data in Pure Storage (37signals.com)
How we use Prometheus to have metrics and alerts for Pure Storage.

Monitoring, Data Storage, Cloud Infrastructure, Prometheus

15 points by doppp 88 days ago | 0 comments

I Track My Health Data in Markdown: Lessons in Digital Longevity (ycombinator.com)
I’ve spent years tracking my sleep, diet, and exercise with apps and wearables. But here’s the problem: when an app gets discontinued or stops syncing, the data—and all the insights—disappear.

Health Data, Markdown, Data Storage, Longevity

17 points by biohackerking 108 days ago | 16 comments

Century-Scale Storage (law.harvard.edu)
This piece looks at a single question. If you, right now, had the goal of digitally storing something for 100 years, how should you even begin to think about making that happen? How should the bits in your stewardship be stored with such a target in mind? How do our methods and platforms look when considered under the harsh unknowns of a century? There are plenty of worthy related subjects and discourses that this piece does not touch at all.

Data Storage, Digital Preservation, Technology

18 points by fanf2 111 days ago | 0 comments

Century Scale Storage (law.harvard.edu)
This piece looks at a single question. If you, right now, had the goal of digitally storing something for 100 years, how should you even begin to think about making that happen? How should the bits in your stewardship be stored with such a target in mind? How do our methods and platforms look when considered under the harsh unknowns of a century?

Data Storage, Technology

37 points by worik 111 days ago | 9 comments

Internet Object – New Age Data Serialization After JSON (internetobject.org)
Revolutionize your data exchange and storage with a format that's built for efficiency, clarity and reliability. A Text Based Data Serialization and Structured Storage Format Beyond JSON!

Data Serialization, JSON, Data Storage, Programming, Internet

6 points by Starz0r 114 days ago | 1 comments

Terabit-scale high-fidelity diamond data storage (nature.com)
In the era of digital information, realizing efficient and durable data storage solutions is paramount.

Data Storage, Technology, Science

4 points by westurner 117 days ago | 3 comments

Hetzner Object Storage (hetzner.com)
Object Storage is the S3 compatible storage solution that grows with your data requirements - highly available, secure and flexible.

Cloud Storage, Data Storage, Hetzner, Object Storage

11 points by matteocontrini 120 days ago | 7 comments

Big Endian's Guide to SQLite Storage (jabid.in)
I wanted to learn how databases like SQLite store data under the hood, so I decided to write some code to inspect the database file. SQLite famously stores the entire database in a single file, and the file format is very well documented. Here is one diagram1 to get started instead of the roughly 13,848 words in that document.