Hacker News with Generative AI: Data Storage

Huawei developing SSD-tape hybrid amid US tech restrictions (blocksandfiles.com)
Huawei’s in-house development of Magneto-Electric Disk (MED) archive storage technology combines an SSD with a Huawei-developed tape drive to provide warm (nearline) and cold data storage.

Hardware, Storage, Data Storage, Technology, China

75 points by akyuu 4 days ago | 100 comments

Transactional Object Storage? (mbrt.dev)
I was frustrated by the gap between stateless and stateful applications in the cloud. While I could easily spin up a stateless application as a “serverless” function in any major cloud provider and pretty much forget about it, persisting data between requests was a game of pick two among three: cheap, strongly consistent, portable.

Cloud Computing, Data Storage, Application Development

24 points by mbrt 4 days ago | 9 comments

Upspin: A framework for naming everyone's everything (upspin.io)
Upspin is an attempt to address problems like these, and many more.

Naming Systems, Decentralized Systems, Data Storage

6 points by nateb2022 6 days ago | 2 comments

Backblaze Drive Stats for Q3 2024 (backblaze.com)
As of the end of Q3 2024, Backblaze was monitoring 292,647 hard disk drives (HDDs) and solid state drives (SSDs) in our cloud storage servers located in our data centers around the world.

Hardware, Data Storage, Cloud Computing, Statistics

39 points by pfych 8 days ago | 3 comments

Floppy Disk Storage (history) (ibm.com)
The once-ubiquitous data storage device gave rise to the modern software industry

History, Data Storage, Technology, Software

10 points by susam 10 days ago | 6 comments

Show HN: OpenDAL, one API to access all the storages (S3, Azblob, HDFS, etc.) (github.com/apache)
Apache OpenDAL™: Access Data Freely

Data Storage, Open Source, Software, Apache

30 points by tison 14 days ago | 2 comments

LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite (rxdb.info)
So you are building that web application and you want to store data inside of your users browser. Maybe you just need to store some small flags or you even need a fully fledged database.

Web Development, Data Storage, Browser Storage, Databases, JavaScript

25 points by thunderbong 25 days ago | 1 comments

Consider adding warnings against using ZFS native encryption (github.com/openzfs)
Among experienced zfs users and developers, it seems to be conventional wisdom that zfs native encryption is not suitable for production usage, particularly when combined with snapshotting and zfs send/recv. There is a long standing data corruption issue with many firsthand user reports:

Software, ZFS, Data Storage, Security

18 points by 6581 26 days ago | 4 comments

19th-century photography technique employed in novel data storage method (ieee.org)
19th-century photography technique employed in novel data storage method

Data Storage, Photography, Technology, Science

74 points by abe94 28 days ago | 39 comments

DNA stores data in bits after epigenetic upgrade (nature.com)
DNA stores data in bits after epigenetic upgrade

Genetics, Science, Biology, Data Storage

11 points by pseudolus 28 days ago | 2 comments

LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite (rxdb.info)
So you are building that web application and you want to store data inside of your users browser. Maybe you just need to store some small flags or you even need a fully fledged database.

Web Development, Data Storage, JavaScript, Browser Storage, Databases

3 points by eventreduce 36 days ago | 0 comments

Icechunk: An open-source, cloud-native transactional tensor storage engine (earthmover.io)
Icechunk is a brand new open-source transactional storage engine for tensor / ND-array data designed for use on cloud object storage.

Open Source, Cloud Computing, Data Storage, Machine Learning

15 points by TomAugsrpuger 37 days ago | 0 comments

LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite (rxdb.info)
So you are building that web application and you want to store data inside of your users browser. Maybe you just need to store some small flags or you even need a fully fledged database.

Web Development, Data Storage, JavaScript, Browser Storage, Databases

17 points by todsacerdoti 37 days ago | 9 comments

Show HN: Vortex – a high-performance columnar file format (github.com/spiraldb)
Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.

Databases, Data Storage, Performance, Data Compression, Open Source

249 points by gatesn 38 days ago | 61 comments

Windows 11 24H2 hoards 8.63 GB of junk you can't delete (theregister.com)
Windows 11 24H2 users are finding there is undeletable data that remains on their devices after installing the recently released feature update.

Windows, Operating Systems, Software, Data Storage

151 points by LinuxBender 40 days ago | 195 comments

Improving Parquet Dedupe on Hugging Face Hub (huggingface.co)
The Xet team at Hugging Face is working on improving the efficiency of the Hub's storage architecture to make it easier and quicker for users to store and update data and models.

Data Storage, Machine Learning, Hugging Face, Efficiency

47 points by ylow 44 days ago | 17 comments

IEEE Roadmap Outlines Development of Mass Digital Storage Technology (ieee.org)

Technology, Computer Science, Data Storage, IEEE, Roadmaps

7 points by doener 44 days ago | 0 comments

SlateDB – An embedded database built on object storage (slatedb.io)
Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage to provide bottomless storage capacity, high durability, and easy replication.

Databases, Object Storage, Software, Data Storage

162 points by notamy 51 days ago | 55 comments

Build a serverless ACID database with this one neat trick (atomic PutIfAbsent) (eatonphil.com)
Delta Lake is an open protocol for serverless ACID databases. Due to its simplicity, scalability, and the number of open-source implementations, it's quickly becoming the DuckDB of serverless transactional databases. Iceberg is a contender too, and is similar in many ways. But since Delta Lake is simpler (simple != better) that's where we'll focus in this post.

Databases, Serverless, Data Storage, Open Source

86 points by todsacerdoti 53 days ago | 30 comments

How Discord stores trillions of messages (2023) (discord.com)
In 2017, we wrote a blog post on how we store billions of messages.

Messaging, Data Storage, Discord, Scaling

435 points by jakey_bakey 54 days ago | 234 comments

A Guide to Imaging Obscure Floppy Disk Formats (zenodo.org)
Memory institutions are grappling with the challenges posed by digital carriers in their collections. While solutions for more recent carriers like hard drives, optical discs, and flash storage are readily available, the landscape becomes trickier when dealing with older formats such as floppy disks.

Digital Preservation, Data Storage, Floppy Disks, Archives

6 points by zdw 54 days ago | 0 comments

Valkey 8 sets a new bar for open-source in-memory NoSQL data storage (beehiiv.com)
Vienna, Austria: Valkey, the Redis fork, is kicking rump and taking names. At Open Source Summit Europe, the Linux Foundation announced the release of Valkey 8.0, a giant step forward to the open-source in-memory NoSQL data store.

Open Source, NoSQL, Data Storage, Redis, Linux Foundation

26 points by thunderbong 55 days ago | 0 comments

Tinymind – Write and sync your blog and memo data with GitHub (github.com/mazzzystar)
Turn your GitHub into a blog & memo data storage place with one-click Sign in. No server needed - every input automatically syncs to your GitHub repository.

GitHub, Blogging, Data Storage, Software, Open Source

25 points by thunderbong 57 days ago | 1 comments

No Data Lasts Forever (lilysthings.org)
No matter what you do, no data will last forever. You hard drive will fail. Your backup drives will fail. Tech companies will go under and sell off their assets. Optical Discs will rot. Books will decompose. Even if none of these things happen, a natural or manmade disaster could come by and destroy it all anyway.

Data Storage, Digital Preservation, Longevity

31 points by todsacerdoti 59 days ago | 14 comments

Mkfs.fat on Linux vs. OS/2 2.1 (uninformativ.de)
I often run VMs with old operating systems in them, like OS/2 2.1. Exchanging data with these VMs is a bit tedious, because there's usually no networking. I can basically only use floppy images or HDD images. For the longest time, I used a 256 MB disk image that I once created manually under MS-DOS. This usually worked, but being confined to a fixed size is not that great.

Operating Systems, Virtual Machines, Data Storage

38 points by zdw 61 days ago | 14 comments

Human genome stored on 'everlasting' memory crystal (phys.org)
University of Southampton scientists have stored the full human genome on a 5D memory crystal—a revolutionary data storage format that can survive for billions of years.

Science, Data Storage, Technology, Genetics

3 points by bookofjoe 62 days ago | 1 comments

5D 'Eternity Crystal' Stores 360 TB of Data for Billions of Years (newatlas.com)
Scientists have stored the entire human genome on a five-dimensional crystal that’s capable of digitally storing up to 360 terabytes of information and is built to survive for billions of years.

Data Storage, Technology, Science, Innovation

5 points by andrewl 62 days ago | 2 comments

Valkey 8 sets a new bar for open-source in-memory NoSQL data storage (beehiiv.com)
Vienna, Austria: Valkey, the Redis fork, is kicking rump and taking names. At Open Source Summit Europe, the Linux Foundation announced the release of Valkey 8.0, a giant step forward to the open-source in-memory NoSQL data store. This release focuses on enhancing performance, reliability, and observability, marking a major milestone for the project initially forked from Redis due to licensing changes.

Open Source, NoSQL Databases, Performance, Redis, Data Storage

6 points by CrankyBear 63 days ago | 1 comments

Human genome stored on 'everlasting' memory crystal (southampton.ac.uk)

Genetics, Technology, Nanotechnology, Data Storage

59 points by breck 63 days ago | 32 comments

M-DISC: The storage medium that lasts 1000 years (wikipedia.org)
M-DISC (Millennial Disc) is a write-once optical disc technology introduced in 2009 by Millenniata, Inc.[1] and available as DVD and Blu-ray discs.[2]

Storage, Technology, Hardware, Data Storage, Archival

11 points by popcalc 64 days ago | 2 comments