Hacker News with Generative AI: Distributed Systems

What If We Could Rebuild Kafka from Scratch? (morling.dev)
The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ’s Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration.
Kafka, Cloud Computing, Software, Distributed Systems, Apache
223 points by mpweiher 1 day ago | 192 comments
Ask HN: Has anyone used Riak? Thoughts? (ycombinator.com)
I’ve just stumbled upon RIAK. It seems like a very cool technology. Almost like an alternative to kubernetes. Has anyone used it in production? Why isn’t it more well known? It seems like an awesome solution.
Databases, Distributed Systems, Cloud Computing, Open Source
10 points by ag_rin 4 days ago | 14 comments
Decomposing Transactional Systems (transactional.blog)
Software, Databases, Distributed Systems
131 points by pongogogo 6 days ago | 8 comments
Decomposing Transactional Systems (transactional.blog)
Software, Architecture, Transactions, Distributed Systems
18 points by adastral 8 days ago | 0 comments
Consistent Hash Ring (selfboot.cn)
Consistent Hashing Ring is a special hashing algorithm primarily used for data distribution and load balancing in distributed systems.
Distributed Systems, Load Balancing, Hashing, Data Distribution
73 points by jcartw 9 days ago | 21 comments
Graham: Synchronizing Clocks by Leveraging Local Clock Properties (usenix.org)
High performance, strongly consistent applications are beginning to require scalable sub-microsecond clock synchronization.
Clock Synchronization, Distributed Systems
6 points by todsacerdoti 9 days ago | 0 comments
KIP-1150: Diskless Kafka Topics (apache.org)
No results
Kafka, Apache, Distributed Systems
32 points by enether 10 days ago | 3 comments
Erlang's not about lightweight processes and message passing (2023) (stevana.github.io)
I used to think that the big idea of Erlang is its lightweight processes and message passing. Over the last couple of years I’ve realised that there’s a bigger insight to be had, and in this post I’d like to share it with you.
Erlang, Programming Languages, Software Development, Distributed Systems, Concurrency
331 points by todsacerdoti 15 days ago | 197 comments
Engineering a Trace Details Page That Handles a Million Spans (signoz.io)
Engineering, Monitoring, Performance Optimization, Distributed Systems
10 points by vikrantgupta25 23 days ago | 1 comments
Building a modern durable execution engine from first principles (restate.dev)
We dive into the architecture details of Restate, a Durable Execution engine we built from the ground up. Restate requires no database/log or other system, but implements a full stack that competes with the best logs in terms of durability and operations.
Software, Architecture, Durability, Distributed Systems
98 points by whoiskatrin 30 days ago | 26 comments
Colossus: How we deliver SSD performance at HDD prices (cloud.google.com)
From YouTube and Gmail to BigQuery and Cloud Storage, almost all of Google’s products depend on Colossus, our foundational distributed storage system.
Cloud Computing, Storage, Google, Hardware, Distributed Systems
7 points by caution 31 days ago | 0 comments
The Synchrony Budget (morling.dev)
For building a system of distributed services, one concept I think is very valuable to keep in mind is what I call the synchrony budget: as much as possible, a service should minimize the number of synchronous requests which it makes to other services.
Distributed Systems, Software Architecture, Microservices
18 points by ingve 34 days ago | 9 comments
Conflict-Free Distributed Architecture for Append-Only Writes to Apache Iceberg (e6data.com)
Apache Iceberg is a cornerstone table format in modern data lakehouse systems. It is renowned for its ability to deliver transactional consistency, schema evolution, and snapshot isolation through a metadata-driven architecture.
Distributed Systems, Big Data, Data Lakehouse, Apache Iceberg
10 points by hackintoshrao 37 days ago | 2 comments
Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework (github.com/ai-dynamo)
NVIDIA Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.
Generative AI, Inference, Distributed Systems
150 points by ashvardanian 39 days ago | 39 comments
When Kafka is not the right Move (rejot.dev)
When designing distributed systems, event streaming platforms such as Kafka are the preferred solution for asynchronous communication. In fact, on Kafka’s official website, the first use-case listed is messaging1. My believe is that this default choice can lead to problems and unnecessary complexity. The main reason for this is the conversion between state and events, which we’ll look at in this article through a game of chess.
Distributed Systems, Software Engineering, Event Streaming, Kafka
11 points by WilcoKruijer 49 days ago | 0 comments
Handling Database Failures in a Distributed System with RabbitMQ Workers (ycombinator.com)
I have a worker that processes tasks from RabbitMQ and inserts data into a database. The system operates at high scale, handling thousands of messages per second, which makes proper failure handling crucial to avoid overwhelming the system.
Distributed Systems, RabbitMQ, Database Management, Failure Handling, High Performance
11 points by yakirbitan 50 days ago | 7 comments
Distributed systems programming has stalled (shadaj.me)
Over the last decade, we’ve seen great advancements in distributed systems, but the way we program them has seen few fundamental improvements.
Distributed Systems, Programming, Technology, Software Development
287 points by shadaj 58 days ago | 217 comments
The Anatomy of a Durable Execution Stack from First Principles (restate.dev)
We dive into the architecture details of Restate, a Durable Execution engine we built from the ground up. Restate requires no database/log or other system, but implements a full stack that competes with the best logs in terms of durability and operations.
Software, Architecture, Distributed Systems, Cloud Computing, Programming
22 points by sewen 65 days ago | 8 comments
The Pijul Manual (pijul.org)
Welcome to the Pijul book, an introduction to Pijul, a distributed version control system that is at the same time theoretically sound, fast and easy to learn and use.
Version Control, Software, Distributed Systems
13 points by tusslewake 73 days ago | 0 comments
What Is the Byzantine Generals Problem in Distributed Systems? (scalablethread.com)
The Byzantine Generals Problem is a thought experiment in distributed computing to understand the challenges of reaching a consensus when some nodes may be untrustworthy or unreliable (node behavior). It shows how difficult it can be to coordinate actions when some system members may act dishonestly.
Distributed Systems, Consensus, Reliability, Computer Science
20 points by thunderbong 83 days ago | 3 comments
Chrono: A Peer-to-Peer Network with Verifiable Causality (arxiv.org)
Logical clocks are a fundamental tool to establish causal ordering of events in a distributed system.
Distributed Systems, Peer-to-Peer Networks, Computer Science, Causality
8 points by wslh 84 days ago | 1 comments
Hydro: Distributed Programming Framework for Rust (hydro.run)
Hydro is a high-level distributed programming framework for Rust.
Rust, Distributed Systems, Programming Frameworks
275 points by ardel95 85 days ago | 49 comments
Durable execution should be lightweight (dbos.dev)
Everyone knows serious programs must make data durable. You persist data on disk or in a database so it doesn’t disappear the second your program crashes or your server is restarted. But we also take it for granted that programs themselves aren’t durable. When you restart your server, your data might be safe in the database, but any programs you were running are gone, and if you want them back, you have to restart them yourself.
Software Development, Programming Languages, Distributed Systems, Cloud Computing
65 points by KraftyOne 86 days ago | 44 comments
Husky: Efficient Compaction at Datadog Scale (datadoghq.com)
In a previous blog post, we introduced our Husky event store system. Husky is a distributed storage system that is layered over object storage (e.g., Amazon S3, Google Cloud Storage, Azure Blob Storage, etc.), with the query system acting as a cache over this storage. We also did a deep dive into Husky’s ingestion pipelines that we built to handle the scale of our customer data. In this post, we’ll cover how we designed Husky’s underlying data storage layer.
Data Storage, Distributed Systems, Cloud Storage, Databases, Event Streaming
30 points by gttalbot 87 days ago | 2 comments
Turning the database inside-out (2015) (kleppmann.com)
Databases are global, shared, mutable state. That’s the way it has been since the 1960s, and no amount of NoSQL has changed that. However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are?
Databases, Software Architecture, Programming, Distributed Systems
192 points by andection 90 days ago | 95 comments
Fault Tolerance in Tandem Computer Systems (1986) [pdf] (azurewebsites.net)
Fault Tolerance, Computer Systems, Distributed Systems, Networking
61 points by sillywalk 90 days ago | 35 comments
Every System is a Log: Avoiding coordination in distributed applications (restate.dev)
Building resilient distributed applications remains a tough challenge.
Distributed Systems, Software Engineering, Reliability, Coordination, Logging
229 points by sewen 92 days ago | 155 comments
Nation-scale Matrix deployments will fail using the community version of Synapse (matrix.org)
Matrix, Synapse, Distributed Systems, Open Source, Security
141 points by zaik 97 days ago | 169 comments
Distributed Transactions at Scale in Amazon DynamoDB (2023) (blogspot.com)
This paper appeared in July at USENIX ATC 2023. If you haven't read about the architecture and operation of DynamoDB, please first read my summary of the DynamoDB ATC 2022 paper. The big omission in that paper was discussion about transactions. This paper amends that. It is great to see DynamoDB, and AWS in general, is publishing/sharing more widely than before.
Distributed Systems, Databases, Transactions
82 points by lambrospetrou 104 days ago | 60 comments
Misty: A secure distributed actor language (mistysystem.com)
Public Domain 2025 Douglas Crockford
Programming Languages, Distributed Systems, Security, Public Domain
90 points by m90 104 days ago | 38 comments