Hacker News with Generative AI: Distributed Systems

CRDTs #2: Turtles All the Way Down (jhellerstein.github.io)
Modern distributed systems often seem to rest on an stack of turtles. For every guarantee we make, we seem to rely on a lower-layer assumption. Eventually we're left wondering: what is at the bottom?

Distributed Systems, Computer Science, Databases, CRDTs

53 points by pfarago 9 hours ago | 2 comments

CRDTs: Pros and Cons (Lattices and Lettuces?) (jhellerstein.github.io)
Over the next few days, I'm going to post a number of observations about CRDTs: Convergent Replicated Data Types. These are data structures that aspire to help us with coordination-free distributed programming, a topic that interests me a lot. How can developers (or languages/compilers) deliver distributed programs that are safe or correct in important ways, without employing expensive mechanisms for coordination that make the global cloud run as slowly as a sequential computer?

Distributed Systems, Programming, Data Structures, Computer Science, Coordination

5 points by KraftyOne 2 days ago | 0 comments

LLM-D: Kubernetes-Native Distributed Inference at Scale (github.com/llm-d)
llm-d is a Kubernetes-native distributed inference serving stack - a well-lit path for anyone to serve large language models at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators.

Kubernetes, Distributed Systems, Inference

10 points by bbzjk7 2 days ago | 2 comments

LLM-D: Kubernetes-Native Distributed Inference (llm-d.ai)
llm-d is a Kubernetes-native high-performance distributed LLM inference framework - a well-lit path for anyone to serve at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators.

Kubernetes, Inference, Distributed Systems, Performance

119 points by smarterclayton 3 days ago | 15 comments

The value of model checking in distributed protocols design (protocols-made-fun.com)
Recently, we have published two technical papers on arXiv that are both using model checkers as the main vehicle for verifying properties of fault-tolerant distributed algorithms.

Distributed Systems, Protocol Design, Model Checking, Formal Verification

9 points by todsacerdoti 3 days ago | 0 comments

Programming Models for Correct and Modular Distributed Systems (eecs.berkeley.edu)
Distributed systems are a fundamental part of modern computing, but they are notoriously difficult to program.

Distributed Systems, Programming, Software Engineering

7 points by matt_d 4 days ago | 0 comments

A lost decade chasing distributed architectures for data analytics? (duckdb.org)
TL;DR: We benchmark DuckDB on a 2012 MacBook Pro to decide: did we lose a decade chasing distributed architectures for data analytics?

Data Analytics, Distributed Systems, Benchmarks, Performance, Hardware

206 points by andreasha 4 days ago | 102 comments

Sheepdog - a distributed storage system for QEMU (github.com/sheepdog)

Distributed Systems, Storage Systems, QEMU, Virtualization

3 points by noctarius 8 days ago | 0 comments

Ask HN: What's your go-to message queue in 2025? (ycombinator.com)
The space is confusing to say the least.Message queues are usually a core part of any distributed architecture, and the options are endless: Kafka, RabbitMQ, NATS, Redis Streams, SQS, ZeroMQ... and then there's the “just use Postgres” camp for simpler use cases.I’m trying to make sense of the tradeoffs between:- async fire-and-forget pub/sub vs. sync RPC-like point to point communication- simple FIFO vs. priority queues and delay queues- intelligent brokers (e.g. RabbitMQ, NATS with filters) vs. minimal brokers (e.g.

Messaging Systems, Software Architecture, Distributed Systems, Programming, Cloud Computing

65 points by enether 8 days ago | 97 comments

Fossil: A Coherent Software Configuration Management System (fossil-scm.org)
Fossil is a simple, high-reliability, distributed SCM system with these advanced features:

Version Control, Distributed Systems, Tools

17 points by stefankuehnel 9 days ago | 1 comments

FlowG – Distributed Systems without raft (part 2) (medium.com)
Recently, I published the v0.37.0 release of FlowG, a Free and OpenSource low-code log processing software:

Distributed Systems, Open Source, Software, Logging

20 points by linkdd 9 days ago | 4 comments

Garbage collection of object storage at scale (warpstream.com)
Over the last 10 years, I’ve built several distributed systems on top of object storage, with WarpStream being the most recent.

Object Storage, Distributed Systems, Cloud Computing, Scalability

96 points by ko_pivot 13 days ago | 10 comments

TScale – Distributed training on consumer GPUs (github.com/Foreseerr)
This repo contains transformer train and inference code written in C++ and CUDA.

Machine Learning, GPUs, Distributed Systems, C++, CUDA

130 points by zX41ZdbW 19 days ago | 27 comments

Building MapReduce (Based on Google Paper) (ycombinator.com)
I read the MapReduce paper recently and wanted to try out the internal working by building it from scratch (at least a minimal version). Hope it helps someone trying to reproduce the same paper in future

Distributed Systems, Programming, MapReduce, Google

9 points by venkat1017x 20 days ago | 0 comments

Using only half the outbox pattern (medium.com)
In distributed systems, reliable communication between services cannot be taken for granted. You might update a database record successfully, but if publishing an event to Kafka or RabbitMQ fails immediately after, inconsistencies can appear — issues that may not be visible right away but can cause serious problems later.

Distributed Systems, Messaging, Reliability, Kafka, RabbitMQ

5 points by trimalchio55 21 days ago | 2 comments

Node.js implementation of the BitTorrent DHT protocol (npmjs.com)
Node.js implementation of the BitTorrent DHT protocol. BitTorrent DHT is the main peer discovery layer for BitTorrent, which allows for trackerless torrents. DHTs are awesome!

Node.js, BitTorrent, Peer-to-Peer, Distributed Systems, Networking

11 points by wslh 23 days ago | 0 comments

Sharding Mastodon, Part 1 (pgdog.dev)
Redirecting…

Mastodon, Social Media, Distributed Systems, Scaling

4 points by levkk 23 days ago | 0 comments

What If We Could Rebuild Kafka from Scratch? (morling.dev)
The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ’s Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration.

Kafka, Cloud Computing, Software, Distributed Systems, Apache

254 points by mpweiher 28 days ago | 220 comments

Ask HN: Has anyone used Riak? Thoughts? (ycombinator.com)
I’ve just stumbled upon RIAK. It seems like a very cool technology. Almost like an alternative to kubernetes. Has anyone used it in production? Why isn’t it more well known? It seems like an awesome solution.

Databases, Distributed Systems, Cloud Computing, Open Source

11 points by ag_rin 31 days ago | 14 comments

Decomposing Transactional Systems (transactional.blog)

Software, Databases, Distributed Systems

132 points by pongogogo 33 days ago | 8 comments

Decomposing Transactional Systems (transactional.blog)

Software, Architecture, Transactions, Distributed Systems

18 points by adastral 36 days ago | 0 comments

Consistent Hash Ring (selfboot.cn)
Consistent Hashing Ring is a special hashing algorithm primarily used for data distribution and load balancing in distributed systems.

Distributed Systems, Load Balancing, Hashing, Data Distribution

73 points by jcartw 36 days ago | 21 comments

Graham: Synchronizing Clocks by Leveraging Local Clock Properties (usenix.org)
High performance, strongly consistent applications are beginning to require scalable sub-microsecond clock synchronization.

Clock Synchronization, Distributed Systems

6 points by todsacerdoti 36 days ago | 0 comments

KIP-1150: Diskless Kafka Topics (apache.org)
No results

Kafka, Apache, Distributed Systems

32 points by enether 37 days ago | 3 comments

Erlang's not about lightweight processes and message passing (2023) (stevana.github.io)
I used to think that the big idea of Erlang is its lightweight processes and message passing. Over the last couple of years I’ve realised that there’s a bigger insight to be had, and in this post I’d like to share it with you.

Erlang, Programming Languages, Software Development, Distributed Systems, Concurrency

331 points by todsacerdoti 42 days ago | 197 comments

Engineering a Trace Details Page That Handles a Million Spans (signoz.io)

Engineering, Monitoring, Performance Optimization, Distributed Systems

10 points by vikrantgupta25 50 days ago | 1 comments

Building a modern durable execution engine from first principles (restate.dev)
We dive into the architecture details of Restate, a Durable Execution engine we built from the ground up. Restate requires no database/log or other system, but implements a full stack that competes with the best logs in terms of durability and operations.

Software, Architecture, Durability, Distributed Systems

98 points by whoiskatrin 57 days ago | 26 comments

Colossus: How we deliver SSD performance at HDD prices (cloud.google.com)
From YouTube and Gmail to BigQuery and Cloud Storage, almost all of Google’s products depend on Colossus, our foundational distributed storage system.

Cloud Computing, Storage, Google, Hardware, Distributed Systems

7 points by caution 58 days ago | 0 comments

The Synchrony Budget (morling.dev)
For building a system of distributed services, one concept I think is very valuable to keep in mind is what I call the synchrony budget: as much as possible, a service should minimize the number of synchronous requests which it makes to other services.

Distributed Systems, Software Architecture, Microservices

18 points by ingve 61 days ago | 9 comments

Conflict-Free Distributed Architecture for Append-Only Writes to Apache Iceberg (e6data.com)
Apache Iceberg is a cornerstone table format in modern data lakehouse systems. It is renowned for its ability to deliver transactional consistency, schema evolution, and snapshot isolation through a metadata-driven architecture.

Distributed Systems, Big Data, Data Lakehouse, Apache Iceberg

10 points by hackintoshrao 64 days ago | 2 comments