Hacker News with Generative AI: Distributed Systems

rqlite turns 10: Lessons from a decade building Distributed Systems (philipotoole.com)
rqlite is a lightweight, open-source, distributed relational database written in Go, which uses SQLite as its storage engine and Raft for consensus.

Distributed Systems, Databases, Open Source, SQLite

9 points by otoolep 35 days ago | 0 comments

Making Postgres Distributed with FoundationDB (fabianlindfors.se)
Turning the revered Postgres into a distributed database is a tall order but not a new idea.

Databases, Postgres, Distributed Systems

45 points by emptysea 35 days ago | 17 comments

CRDTs #2: Turtles All the Way Down (jhellerstein.github.io)
Modern distributed systems often seem to rest on an stack of turtles. For every guarantee we make, we seem to rely on a lower-layer assumption. Eventually we're left wondering: what is at the bottom?

Distributed Systems, Computer Science, Databases, CRDTs

67 points by pfarago 36 days ago | 2 comments

CRDTs: Pros and Cons (Lattices and Lettuces?) (jhellerstein.github.io)
Over the next few days, I'm going to post a number of observations about CRDTs: Convergent Replicated Data Types. These are data structures that aspire to help us with coordination-free distributed programming, a topic that interests me a lot. How can developers (or languages/compilers) deliver distributed programs that are safe or correct in important ways, without employing expensive mechanisms for coordination that make the global cloud run as slowly as a sequential computer?

Distributed Systems, Programming, Data Structures, Computer Science, Coordination

6 points by KraftyOne 37 days ago | 0 comments

From RPC to transactions and durable executions (pramodb.com)
I spent some time reading about “Durable Execution Engines” (eg: Temporal) and explored possible connections to earlier concepts like database transactions, distributed transactions, and building RPC/Microservice based systems in a fault tolerant manner. In this post I’ll try to summarize some of my learnings. How useful it is will depend on how much of this you already know! Among other things, I relied on these great overviews: The Modern Transactional Stack (by some a16z folks) and What is Durable Execution?.

Microservices, Distributed Systems, Transactions, Fault Tolerance

30 points by pramodbiligiri 37 days ago | 1 comments

LLM-D: Kubernetes-Native Distributed Inference at Scale (github.com/llm-d)
llm-d is a Kubernetes-native distributed inference serving stack - a well-lit path for anyone to serve large language models at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators.

Kubernetes, Distributed Systems, Inference

10 points by bbzjk7 38 days ago | 2 comments

LLM-D: Kubernetes-Native Distributed Inference (llm-d.ai)
llm-d is a Kubernetes-native high-performance distributed LLM inference framework - a well-lit path for anyone to serve at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators.

Kubernetes, Inference, Distributed Systems, Performance

120 points by smarterclayton 38 days ago | 15 comments

The value of model checking in distributed protocols design (protocols-made-fun.com)
Recently, we have published two technical papers on arXiv that are both using model checkers as the main vehicle for verifying properties of fault-tolerant distributed algorithms.

Distributed Systems, Protocol Design, Model Checking, Formal Verification

9 points by todsacerdoti 39 days ago | 0 comments

Programming Models for Correct and Modular Distributed Systems (eecs.berkeley.edu)
Distributed systems are a fundamental part of modern computing, but they are notoriously difficult to program.

Distributed Systems, Programming, Software Engineering

7 points by matt_d 39 days ago | 0 comments

A lost decade chasing distributed architectures for data analytics? (duckdb.org)
TL;DR: We benchmark DuckDB on a 2012 MacBook Pro to decide: did we lose a decade chasing distributed architectures for data analytics?

Data Analytics, Distributed Systems, Benchmarks, Performance, Hardware

214 points by andreasha 39 days ago | 113 comments

Sheepdog - a distributed storage system for QEMU (github.com/sheepdog)

Distributed Systems, Storage Systems, QEMU, Virtualization

3 points by noctarius 43 days ago | 0 comments

Ask HN: What's your go-to message queue in 2025? (ycombinator.com)
The space is confusing to say the least.Message queues are usually a core part of any distributed architecture, and the options are endless: Kafka, RabbitMQ, NATS, Redis Streams, SQS, ZeroMQ... and then there's the “just use Postgres” camp for simpler use cases.I’m trying to make sense of the tradeoffs between:- async fire-and-forget pub/sub vs. sync RPC-like point to point communication- simple FIFO vs. priority queues and delay queues- intelligent brokers (e.g. RabbitMQ, NATS with filters) vs. minimal brokers (e.g.

Messaging Systems, Software Architecture, Distributed Systems, Programming, Cloud Computing

66 points by enether 43 days ago | 97 comments

Fossil: A Coherent Software Configuration Management System (fossil-scm.org)
Fossil is a simple, high-reliability, distributed SCM system with these advanced features:

Version Control, Distributed Systems, Tools

17 points by stefankuehnel 44 days ago | 1 comments

FlowG – Distributed Systems without raft (part 2) (medium.com)
Recently, I published the v0.37.0 release of FlowG, a Free and OpenSource low-code log processing software:

Distributed Systems, Open Source, Software, Logging

20 points by linkdd 44 days ago | 4 comments

Garbage collection of object storage at scale (warpstream.com)
Over the last 10 years, I’ve built several distributed systems on top of object storage, with WarpStream being the most recent.

Object Storage, Distributed Systems, Cloud Computing, Scalability

96 points by ko_pivot 48 days ago | 10 comments

TScale – Distributed training on consumer GPUs (github.com/Foreseerr)
This repo contains transformer train and inference code written in C++ and CUDA.

Machine Learning, GPUs, Distributed Systems, C++, CUDA

130 points by zX41ZdbW 54 days ago | 27 comments

Building MapReduce (Based on Google Paper) (ycombinator.com)
I read the MapReduce paper recently and wanted to try out the internal working by building it from scratch (at least a minimal version). Hope it helps someone trying to reproduce the same paper in future

Distributed Systems, Programming, MapReduce, Google

9 points by venkat1017x 55 days ago | 0 comments

Using only half the outbox pattern (medium.com)
In distributed systems, reliable communication between services cannot be taken for granted. You might update a database record successfully, but if publishing an event to Kafka or RabbitMQ fails immediately after, inconsistencies can appear — issues that may not be visible right away but can cause serious problems later.

Distributed Systems, Messaging, Reliability, Kafka, RabbitMQ

5 points by trimalchio55 56 days ago | 2 comments

Node.js implementation of the BitTorrent DHT protocol (npmjs.com)
Node.js implementation of the BitTorrent DHT protocol. BitTorrent DHT is the main peer discovery layer for BitTorrent, which allows for trackerless torrents. DHTs are awesome!

Node.js, BitTorrent, Peer-to-Peer, Distributed Systems, Networking

11 points by wslh 58 days ago | 0 comments

Sharding Mastodon, Part 1 (pgdog.dev)
Redirecting…

Mastodon, Social Media, Distributed Systems, Scaling

4 points by levkk 58 days ago | 0 comments

What If We Could Rebuild Kafka from Scratch? (morling.dev)
The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ’s Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration.

Kafka, Cloud Computing, Software, Distributed Systems, Apache

254 points by mpweiher 64 days ago | 220 comments

Ask HN: Has anyone used Riak? Thoughts? (ycombinator.com)
I’ve just stumbled upon RIAK. It seems like a very cool technology. Almost like an alternative to kubernetes. Has anyone used it in production? Why isn’t it more well known? It seems like an awesome solution.

Databases, Distributed Systems, Cloud Computing, Open Source

11 points by ag_rin 67 days ago | 14 comments

Decomposing Transactional Systems (transactional.blog)

Software, Databases, Distributed Systems

132 points by pongogogo 68 days ago | 8 comments

Decomposing Transactional Systems (transactional.blog)

Software, Architecture, Transactions, Distributed Systems

18 points by adastral 71 days ago | 0 comments

Consistent Hash Ring (selfboot.cn)
Consistent Hashing Ring is a special hashing algorithm primarily used for data distribution and load balancing in distributed systems.

Distributed Systems, Load Balancing, Hashing, Data Distribution

73 points by jcartw 71 days ago | 21 comments

Graham: Synchronizing Clocks by Leveraging Local Clock Properties (usenix.org)
High performance, strongly consistent applications are beginning to require scalable sub-microsecond clock synchronization.

Clock Synchronization, Distributed Systems

6 points by todsacerdoti 71 days ago | 0 comments

KIP-1150: Diskless Kafka Topics (apache.org)
No results

Kafka, Apache, Distributed Systems

32 points by enether 72 days ago | 3 comments

Erlang's not about lightweight processes and message passing (2023) (stevana.github.io)
I used to think that the big idea of Erlang is its lightweight processes and message passing. Over the last couple of years I’ve realised that there’s a bigger insight to be had, and in this post I’d like to share it with you.

Erlang, Programming Languages, Software Development, Distributed Systems, Concurrency

331 points by todsacerdoti 77 days ago | 197 comments

Engineering a Trace Details Page That Handles a Million Spans (signoz.io)

Engineering, Monitoring, Performance Optimization, Distributed Systems

10 points by vikrantgupta25 85 days ago | 1 comments

Building a modern durable execution engine from first principles (restate.dev)
We dive into the architecture details of Restate, a Durable Execution engine we built from the ground up. Restate requires no database/log or other system, but implements a full stack that competes with the best logs in terms of durability and operations.

Software, Architecture, Durability, Distributed Systems

98 points by whoiskatrin 92 days ago | 26 comments