Hacker News with Generative AI: Scalability

Garbage collection of object storage at scale (warpstream.com)
Over the last 10 years, I’ve built several distributed systems on top of object storage, with WarpStream being the most recent.

Object Storage, Distributed Systems, Cloud Computing, Scalability

96 points by ko_pivot 13 days ago | 10 comments

Simplifying the Ethereum Layer 1 (eth.limo)
Ethereum aims to be the world ledger: the platform that stores civilization's assets and records, the base layer for finance, governance, high-value data authentication, and more. This requires two things: scalability and resilience.

Ethereum, Blockchain, Scalability, Finance

9 points by gasull 13 days ago | 3 comments

Engineering a Trace Details Page That Handles a Million Spans (signoz.io)

Engineering, Observability, Performance, Scalability, Software

13 points by pranay01 61 days ago | 1 comments

Handling billions of invocations – best practices from AWS Lambda (amazon.com)
AWS Lambda is a highly scalable and resilient serverless compute service. With over 1.5 million monthly active customers and tens of trillions of invocations processed, scalability and reliability are two of the most important service tenets. This post provides recommendations and insights for implementing highly distributed applications based on the Lambda service team’s experience building its robust asynchronous event processing system. It dives into challenges you might face, solution techniques, and best practices for handling noisy neighbors.

Serverless Computing, AWS Lambda, Best Practices, Scalability, Reliability

15 points by mariuz 66 days ago | 0 comments

How much traffic can a pre-rendered Next.js site handle? (martijnhols.nl)
I've often said things like "A pre-rendered site can easily serve hundreds of concurrent users", because, well, I've never seen one fail.

Web Development, Next.js, Performance, Scalability

75 points by MartijnHols 78 days ago | 61 comments

Can atproto scale down? (bad-example.com)
It's frequently stated[by who?] that some core components of the AT-Protocol architecture are expensive to host and don't scale down.

Scalability, AT-Protocol, Web Development

9 points by icy 96 days ago | 3 comments

Are efficiency and horizontal scalability at odds? (buttondown.com)
Why are scalable systems locally-inefficent, and locally-efficient systems unscalable? Plus, new book release!

Software, Scalability, Efficiency

49 points by todsacerdoti 100 days ago | 37 comments

Scalable OLTP in the Cloud: What's the Big Deal? (blogspot.com)
This paper is from Pat Helland, the apostate philosopher of database systems, overall a superb person, and a good friend of mine. The paper appeared this week at CIDR'24. (Check out the program for other interesting papers). The motivating question behind this work is: "What are the asymptotic limits to scale for cloud OLTP (OnLine Transaction Processing) systems?" Pat says that the CIDR 2023 paper "Is Scalable OLTP in the Cloud a Solved Problem?" prompted this question.

Database Systems, Cloud Computing, Scalability

93 points by SchwKatze 116 days ago | 49 comments

The hidden complexity of scaling WebSockets (composehq.com)
With the rising demand for sync engines and real-time feature, WebSockets have become a critical component for modern applications.

WebSockets, Real-Time Applications, Software Development, Scalability, Networking

194 points by atul-jalan 119 days ago | 68 comments

Scaling to users requires Synapse Pro (element.io)
Since launching Synapse Pro to support nation-scale Matrix deployments, we’ve had a lot of questions asking how it differs from vanilla “community” AGPL Synapse.

Software, Open Source, Scalability

15 points by gmemstr 126 days ago | 15 comments

PostgreSQL Meets ScyllaDB's Lightning Speed and Monstrous Scalability (medium.com)
PostgreSQL’s inability to handle our growing write-intensive workloads and real-time demands made it clear that a change was necessary.

Databases, Performance, Scalability, PostgreSQL, ScyllaDB

7 points by sebg 148 days ago | 2 comments

400TB Single Cluster: OceanBase Powers Kwai`s Core Business (oceanbase.github.io)
Kwai is a short video app boasting more than 10 million daily active users. How does it efficiently process highly concurrent user requests? Kwai once deployed multiple MySQL clusters in the backend to support high traffic with large data storage and satisfactory performance. What are the weak points of this conventional sharding solution? What pushed Kwai to select distributed databases and eventually deploy OceanBase Database?

Databases, Scalability, Distributed Systems

44 points by jinqueeny 149 days ago | 10 comments

Managing Large-Scale Redis Clusters on K8s – Kuaishou's Approach (kubeblocks.io)
As a popular short-form video application, Kuaishou relies heavily on Redis to deliver low-latency responses to its users.

Redis, Kubernetes, Scalability, Performance, Microservices

26 points by west0n 179 days ago | 5 comments

Docker Compose Isn't Enough (tealok.tech)
Docker Compose creates problems as you scale a single server to many applications. We should learn from it and build something better.

Docker, Microservices, Scalability, DevOps

126 points by hartspear 191 days ago | 198 comments

Virtual Texturing (playerunknownproductions.net)
Building a game engine that supports earth-sized environments requires us to pay attention to the scalability of our engine. For example, the amount of data existing on the planet cannot be loaded into RAM or VRAM. Every system we design needs to take into account the assumption that data is not loaded and needs to be streamed in or in some cases even needs to be generated first.

Game Development, Virtual Reality, Scalability

4 points by carlos-menezes 193 days ago | 0 comments

Why software only moves forward (swizec.com)
At scale there are no rollbacks and no cut-overs. Your software only moves forward.

Software Development, Software Architecture, Scalability

92 points by speckx 199 days ago | 64 comments

How do you deploy in 10 seconds? (bearblog.dev)
This post describes my lessons learned after 10 years running production environments in sizes ranging from "just getting started" to a "Series F" company with 4-9 SLAs.

Deployment, Software Engineering, Scalability, Production Environments

62 points by mpweiher 207 days ago | 54 comments

Building a better and scalable system for data migrations (yorickpeterse.com)
Recently I read about a new Rust web framework called rwf. I was especially curious about its approach to handling database migrations, as many frameworks released in recent years tend to either overlook this or provide a mechanism that won't scale to larger projects.

Software Development, Databases, Rust, Scalability

68 points by YorickPeterse 211 days ago | 36 comments

Join-Accumulate Machine: A Semi-Coherent Scalable Trustless VM (graypaper.com)
We present a comprehensive and formal definition of Jam, a protocol combining elements of both Polkadot and Ethereum.

Blockchain, Scalability, Trustless Computing, Cryptography, Distributed Systems

16 points by danicuki 224 days ago | 1 comments

Were RNNs all we needed? (arxiv.org)
The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training.

Deep Learning, Sequence Modeling, Scalability

520 points by beefman 232 days ago | 260 comments

Stop Designing Your Web Application for Millions of Users When You Dont Have 100 (darrenhorrocks.co.uk)
It’s easy to get carried away when you’re building a new web app. You’ve got big ideas, you picture millions of users flocking to your platform, and you start imagining the kind of infrastructure needed to handle all that traffic. So, you build for scale from day one—optimising databases, setting up powerful servers, and ensuring everything is robust enough for massive growth.

Web Development, Optimization, Scalability

120 points by riz_ 245 days ago | 98 comments

SwiftUI Apps at Scale (jacobbartlett.substack.com)

SwiftUI, Mobile Development, User Interfaces, Scalability

17 points by jakey_bakey 262 days ago | 0 comments

How to Build a Scalable Ingestion Pipeline for Enterprise GenAI Applications (enterprisebot.ai)

Generative AI, Enterprise Applications, Data Engineering, Scalability

7 points by ritzaco 262 days ago | 0 comments

Behind AWS S3's Scale (highscalability.com)

Cloud Computing, Scalability

22 points by zerojames 266 days ago | 11 comments

Unified Grid: How we re-architected Slack for our largest customers (slack.engineering)