Hacker News with Generative AI: Scaling

The Practical Guide to Scaling Django (slimsaas.com)
Most Django scaling guides focus on theoretical maximums. But real scaling isn’t about handling hypothetical millions of users - it’s about systematically eliminating bottlenecks as you grow. Here’s how to do it right, based on patterns that work in production.

Django, Web Development, Scaling, Performance Optimization, Software Architecture

149 points by rbanffy 6 days ago | 33 comments

Data movement bottlenecks to large-scale model training: Scaling past 1e28 FLOP (epochai.org)
Data movement bottlenecks limit LLM scaling beyond 2e28 FLOP, with a "latency wall" at 2e31 FLOP. We may hit these in ~3 years. Aggressive batch size scaling could potentially overcome these limits.

Machine Learning, Computer Science, Scaling

12 points by jasondavies 18 days ago | 1 comments

Possible futures for the Ethereum protocol, part 2: The Surge (eth.limo)
At the beginning, Ethereum had two scaling strategies in its roadmap. One (eg. see this early paper from 2015) was "sharding": instead of verifying and storing all of the transactions in the chain, each node would only need to verify and store a small fraction of the transactions. This is how any other peer-to-peer network (eg. BitTorrent) works too, so surely we could make blockchains work the same way.

Ethereum, Blockchain, Scaling, Decentralized Networks, Cryptocurrency

95 points by bpierre 35 days ago | 52 comments

Upgrading Uber's MySQL Fleet (uber.com)

Databases, MySQL, Scaling, Uber, Technology

236 points by benocodes 38 days ago | 198 comments

What can we do to make games scale? (twitter.com)

Game Development, Scaling, Optimization

4 points by LorenDB 46 days ago | 2 comments

Does It Scale (Down)? (bugsink.com)
It’s 2024, and software is in a ridiculous state.

Software Engineering, Scaling, Trends

11 points by vanschelven 46 days ago | 3 comments

How we built ngrok's data platform (ngrok.com)
At ngrok, we manage an extensive data lake with an engineering team of one (me!).

Data Platforms, Engineering, Scaling, Cloud Computing, Software

164 points by samber 52 days ago | 40 comments

How Discord stores trillions of messages (2023) (discord.com)
In 2017, we wrote a blog post on how we store billions of messages.

Messaging, Data Storage, Discord, Scaling

435 points by jakey_bakey 54 days ago | 234 comments

Sharding and Scaling PostgreSQL, No Citus (pg-sharding.tech)

PostgreSQL, Databases, Scaling, Performance

12 points by rkhapov 64 days ago | 2 comments

Scaling Rails and Postgres to millions of users at Microsoft (stepchange.work)

Scaling, Rails, Postgres, Databases, Microsoft

202 points by htormey 85 days ago | 91 comments

We run migrations across 2,800 microservices (monzo.com)

Microservices, Software Development, Deployment, Scaling, Cloud

29 points by willsewell 86 days ago | 12 comments

How OpenAI Scaled Kubernetes to 7,500 Nodes by Removing One Plugin (betterstack.com)

Kubernetes, OpenAI, Scaling, Optimization, Cloud Computing

10 points by richbray 113 days ago | 1 comments

Scaling One Million Checkboxes to 650M checks (eieio.games)

Scaling, User Interfaces, Performance, Games

237 points by todsacerdoti 118 days ago | 51 comments

SPQR 1.5.0: a production-ready system for horizontal scaling of PostgreSQL (github.com/pg-sharding)

Databases, PostgreSQL, Open Source, Software, Scaling

12 points by denchick 125 days ago | 4 comments

Our Wandering Path to Supporting 1000s of Domain Names (fusionauth.io)

Domain Names, Scaling, Infrastructure, Software, Cloud Computing

16 points by mooreds 128 days ago | 0 comments

Building and scaling Notion's data lake (notion.so)

Data Lakes, Databases, Scaling, Notion

239 points by alexzeitler 130 days ago | 82 comments

SPQR: Scaling PostgreSQL via Sharding (mintlify.app)

Databases, PostgreSQL, Scaling, Sharding

14 points by denchick 139 days ago | 4 comments

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data (epochai.org)

Data, Artificial Intelligence, Scaling, Limits

56 points by trott 156 days ago | 58 comments

Nexus zkVM: Efficient, massively-parallel, zero-knowledge proving (nexus.xyz)

Zero-Knowledge Proofs, Cryptography, Blockchain, Privacy, Scaling

12 points by SkyMarshal 156 days ago | 3 comments

Pinterest Scaled to 11M Users with Only 6 Engineers (medium.com)

Pinterest, Engineering, Scaling, User Growth

13 points by brainer 158 days ago | 4 comments

Meritocracy at Scale (scale.com)

Meritocracy, Scaling, Business, Technology

39 points by tosh 160 days ago | 56 comments

Scaling Clubhouse From 10K to 10M Users In 6 Months With Postgres (stepchange.work)

Scaling, Clubhouse, Postgres, User Growth, Database

16 points by radimm 188 days ago | 3 comments

Autoscale Kubernetes workloads on any cloud using any event (kedify.io)

Kubernetes, Cloud Computing, Automation, Scaling

55 points by innovate 205 days ago | 44 comments

Scaling to Count Billions (canva.dev)

Scaling, Data, Technology, Software

52 points by kiyanwang 206 days ago | 14 comments

Using ClickHouse to scale an events engine (github.com/getlago)

Databases, Scaling, Events, ClickHouse

237 points by wyndham 224 days ago | 97 comments

Scaling will never get us to AGI (garymarcus.substack.com)

Artificial Intelligence, AGI, Scaling

43 points by isaacfrond 224 days ago | 119 comments