Sharding & partitioning
Split one dataset across many machines when it no longer fits or keeps up on one.
Visual model
Route each key to one slice of the data
When and why to shard
Vertical scaling means buying a bigger machine. It eventually hits a ceiling on storage, memory or write throughput.
Sharding splits data across several nodes. Each node owns a subset, so storage and write capacity grow with the number of shards.
Sharding is a heavy tool. Cross-shard queries, transactions and joins become hard or impossible. Use it only when caching and read replicas are no longer enough.
Choose the partition key carefully. Changing it later is painful.
Choosing a partition key
The partition key decides where each row lives. A good key spreads data and traffic evenly.
It should also match the common query, so one request reaches one shard instead of all of them.
- Hash-based: hash the key to pick a shard. Distribution is even, but range scans must query every shard.
- Range-based: assign a continuous range to each shard. Range queries are easy, but recent data can create a hot shard.
- Watch for hot partitions. A celebrity user or an increasing ID can concentrate traffic on one shard.
- Consistent hashing limits how much data moves when you add or remove a shard.
See it
Balanced storage can still hide hot traffic
Key takeaways
- Shard only after caching and replicas. It trades query power for write scalability.
- The partition key is the hardest decision to change. Pick one that avoids hot spots.
- Cross-shard joins and transactions are expensive. Keep common access patterns inside one shard.