News Feed / Timeline

Clarify the problem before you draw a single box

In a FAANG interview the first few minutes are about scoping, not architecture. State the functional requirements crisply, then pin down the non-functional ones, because the non-functional numbers are what actually drive the design. The functional surface here is small: a user can post content; a user can follow and unfollow others; a user can load a home timeline of recent posts from the people they follow; and the timeline is ordered by something reasonable — pure recency to start, ranking later. Explicitly cut scope you will not build now: search, notifications, direct messages, comment threads, and the ranking model itself. Saying out loud what you are not building is a signal of seniority.

The non-functional requirements are where this problem lives. Feed reads are extremely hot and latency-sensitive, so reads must be served from precomputed data, never assembled by querying the social graph on the request path. Fan-out must tolerate celebrity accounts whose follower counts are five-to-eight orders of magnitude larger than a normal user's. And the timeline can be eventually consistent: it is completely acceptable for a post to appear in a follower's feed a few seconds after it is published. That last concession is the single most important thing you negotiate, because it unlocks asynchronous fan-out and caching.

Turn those into a measurable contract, which is exactly what the simulator encodes. The targets to design against: availability at or above 98 percent (feeds keep loading even when the database stalls), p99 feed latency at or below 120ms, sustained read throughput of at least 12,000 requests per second through the spike, peak saturation at or below 0.9 so no component pins at 100 percent, and a cost footprint at or below 9 provisioned units. Those five numbers are the rubric. Keep them on the whiteboard.

• Functional: post; follow/unfollow; render home timeline; reasonable recency/ranking.
• Out of scope (say it): search, notifications, DMs, the ranking ML model.
• Non-functional contract: availability >= 0.98, p99 <= 120ms, throughput >= 12k rps, saturation <= 0.9, cost <= 9 units.
• The unlock: eventually-consistent feeds are acceptable — a few seconds of lag is fine.

Back-of-the-envelope: find where the pain actually is

Estimate at full product scale first, then map onto what the simulator stresses. Assume 300 million daily active users. A user opens the app and reloads the feed perhaps 10 times a day, giving roughly 3 billion feed reads per day. Divided by 86,400 seconds that is about 35,000 reads per second on average, and real traffic is peaky, so a 2x to 3x diurnal peak puts you near 70,000 to 100,000 reads per second. Posts are far rarer: assume one post per active user per day, so 300 million posts per day, about 3,500 posts per second average. The read-to-write ratio is therefore roughly 10 to 1, and real systems run anywhere from 10:1 to 100:1. The headline: this is an overwhelmingly read-dominated system, so reads must be cheap and writes can afford to do extra work.

Now quantify the write amplification, because that is the part that bites. Fan-out-on-write means each post is copied into every follower's timeline. If the average post reaches, say, 500 followers, then 3,500 posts per second becomes 3,500 x 500 = about 1.75 million timeline inserts per second across the fleet. That is large but shardable — spread across hundreds of feed-store partitions it is a few thousand writes per partition per second. The problem is not the average; it is the tail. A celebrity with 50 million followers who posts once generates 50 million inserts from a single API call. Even at an aggressive 100,000 inserts per second of drain, that one post takes 500 seconds to fully fan out. Do it synchronously inside the post request and the request times out and the datastore melts. This single calculation is the entire justification for asynchronous fan-out plus a pull path for celebrities.

Storage sanity-checks the cache decision. A materialized timeline entry needs little more than a post id, an author id, a timestamp, and a ranking score — call it 32 to 64 bytes. Keep the most recent 800 entries per user and a timeline is about 50KB; across 300 million users that is roughly 15TB of materialized feeds. That fits comfortably in a sharded in-memory cache cluster, which is why precomputed timelines live in a cache rather than being rebuilt per request. The posts themselves (a few hundred bytes of text each) accumulate to tens of TB per year in the durable store, and media bytes never travel this path at all — they are served from a CDN by URL.

Map this onto the gym. The simulator collapses the full system into one representative read shard carrying 12,000 feed reads per second, with a celebrity event that triples offered load to 36,000. Your job is to keep that read tier fast and available while the write storm and database failures hit. The real-world math tells you which knobs matter: a very high cache hit ratio, a queue that absorbs the fan-out burst, and enough read-API replicas that losing one does not pin the rest.

• ~300M DAU x ~10 opens = ~3B reads/day ~= 35k rps avg, ~70k-100k rps at peak.
• ~300M posts/day ~= 3,500 posts/sec; read:write ~= 10:1 — read-dominated.
• Fan-out amplification: 3,500 posts/s x ~500 followers ~= 1.75M timeline writes/sec.
• Celebrity tail: 50M followers x 1 post = 50M writes — ~500s even at 100k/s drain. Never synchronous.
• Materialized feeds: ~50KB/user x 300M ~= 15TB — fits a sharded cache.
• Gym mapping: 12k base reads x 3 = 36k offered at the celebrity peak.

High-level architecture: five components and the two paths

The reference topology is a five-stage pipeline: clients, a timeline cache, a feed API service, a fan-out queue, and a posts datastore. The trick to understanding it is to see that two different request types flow through these components. The read path (loading a timeline) is the hot one: a client request hits the timeline cache, and on the overwhelming majority of requests the materialized feed is already sitting there and is returned in single-digit milliseconds. The write path (publishing a post) is the rare one: the post is durably written, then handed to the fan-out queue, and asynchronous workers drain the queue to insert the new post into each follower's materialized timeline in the cache. Posting returns to the user immediately; the fan-out happens in the background.

Walk the components and justify each against the requirements. The timeline cache holds per-user materialized feeds and absorbs essentially all read traffic; it is the component that makes reads fast and is your primary availability shield when the database is unhealthy. The feed API is a stateless, replicated service that handles cache misses, merges in celebrity posts at read time (the pull half of the hybrid), and accepts new posts; replication gives both throughput headroom and failure tolerance. The fan-out queue decouples a bursty, potentially gigantic write amplification from the durable store and the cache, so a celebrity post is buffered and drained at a controlled rate rather than landing as a synchronous tidal wave. The posts datastore is the durable source of truth for post content, partitioned by post id or author id. Media is intentionally absent here: it is uploaded to object storage and served through a CDN, so image and video bytes never touch the feed read path.

The reference graph the simulator scores you against makes the sizing concrete: the timeline cache at a 0.95 hit ratio (sized at 120,000 rps of read capacity), the feed API at 4 replicas of 6,000 rps each (24,000 rps of capacity), the fan-out queue draining at 40,000 with an 80,000-request buffer, and the posts store at 12,000 rps. The footprint scores at 7 units — cache 1, feed API 4 (one per replica), queue 1, database 1 — comfortably under the cost ceiling of 9. The shape matters more than the exact numbers: a fat cache in front, a modestly replicated stateless tier, a deep buffer to ride out write bursts, and a database that is never on the synchronous read path for cache hits.

• Read path: client -> timeline cache (hit) -> return materialized feed in ~ms.
• Write path: post -> durable store -> fan-out queue -> async workers -> followers' cached feeds.
• Reference sizing: cache hit 0.95 (cap 120k), Feed API 4x6,000, queue drain 40k / buffer 80k, posts DB 12k.
• Footprint = 7 units (cache 1 + API 4 + queue 1 + db 1), under the cost cap of 9.
• Media is off-path: object storage + CDN, never through the feed API.

Deep dive 1 — Fan-out: push, pull, and the celebrity bomb

Fan-out is the defining decision of this problem, and there are three honest models. Fan-out-on-write (push) materializes each follower's timeline when the author posts: at read time the feed is already assembled, so reads are extremely fast, but writes are amplified by the follower count. Fan-out-on-read (pull) stores only the posts; at read time you gather the people a user follows, fetch their recent posts, and merge them: writes are trivial, but reads become an expensive scatter-gather across the social graph and get slower the more accounts a user follows. Recomputing every feed from scratch on each load is the non-answer — it is fan-out-on-read with no caching and it does not survive contact with traffic.

Pure push is excellent until a celebrity posts. The earlier estimate is the whole argument: 50 million followers means 50 million timeline writes for one post, and doing that synchronously in the request path guarantees a timeout and a database meltdown. Pure pull avoids that write explosion but punishes every normal read, because users who follow hundreds of accounts now trigger hundreds of fetches per feed load. The standard answer — and the one the rubric awards full marks — is the hybrid: push for normal accounts so the common read is a single cache lookup, and pull for celebrities so their posts are merged in at read time instead of being written 50 million times. A follower's feed becomes the union of their precomputed push timeline and a small, cached set of recent posts from the handful of celebrities they follow.

Two execution details separate a passing answer from a strong one. First, the push half runs asynchronously: the post API writes the post durably, enqueues a fan-out job, and returns; a worker fleet pulls from the queue and performs the inserts. The queue is what absorbs the burst — in the reference it drains at 40,000 per second with an 80,000-request buffer, so a sudden pile of fan-out work is buffered and bled out over time rather than slamming the database synchronously. Add backpressure and batching for high-follower accounts so a single huge job does not starve everyone else, and define a follower threshold (commonly in the tens of thousands) above which an account flips from push to pull. Second, because queues deliver at-least-once, fan-out workers must be idempotent: dedupe on the pair (post id, follower id) so a retried job does not insert the same post twice. The interviewer is listening for the words asynchronous, backpressure, and idempotent.

• Push: fast reads, write amplification = follower count. Pull: cheap writes, slow scatter-gather reads.
• Hybrid (the answer): push for normal users, pull-and-merge for celebrities above a follower threshold.
• Celebrity post = 50M writes; do it async through a queue, never synchronously in the request.
• Queue absorbs the burst: drain 40k/s, buffer 80k — bleed the storm out over time.
• At-least-once delivery means workers must be idempotent: dedupe on (post_id, follower_id).

Deep dive 2 — The materialized timeline cache is the whole ballgame

If fan-out is how writes scale, the cache is how reads survive. The home timeline is a materialized, per-user list kept hot in a sharded cache, and the hit ratio on that cache is the single most important number in the design — it drives latency, saturation, and, decisively, availability under failure. Start with latency. A cache hit returns in roughly 3ms; a miss falls through the feed API, the queue, and the database, costing on the order of 40 to 50ms. Because p99 is by definition a tail request, at a 5 percent miss rate the 99th-percentile read is itself a miss — so the simulator's scored median p99 lands near 48ms, comfortably under the 120ms ceiling. The interesting question is what a sagging hit ratio costs. It does two things at once: more reads take the slow path, and the extra miss traffic piles onto the downstream tier. On a well-provisioned design the tail only creeps up (the reference still holds about 57ms even at a 0.80 hit ratio); on an under-provisioned one the miss flood saturates the API, queue, and DB and the tail blows past the ceiling — which is exactly what happens to the under-tuned starter, whose p99 balloons to roughly 195ms.

Availability is where the hit ratio truly earns its keep, and the simulator's database outage proves it. When the posts store stalls, only the traffic that misses the cache and falls through to the dead database actually fails — cache hits are served regardless. At a 0.95 hit ratio, just 5 percent of reads are exposed during the outage; at 0.80, four times as many fail, and because the outage overlaps the 3x celebrity peak those failures are weighted by the heaviest load of the run. Carry the arithmetic over the scored window and that is the difference between 98.8 percent availability (a pass) and 95.4 percent (a clear fail against the 98 percent bar). Raising the cache hit ratio from 0.80 to 0.95 is, almost single-handedly, what clears the availability, latency, and saturation SLOs at once — verified by re-running the under-tuned starter with nothing changed but the hit ratio. This is the lesson the chaos timeline is built to teach: a high cache hit ratio turns a database outage from an incident into a non-event.

Caches at this scale introduce two structural concerns. Sharding: partition materialized timelines by user id so the read load and the 15TB of feed data spread evenly across nodes; a normal post's fan-out naturally scatters writes across many follower shards, so write hot spots are rare on the push path. Hot keys: the heat concentrates on the pull path instead, where a celebrity author's recent-posts list is read by millions of followers per second. Handle it by caching that small celebrity post set with a short TTL and replicating it across cache replicas so the reads spread, rather than letting one key pin one node. The pull path deliberately converts a write hot spot (50 million inserts) into a read hot spot (one small, cacheable, replicated list) — a far cheaper thing to serve.

• Hit ~3ms; miss falls through API+queue+DB at ~40-50ms. At a 5% miss rate the p99 request IS a miss, so the scored median p99 ~= 48ms (< 120ms).
• On DB outage only cache misses fail: 0.95 hit ratio exposes 5%, 0.80 exposes 20%.
• That gap is 98.8% vs 95.4% availability in the sim — pass vs fail the 98% SLO.
• Low hit ratio also blows the tail on an under-sized design: the starter's p99 hits ~195ms.
• Shard timelines by user_id to spread the ~15TB and the read load.
• Celebrity read hot key: cache the small recent-posts list with a short TTL and replicate it.

Bottlenecks under load and how to scale each one

The simulator runs a deliberate gauntlet, and each event maps to a real bottleneck and a specific scaling move. Walk it in order. Baseline is 12,000 reads per second rising to 1.6x at peak hours; at a 0.95 hit ratio the cache (sized at 120,000 rps) serves about 95 percent of that directly, so its utilization sits near 0.1 at baseline and only reaches about 0.3 at the 3x celebrity peak — meaning the scored peak-saturation metric lands around 0.30, well under the 0.9 cap. Only the 5 percent miss stream flows down to the API, queue, and database. The fix for ordinary read growth is simply to keep the hit ratio high and let the cache do the work — adding database capacity would be solving the wrong layer.

At 16 seconds a feed API node is killed. Because the feed API only sees cache misses, the offered load on that tier is small, and the reference runs 4 replicas of 6,000 rps; losing one still leaves 18,000 rps of capacity against a few thousand rps of miss traffic. This is N+1 (really N+several) redundancy: replicas here buy failure tolerance and saturation headroom more than raw throughput. Under-provision the read path — too few replicas, too shallow a queue, or a low hit ratio that swells the miss stream — and the kill plus the celebrity spike combine to pin a survivor above 0.9 saturation (the under-tuned starter peaks at over 1.2). The scaling move is to provision the miss-handling tier so that losing one node during the spike still leaves comfortable headroom.

At 20 seconds the celebrity posts and offered load triples to 36,000 reads per second. The cache, sized at 120,000 rps of capacity, rises only to about 0.30 utilization, so reads stay fast and nothing redlines — provided the hit ratio is high enough that the miss stream (now about 1,800 rps) stays small. On the write side, the same event is the fan-out storm: the queue's 40,000 drain rate and 80,000 buffer absorb the surge of timeline inserts and bleed them toward the database over time instead of synchronously. At 24 seconds the database stalls outright; as analyzed above, only the 5 percent miss stream fails, holding availability near 98.8 percent. At 32 seconds the database latency spikes 8x; this inflates the miss path's latency, but because the spike is brief relative to the full 40-second scored window — the SLO takes the median of per-tick p99 — and because the rare miss traffic keeps the slowed DB at low utilization, the measured p99 barely moves and stays under 120ms. The throughput SLO of 12,000 is met throughout because the cache keeps serving the full read volume — the median served rate the simulator scores sits around 23,000 rps (the run spends much of its time at 1.6x to 3x load, and the cache serves all of it), well above the floor.

The summary of scaling moves: scale reads by raising the cache hit ratio and keeping timelines materialized, not by enlarging the database; scale the write/fan-out burst with a deeper queue buffer and more async workers, not by writing inline; scale media with a CDN so it never touches this path; and add API replicas for redundancy and saturation headroom, sized so a node loss during the spike is survivable. The one move that is always wrong is recomputing feeds with a big join across posts and follows on every read — that join over the social graph per request is exactly what all of this exists to avoid.

• Read growth: keep hit ratio high (cache util ~0.1 at baseline, ~0.3 even at the 3x peak) — do not scale the DB for this.
• Node kill: 4 API replicas means losing one still leaves ~18k rps vs a few-k miss stream (N+several).
• Celebrity 3x: cache rides it (util ~0.3); queue drain 40k + buffer 80k absorbs the fan-out storm.
• DB outage / latency x8: only the ~5% miss stream is affected; availability ~98.8%, median p99 stays < 120ms.
• Never: one big SQL JOIN across posts+follows per read. That is the anti-pattern.

Key trade-offs to articulate

A senior answer names the tensions it is choosing between. The biggest is consistency versus latency. Asynchronous fan-out means a follower can see a celebrity's post (via pull) before a normal author's post has finished propagating (via push), and a feed can be a few seconds stale. You accept this because the requirement explicitly allows eventual consistency, and the payoff is that reads are a single cache lookup. If the product needed read-your-writes or strict ordering, the whole push pipeline would have to change.

The second tension is storage versus compute, which is exactly the push-versus-pull axis. Push spends storage and write bandwidth to pre-materialize feeds so reads are cheap; pull spends read-time compute to assemble feeds so writes are cheap. The hybrid places the boundary at a follower threshold, paying storage for the many normal accounts where it is cheap and paying read-time compute for the few celebrities where pre-materializing would be ruinous. Moving that threshold is a tuning knob: lower it and more accounts pull, shrinking write amplification but adding read-time merge cost; raise it and the opposite. Be ready to defend a number in the tens of thousands of followers.

Finally, cost versus headroom. The reference footprint scores 7 units against a ceiling of 9, so there is room to add a replica, but over-provisioning is penalized — the discipline is to scale to the load and the failure modes, not past them. Replicas exist for redundancy and saturation headroom, not vanity; the queue buffer is sized to ride out the realistic burst, not infinite; and the database stays modest precisely because the cache keeps it off the hot path. The recurring theme across every trade-off is the same: push the cost onto the rare write so the common read stays fast, cheap, and available.

• Consistency vs latency: accept seconds of staleness to make reads one cache hit.
• Storage vs compute: push pre-materializes (storage), pull assembles at read (compute) — hybrid splits at a follower threshold.
• Cost vs headroom: 7 of 9 units; replicas and buffers sized for the failure modes, not for show.
• Unifying principle: make the rare write pay so the common read stays fast and available.