Notification System

Clarifying the requirements

Start by pinning down what "notification system" means here, because the scope determines the architecture. The functional requirements are narrow and concrete: accept events and deliver them across three channels (push via APNs/FCM, SMS, and email); respect per-user preferences, opt-outs and quiet hours; and deduplicate plus rate-limit per user so nobody gets the same alert five times. The non-functional requirements are where the difficulty lives: absorb sudden fan-out spikes where a single event explodes into millions of sends; tolerate flaky and throttling third-party providers without collapsing; and guarantee at-least-once delivery for important (transactional) messages.

Two clarifying questions earn points immediately. First, what delivery guarantee do we need? The answer is at-least-once for transactional traffic (OTPs, password resets, payment receipts) and best-effort for marketing. Exactly-once across third-party providers is impossible: you hand a message to APNs, the call times out, and you genuinely do not know whether it was delivered. So you must retry, and retries mean duplicates, which is why dedup is not optional cosmetics but the mechanism that makes at-least-once safe to use. Second, who triggers events and how fast must they get a response? Producers (the news service, the billing service) fire events and must not block on the fan-out. That single constraint forces an asynchronous, queue-backed design and is the highest-weighted decision in the rubric.

Explicitly scope out what you are not building so you do not drown: you are not designing the in-app notification feed or websocket presence layer, not the campaign-authoring UI, and not the analytics warehouse. You are building the delivery pipeline from "event accepted" to "handed to a provider," plus the bookkeeping (preferences, dedup, delivery status) that pipeline needs.

• Functional: multi-channel send (push/SMS/email); honor preferences, opt-outs, quiet hours; dedup and per-user rate-limit.
• Non-functional: absorb fan-out spikes; survive throttling and crashing providers; at-least-once for transactional messages.
• Guarantee: at-least-once + idempotency/dedup, never exactly-once.
• Out of scope: in-app feed, campaign UI, analytics pipeline.

Back-of-the-envelope capacity estimation

Anchor on a plausible scale, then map it to the simulator. Assume 100 million daily active users who each receive about 10 notifications per day across all channels. That is one billion notifications per day, or one billion divided by 86,400 seconds, roughly 11,600 sends per second on average. The simulator's baseline of 10,000 events per second is exactly this order of magnitude, which is not an accident: it is the steady-state delivery rate of a real consumer-scale service.

The average is the easy part; the spike is the design driver. A breaking-news push to 20 million eligible devices that you want drained within five minutes is 20,000,000 divided by 300, about 67,000 sends per second sustained, layered on top of baseline. The simulator compresses this fan-out into a load multiplier rather than literally injecting 20 million rows: it ramps to 1.5x at 10 seconds (a marketing blast, 15,000 per second) and 1.8x at 20 seconds (breaking news, 18,000 per second). With the engine's roughly 5 percent sinusoidal load jitter, peak offered load is about 18,900 per second. Treat that 18,900 as the number every downstream stage must survive.

Channel mix matters for provider planning. Push is cheap and high-throughput (APNs and FCM accept batched multicast payloads), so call it 80 percent of volume; email is maybe 15 percent; SMS is 5 percent but the most constrained, because SMS providers rate-limit hard (a single long-code number does on the order of one message per second, and even short codes and number pools cap in the hundreds to low thousands). That asymmetry is why each channel gets its own worker pool and its own rate budget rather than a shared sender.

Storage and state are modest and should not be over-engineered. A delivery log record (user id, channel, template id, status, timestamps) is about 300 bytes; one billion per day is roughly 300 GB per day, so 30 days of hot history is about 9 TB in a wide-column or time-series store. Dedup keys live in an in-memory store like Redis with a TTL (say 24 hours): even one billion keys at about 50 bytes is on the order of 50 GB, trivially sharded. The preferences table is 100 million users at a couple hundred bytes, about 20 GB, small enough to cache almost entirely. Payloads are about 1 KB, so even the 67,000-per-second peak is only about 67 MB/s of egress: bandwidth is never the bottleneck. The bottleneck is always compute and provider rate limits.

• Average load: 100M DAU x 10/day = 1B/day = about 11.6k sends/sec (sim baseline 10k).
• Spike: 20M-recipient push drained in 5 min = about 67k/sec; sim models this as 1.8x = 18,000/sec, about 18,900 with jitter.
• Channel mix: push 80% (batchable), email 15%, SMS 5% (severely rate-limited).
• State: about 300 GB/day delivery log, about 9 TB hot; dedup keys about 50 GB in Redis with TTL; preferences about 20 GB, cacheable.

High-level architecture and why each piece exists

The pipeline is five logical stages, and the simulator's reference graph maps onto them one-to-one. A Notify API gateway accepts events, authenticates the producer, looks up preferences and opt-outs, deduplicates, and writes the message to a queue, then returns immediately. A durable send queue buffers the fan-out so a spike never propagates back to producers. A fleet of channel-sender workers consumes the queue, applies per-provider rate limiting and retries, and calls the providers. The providers themselves (APNs, FCM, SMS, email gateways) are the downstream sink. Around this spine sit three supporting stores that the simplified sim folds away but a real system needs: a preferences/opt-out store (cached), a dedup store (Redis with TTL), and a delivery-status store plus a dead-letter queue for messages that exhaust their retries.

The reason for the queue is the single most important idea in the problem and the highest-scoring rubric answer: it decouples acceptance from delivery. If you sent synchronously inside the request that triggered the event, the news service's publish call would block on millions of provider round-trips and time out. With a queue, the producer's write is an O(1) enqueue, the spike is absorbed as queue depth, and workers drain it at whatever rate they and the providers can sustain. Throughput and producer latency become independent.

The reason for a separate worker fleet, rather than doing the work in the gateway, is failure isolation and independent scaling. Provider calls are slow and unreliable; you want the part that talks to flaky third parties to be horizontally scalable, individually killable, and restartable without touching the acceptance path. In the simulator the gateway is provisioned at 30,000 per second per replica times 2 replicas (60,000 total) precisely so acceptance is never the constraint, while the workers are the part that gets killed by chaos and must be sized for survival.

The simulator's reference topology makes the sizing explicit: Notify API gateway at 60,000 per second, send queue draining at 30,000 per second with an 80,000-deep buffer, six channel-sender workers at 3,500 per second each (21,000 aggregate), and providers modeled at 30,000 per second. Notice that every stage except the workers is provisioned well above the 18,900 peak, so the worker fleet is deliberately the tightest link, and the engine confirms it: the senders are the hottest node on every single scored tick. That is the whole lesson of the design encoded in capacity numbers.

• Notify API (gateway): auth, preference/opt-out filtering, dedup, enqueue, instant ack. Sized at 30,000/sec x 2 replicas = 60,000/sec so it never blocks producers.
• Send queue: durable buffer that absorbs fan-out; drains at 30,000/sec with an 80,000 buffer. Converts spikes into depth, not back-pressure on producers.
• Channel senders (workers): per-channel pools that rate-limit, retry, and call providers; 6 x 3,500/sec = 21,000/sec, the deliberately tight link and the hottest node in the run.
• Providers (sink): APNs/FCM/SMS/email, modeled at 30,000/sec; plus side stores for preferences, dedup, status, and a dead-letter queue.

Deep dive 1: fan-out and a worker fleet that survives losing a replica

The central hard problem is sizing the worker fleet so that it sustains peak load even while pieces of it are dying. The simulator dramatizes this: it kills a sender at 12 seconds and another at 28 seconds, each kill lasting 6 seconds, and each kill removes one replica from the fleet (effective capacity is alive-replicas times per-replica capacity). The second kill lands squarely during the 18,000-per-second breaking-news stage, which is the worst possible moment.

Do the arithmetic the way the engine does. Peak offered load is about 18,900 per second after jitter. If you size the fleet at exactly peak, one death drops you below it and you start shedding load. The correct rule is to size so that the fleet still covers peak after losing a replica: choose N workers such that (N minus 1) times per-replica capacity is at least peak. With 3,500 per second per worker, you need (N-1) x 3,500 >= 18,900, so N-1 >= 5.4, so N = 7 to keep full headroom. The reference ships 6 workers, which gives 21,000 aggregate; that comfortably covers 18,900 when all are alive, and even with one dead (5 x 3,500 = 17,500) it only sheds about 500 per second, roughly 2.8 percent, and only during the final 6-second window. That is why availability still lands at 99.4 percent, well above the 97 percent target.

Here is the subtle, score-revealing detail. Six workers passes availability handily but runs the fleet hot: during the second kill the workers sit at 18,000 over 17,500, about 103 percent utilization, and with jitter the peak sample reaches 18,900 over 17,500, about 108 percent. The saturation SLO grades the 90th-percentile sample, which the engine measures at 99.7 percent, just over the 90 percent target, so six workers earns only partial credit there (a saturation sub-score near 0.78). The run still scores 97 — a clean S — because availability, throughput and cost all pass outright. Bumping to seven workers (24,500 aggregate; 21,000 even with one dead) holds the jittered peak to 18,900 over 21,000 = 90 percent and drops the scored saturation to about 83 percent, clearing the target outright and lifting the run to a perfect 100, still at only 11 instances (under the cost ceiling of 12). That seventh replica is the difference between a 97 and a perfect 100 — both grade S — and articulating that trade-off out loud is exactly what an interviewer is listening for.

Note what the queue does and does not do here. The queue drains at 30,000 per second, far above the 18,900 peak, so it is never the bottleneck and its 80,000 buffer stays nearly empty in steady state. Its job is insurance: when a true fan-out burst briefly exceeds drain rate, the durable buffer holds the overflow so it is delivered late rather than dropped. In the simplified simulator the queue sits in front of the workers and does not exert back-pressure on them, so worker overflow during a kill is counted as dropped; in a real system the durable log converts that drop into delay, which is precisely why at-least-once plus a deep durable queue is the right backbone for transactional traffic.

• Chaos: a worker is killed at t=12 and again at t=28 (6 seconds each); each kill removes one replica.
• Survival rule: pick N so (N-1) x per-worker-capacity >= peak. At 3,500/worker and about 18,900 peak, that is N=7 for full headroom.
• Availability vs saturation: 6 workers passes availability (99.4%) but runs hot — scored saturation 99.7% (instantaneous 103-108% during the kill) — so saturation only earns partial credit, yet the run still scores 97/S; 7 workers drops scored saturation to about 83% (peak 90%) for a perfect 100 at cost 11 (under 12).
• Queue role: drain capacity above peak so it never bottlenecks; deep buffer turns transient overflow into delay, not loss.

Deep dive 2: surviving flaky, throttling providers without making it worse

The second hard problem is that your dependencies are hostile. The simulator throttles a provider at 20 seconds (an 8x latency spike for 6 seconds). The rubric offers a deliberate trap: retrying instantly in a tight loop until it works. That answer is marked wrong because hammering a throttled provider deepens the throttle and turns a brief slowdown into a self-inflicted outage (a retry storm). The correct toolkit is four things working together: exponential backoff with jitter, per-provider rate limiting, a dead-letter queue, and a circuit breaker.

Per-provider rate limiting is the foundation. Each provider has a known ceiling, and SMS providers especially are strict, so each channel-sender pool meters its outbound calls to stay under that ceiling using a token bucket. This is also why the channels are separate pools: a marketing email blast must not consume the SMS rate budget that a two-factor code needs. When a provider returns a 429 or times out, the worker does not retry immediately; it backs off exponentially (for example 1, 2, 4, 8 seconds) with random jitter so that thousands of workers do not retry in lockstep and create a synchronized thundering herd. A circuit breaker watches the error rate per provider and, once it trips, stops sending to that provider for a cool-down window, optionally failing over to a secondary provider, so one sick dependency cannot tie up the whole worker fleet in slow, doomed calls.

Messages that exhaust their retry budget go to a dead-letter queue rather than being dropped silently or retried forever. The DLQ is both a safety net (you can replay it once the provider recovers) and an observability signal (a growing DLQ is your alarm that a provider is down). This is the difference between losing messages and merely delaying them.

Retries force the idempotency conversation, which closes the loop with dedup. Because you retry, the same logical notification may be handed to a provider more than once, and because the gateway already deduplicates on a per-user, per-event key, you must carry that key all the way through. Each send writes its dedup key to Redis with a TTL before dispatch; a retry or a duplicate event checks the key first and short-circuits if it is already delivered. That is how at-least-once delivery stays safe: the system may try many times, but the recipient sees the message once. Honoring preferences, opt-outs and quiet hours at the gateway, before enqueue, is the same idea applied upstream: filtering ineligible recipients out at acceptance is both a correctness guarantee (no spam) and a capacity win (less downstream load).

• Trap to avoid: tight-loop instant retries deepen throttling into a retry storm and a self-inflicted outage.
• Backoff with jitter so thousands of workers do not retry in lockstep.
• Per-provider token-bucket rate limits, with separate pools per channel so marketing cannot starve transactional SMS.
• Circuit breaker per provider with cool-down and optional failover; dead-letter queue for exhausted retries (replayable, and an alarm signal).
• Idempotency: carry a per-user/per-event dedup key end-to-end with a TTL so at-least-once retries never become duplicates.

Bottlenecks under load and how to scale

The fastest way to internalize the design is to diagnose why the simulator's under-tuned starter fails. The starter ships a queue that drains at only 8,000 per second and a worker fleet of three replicas at 1,500 each (4,500 aggregate). Both are below even the 10,000-per-second baseline, so the system is broken before any spike or chaos arrives. The workers are the true floor: they serve only about 4,500 per second and drop everything above that, while the queue's 30,000-deep buffer backs up (8,000 drain against 10,000-plus offered) and overflows within the first dozen seconds once the marketing blast hits, compounding the loss. Availability collapses to about 27 percent and throughput sits near 4,500, failing the 97 percent and 10,000 targets badly. Saturation is hopeless too: the workers run at about 180 percent of capacity in steady state and around 270 percent when a sender is killed (8,000 forwarded into 3,000 of surviving capacity), so the scored peak saturation lands near 270 percent versus the 90 percent target. The one thing the starter passes is cost (footprint 7) — it is cheap because it is under-built — and it scores an F (14 out of 100).

Fixing it is a two-axis move on the worker fleet: bigger instances and more of them. Per-replica capacity goes from 1,500 to 3,500 and replicas from 3 to 6, taking aggregate worker capacity from 4,500 to 21,000, a 4.7x increase. The queue's drain rate goes from 8,000 to 30,000 so it sits above peak and stops being a bottleneck, and its buffer grows from 30,000 to 80,000 for burst insurance. The gateway is doubled to 60,000 so acceptance has ample headroom. Each change targets a specific binding constraint, and you should name the constraint before you touch the knob.

Under sustained growth, the worker fleet scales horizontally and linearly: it is stateless, it pulls from the queue, so you add replicas (or autoscale on queue depth, which is the right signal). The queue scales by partitioning, for example Kafka partitions, with consumer count bounded by partition count, so plan partitions for your target consumer parallelism up front. The provider boundary is the real-world ceiling that compute cannot fix: you scale it by using batch and multicast APIs (one APNs/FCM call for many tokens), by sharding across multiple provider accounts or SMS number pools, and by failing over to secondary providers. The supporting stores scale independently: shard Redis for dedup, cache preferences so the lookup never touches the database on the hot path.

Finally, protect the important traffic from the unimportant. The simulator's marketing blast at 10 seconds and breaking-news spike at 20 seconds are the same shape, but in production a marketing campaign must never delay an OTP. The standard answer is priority lanes: separate queues (or separate consumer groups) for transactional versus bulk traffic, with the transactional lane getting guaranteed worker capacity and the bulk lane getting whatever is left. A deep buffer that is a feature for marketing (drain over minutes) is a bug for a login code (must arrive in seconds), and separate lanes let you tune each correctly.

• Starter failure: queue drains 8,000 and workers do 4,500 aggregate, both below baseline; availability about 27%, throughput about 4,500, saturation about 270%. Only cost passes — an F (14).
• Fix the binding constraints by name: workers 4,500 -> 21,000 (bigger + more), queue drain 8,000 -> 30,000, buffer 30,000 -> 80,000, gateway -> 60,000.
• Scale levers: autoscale workers on queue depth; partition the queue; batch/multicast and shard provider accounts at the provider ceiling; shard Redis and cache preferences.
• Priority lanes: separate transactional and bulk queues so a marketing blast never delays an OTP.

Key trade-offs and how this scores in the gym

At-least-once versus exactly-once is the defining trade-off. Exactly-once across a third party is unattainable, so you choose at-least-once and pay for it with idempotency and a dedup store. The cost is extra state (Redis keys with TTL) and the discipline of carrying a key end-to-end; the benefit is that you can retry freely against flaky providers without spamming users. Say this explicitly in an interview, because choosing at-least-once and then explaining how dedup makes it safe is the senior-level answer.

Latency versus buffering is the second trade-off. A deep durable queue absorbs enormous spikes and guarantees no loss, but depth is delay: a message can sit in the buffer for minutes during a fan-out. That is perfect for marketing and unacceptable for a verification code, which is the entire justification for priority lanes. Cost versus headroom is the third: the simulator's cost ceiling of 12 instances deliberately punishes the lazy answer of throwing 24 workers at the problem (that alone is a footprint of 28). You must right-size. Six workers is the lean answer that scores an S at footprint 10; seven also clears saturation for a perfect 100 at footprint 11. The reference design lands at footprint 10 (gateway 2, queue 1, workers 6, providers 1) and scores 97/S because the math is tight, not because it is over-built.

Concretely, here is how the reference scores against the four SLOs. Availability lands at 99.4 percent, comfortably above the 97 percent target (the only drops are about 500 per second during the final 6-second worker kill), earning full credit on the heaviest-weighted objective. Throughput, the median sustained served rate, sits around 15,500 per second, well above the 10,000 floor. Saturation is the weakest link at 99.7 percent (the 90th-percentile sample) versus the 90 percent target, so it earns only partial credit — and it is the one knob (add a seventh worker) that converts a 97 into a perfect 100, both grade S. Cost at 10 is under the 12 ceiling. Weighted by 3, 2, 1, 1, the run scores 97 — an S — and understanding why each number falls where it does is the difference between memorizing a diagram and actually designing the system.

• At-least-once + dedup beats unattainable exactly-once: pay in state, gain safe retries.
• Buffering trades latency for durability; priority lanes resolve the conflict between bulk and transactional.
• Cost ceiling forces right-sizing: 6 workers scores S at footprint 10, 7 also clears saturation for a perfect 100 at footprint 11, both under the cap of 12.
• Reference scorecard: availability 99.4% (target 97%), throughput about 15.5k (target 10k), saturation 99.7% (target 90%, partial), cost 10 (cap 12) — scores 97, an S run.