Ad Click Aggregator

Clarify the requirements

Functionally the system does three things: ingest click events at scale, aggregate counts by ad / campaign / time window, and serve both near-real-time and historical queries for dashboards. That is the whole feature surface — keep it tight.

The non-functional requirements are where the difficulty lives, and they are exactly what the simulator grades. No event may be lost, because clicks map to money (availability target ≥ 98%, the heaviest-weighted SLO). The pipeline must sustain the full ingest rate (≥ 25,000 events/sec) and absorb a sudden ~2× spike without dropping. Aggregation should be effectively exactly-once, so a retried or replayed event isn't double-counted. And it should do all this with a lean fleet, not by brute-forcing capacity everywhere.

• Functional: ingest clicks, aggregate by ad/campaign/window, serve real-time + historical queries.
• Graded: no clicks lost (availability ≥ 98%, weight 3), sustain ≥ 25k events/sec (weight 3), keep headroom (weight 1), lean fleet (weight 1).
• Accuracy over freshness: a few seconds of aggregation lag is fine; a lost or double-counted click is not.

Back-of-the-envelope

Anchor on 25,000 click events per second at steady state, climbing toward roughly 45,000/sec at the viral peak. That is about 2.2 billion events per day. Each click event is small — ad id, campaign id, user/device, timestamp, and a little context, call it ~200 bytes — so raw ingest is on the order of 5 MB/sec (~40 Mbps) and a few hundred GB/day of raw events before aggregation.

The key insight is the compression that aggregation buys you. Dashboards don't need 2.2 billion rows; they need counts per ad per minute. Rolling up to per-minute windows collapses the firehose into a tiny fraction of its volume, which is why the analytics store can be modest even though ingest is huge. Storage and query cost live on the aggregated side; throughput and durability pressure live on the ingest side.

• ~25k/sec steady, ~45k/sec peak → ~2.2B events/day; ~200 B/event → ~5 MB/sec raw.
• Per-minute rollups shrink billions of events into thousands of counts — the OLAP store stays small.
• The pressure is throughput + durability on ingest, not storage on the query side.

High-level architecture

The reference pipeline is a classic streaming shape: a stateless Ingest API (gateway, replicated) accepts and validates events and writes them to a durable event stream (Kafka-style log). A fleet of aggregator workers consumes the stream, rolls events up into time-windowed counts, and writes those rollups to a columnar OLAP store that dashboards query. The simulator requires all five roles — client, gateway, stream, worker, db.

The event stream is the heart of the design and the reason nothing gets lost. Producers (the Ingest API) and consumers (the aggregators) are fully decoupled: when aggregators die or slow down, the stream keeps accepting events and simply buffers them, and when the workers recover they resume from their last committed offset and catch up. The stream turns a downstream failure into latency instead of data loss — which is precisely the failure the gym injects when it kills aggregators mid-run.

• Ingest API (gateway): validate and publish events; stateless, replicated.
• Event stream (Kafka): durable buffer that decouples ingest from processing and enables replay.
• Aggregator workers: windowed stream processors that roll up counts; scale by adding consumers.
• OLAP store: columnar analytics DB holding the small aggregated rollups for dashboards.

Deep dive: don't lose events, don't double-count

Durability comes from the stream. As long as the Ingest API successfully appends an event to the log (which is replicated across brokers), the event survives any number of aggregator crashes — they just replay from the log. So the availability SLO is really 'size the stream's buffer for the spike and make sure aggregators can drain it', not 'keep every worker alive'.

Exactly-once aggregation is the subtle part, because stream delivery is at-least-once: a worker that crashes after processing but before committing its offset will reprocess events on restart. Solve it by making aggregation idempotent — commit the windowed count and the consumed offset atomically, or key counts by (window, ad, dedup-key) so replays overwrite rather than add. Then a replayed event lands in the same bucket instead of inflating it. This is why the spec calls for exactly-once aggregation: at-least-once delivery plus non-idempotent counting equals overbilling.

Bottlenecks and scaling under load

Two pressure points show up under the scenario. First, the aggregators: when one is killed and load climbs, the survivors must absorb both the redistributed partitions and the higher rate, or the stream backlog grows without bound. The fix is headroom — run enough aggregator replicas that losing one or two still leaves capacity to drain the buffer (the reference design roughly doubles the worker fleet versus the naive starter). Second, the OLAP store: when it slows under a latency spike, writes back up into the workers; batching rollups and writing aggregated counts (not raw events) keeps the write rate low enough to ride out the blip.

The stream's buffer is your shock absorber. Size it for the peak so a transient consumer slowdown is soaked up rather than dropped, and the availability SLO holds even through the kills and the latency spike.

Trade-offs

The big trade-off is latency versus accuracy and durability, and here we choose accuracy. A synchronous 'count it in the request path' design would be simpler and fresher but would drop events the instant a downstream hiccuped — unacceptable when clicks are money. The streaming pipeline adds a few seconds of aggregation lag in exchange for never losing an event and surviving worker failure, which is the right call for billing data.

A second trade-off is real-time versus historical serving. The same rollups can feed a fast 'last few minutes' view and a historical store; many real systems run a fast approximate path alongside an exact batch reconciliation. For this problem, exact windowed aggregation into an OLAP store covers both near-real-time and historical queries from one pipeline.