Distributed Rate Limiter

A rate limiter is the one component that touches every single request before any useful work happens. That position is what makes it both critical and dangerous: if it is slow, the whole API is slow; if it is down, the whole API can be down; if it is wrong, you either let an abusive botnet through or you throttle your paying customers. The job sounds trivial — count requests per key, reject past a threshold — but the moment you put it in front of a fleet of stateless servers, the counting has to be shared, atomic, and microsecond-cheap all at once. This guide walks from a blank requirements sheet to the exact reference architecture the SysGym simulator grades you against: a token-bucket limiter backed by a central atomic counter store, sitting in front of a generously replicated API, designed to fail open. The scenario throws 8,000 requests per second at you, doubles it to 16,000 during an abuse surge, settles to 1.6x sustained pressure, then kills API replicas at 12s and 28s and stalls the API tier mid-surge at 20s. The targets are availability at or above 98.5 percent, hot-path p99 at or below 85 ms, sustained throughput at or above 8,000 rps, and peak saturation at or below 90 percent. We will derive each design choice and show the math that clears those bars.