System design deep dive17 min read

Payment System

A payment system has one property every other system can live without: it has to move money exactly once. If a redirect service drops a request, the user hits refresh. If a payment service drops, double-fires, or half-commits a request, someone is out real money and a regulator eventually wants to know why. The hard part is that the failures you must survive are not exotic. A client retries after a timeout, a bank call hangs for four seconds, a worker process dies mid-charge, the ledger gets slow under contention. All of those happen on a normal Tuesday, and especially during a flash sale. This guide builds the design the way you should build it in an interview: start from the requirements, do the arithmetic, then assemble the architecture piece by piece and justify every capacity number. The target is the same reference the SysGym simulator grades you against: 6,000 checkouts per second spiking to 12,000 during a flash sale while PSP callers die and the ledger slows, with a hard floor of 99 percent of payments completing and zero double-charges. By the end you should be able to derive that design yourself, not memorize it.

Step 1 — Clarify the requirements

Start by separating what the system does from how well it must do it. The functional surface for this problem is narrow and deep: accept and authorize a payment, talk to an external processor (a PSP, meaning a bank or card network), and record every money movement in a ledger that can later be reconciled. Refunds, disputes, payouts, and multi-currency are real but out of scope here; say so explicitly and move on. The interviewer is testing whether you can make money movement correct under failure, not whether you can enumerate features.

The non-functional requirements are where this problem lives, and they are unusually strict. First, exactly-once or idempotent processing: a client that retries a charge after a timeout must never be charged twice. Second, strong consistency on the ledger: balances and entries must be correct the instant they are read, not eventually. Third, high availability and durability: a payment the system accepted must never silently vanish, even if a downstream processor is slow or a worker dies. Latency, notably, is the requirement you are allowed to relax. Users tolerate a checkout that takes a moment; they do not tolerate being charged twice. That single insight reshapes the whole design toward asynchrony.

Pin down the confirmation model before drawing anything, because it determines the entire shape of the system. Ask: does the client need a synchronous yes/no from the bank, or can it accept a 'pending' result and learn the final outcome via a webhook or by polling? For anything at scale the answer must be asynchronous. A synchronous design that holds the user's HTTP request open until the bank replies is the single most common wrong answer for this problem, and we will show with arithmetic why it collapses under load.

Functional: accept and authorize payments; call external PSPs; maintain a double-entry ledger plus reconciliation; expose payment status to the client.
Non-functional: idempotent (exactly-once effects), strongly-consistent ledger, high availability, durability, full auditability.
Explicitly out of scope: disputes, payouts, multi-currency FX, fraud scoring — name them so the interviewer knows you scoped deliberately.
Key clarifying question: synchronous confirmation or async (pending + webhook)? At this scale it must be async.

Step 2 — Back-of-the-envelope estimation

The design point is 6,000 checkouts per second at baseline. A sale event lifts that to 9,000 per second (1.5x), and a flash sale to 12,000 per second (2x). Treat 12,000 per second as the number every tier must survive. Note for honesty: 6,000 per second sustained is about 518 million payments per day (6,000 times 86,400), which is in the same league as the busiest global card networks at their daily peak — so read this as the peak load of a hot region or shard, not a sustained global average. The simulator runs it as a sustained 36-second storm precisely to stress the failure modes.

Now the number that forces the architecture. A payment is not a CPU-bound request; it is dominated by an outbound call to a bank that you do not control. Apply Little's Law: concurrent in-flight calls equals throughput times latency. At 12,000 per second with a healthy 500 ms bank round-trip, that is 6,000 concurrent open calls. When the PSP slows down by 8x (the exact latency spike the simulator injects), each call now takes about 4 seconds, and concurrency balloons to 12,000 times 4, which is 48,000 simultaneous in-flight calls. If each of those is pinned to a user-facing thread or connection, you are dead. This is the quantitative proof that the bank call must be decoupled from the request: you need a buffer and a worker pool, not a synchronous hold.

Write load on the ledger is the other constraint. Double-entry bookkeeping records at least two entries per payment (a debit and a balancing credit), so 6,000 payments per second is at least 12,000 entry writes per second at baseline and 24,000 at peak, all inside strongly-consistent transactions of one commit per payment. Strong consistency caps how fast a single ledger can commit, which is why the ledger, not the gateway, is the tier you watch. Storage is cheaper to reason about: roughly 2 KB of durable record per payment (intent, amounts, status, references, two ledger entries) is about 12 MB/s, or roughly 1 TB/day. Compliance retention of seven-plus years pushes total storage into the multi-petabyte range, so the ledger must be partitioned and cold data archived. Idempotency keys are tiny and short-lived: about 518 million keys/day at roughly 200 bytes with a 24-hour TTL is only about 100 GB of rolling state, which fits comfortably in a fast keyed store.

Traffic: 6,000 rps baseline, 9,000 rps sale (1.5x), 12,000 rps flash sale (2x) — size every tier to 12,000.
PSP concurrency (Little's Law): 12,000 rps x 0.5 s = 6,000 in-flight; at an 8x slowdown (4 s) that is 48,000 in-flight. Async is mandatory.
Ledger writes: at least 2 entries/payment = 12,000/s baseline, 24,000/s peak, one strongly-consistent commit each.
Storage: about 2 KB/payment = roughly 1 TB/day; 7-year retention means multi-PB and partitioning.
Idempotency keys: about 100 GB rolling at a 24-hour TTL — small enough to keep hot.

Step 3 — High-level architecture

The data path is a straight line, and that linearity is intentional: every checkout flows through the same pipeline so correctness is enforced at each stage exactly once. The chain is client to API gateway to payment service to a PSP queue to PSP-caller workers to the ledger, with a reconciliation job and a webhook closing the loop back to the client. There is deliberately no read-cache shortcut anywhere, because unlike a URL shortener there are no cheap reads to hide behind; every request mutates money.

The API gateway authenticates the caller and enforces that an idempotency key is present and well-formed on every request. It is sized far above peak (reference: 18,000 rps per replica times 2 replicas equals 36,000 rps, about 3x the 12,000 peak) because a gateway is cheap and must never be the thing that drops a payment at the front door. The payment service is the brain: it validates the request, performs the idempotency check against a strongly-consistent store, records the payment intent and enqueues the PSP job in one atomic step (the transactional outbox from Deep Dive 2), after which the workers drive the bank call and post the matching debit/credit pair to the ledger. It is sized to peak at 4,000 rps per replica times 3 replicas equals 12,000 rps.

The PSP queue decouples the slow, unreliable bank call from the user's request. It drains at 20,000 rps — comfortably above the 12,000 peak — behind an 80,000-deep buffer, so a momentary burst that arrives faster than the queue can drain is held rather than dropped. Be precise about what the buffer does and does not protect: it cushions load that exceeds the queue's own drain rate, not a worker fleet that can't keep up. The PSP callers are a replicated worker fleet (3,000 rps per replica times 5 replicas equals 15,000 rps) that pull jobs, call the bank with retry and backoff, and advance the payment state machine. They sit downstream of the queue with no buffer of their own, so the real defense against a dying worker is the worker fleet's own headroom, not the queue (see the bottlenecks section). Finally the ledger is a strongly-consistent, double-entry database sized to 18,000 rps, which is 1.5x the 12,000 peak so a contention spike slows commits rather than dropping them. The whole footprint is 12 provisioned instances (2 gateway plus 3 service plus 1 queue plus 5 workers plus 1 ledger), under the cost ceiling of 13.

API gateway: auth + idempotency-key enforcement; 18,000 x 2 = 36,000 rps (about 3x peak headroom).
Payment service: validate, dedup, record intent, enqueue (atomically, via outbox); 4,000 x 3 = 12,000 rps (sized to peak).
PSP queue: decouple the bank call; drain 20,000 rps (above the 12,000 peak), buffer 80,000 jobs as insurance for bursts that exceed the drain rate.
PSP callers (workers): call the bank with retry/backoff; 3,000 x 5 = 15,000 rps (1.25x peak, so a death still covers 12,000).
Ledger (db): double-entry, strongly consistent; 18,000 rps (1.5x peak headroom for contention).
Total footprint: 12 instances, under the cost SLO of 13.

Deep dive 1 — Idempotency and exactly-once

Exactly-once is the headline requirement and also a small lie you have to understand. Across a network you cannot guarantee a message is delivered exactly once; the only honest guarantee is at-least-once delivery. The way you get exactly-once behavior is at-least-once execution plus idempotent effects: make the operation safe to run more than once and converge to the same result. Everything in this section exists to turn an unavoidable retry into a no-op.

The mechanism is an idempotency key: a unique token (typically a client-generated UUID) sent on every charge request, which Stripe exposes as the Idempotency-Key header and keeps for 24 hours. The gateway enforces that the key is present; the real work happens at the payment service. Before doing anything, the service performs an atomic claim on the key: an insert into a strongly-consistent store guarded by a unique constraint (an INSERT ... ON CONFLICT, or a conditional put). The first request to claim the key wins and proceeds. A concurrent or later retry with the same key hits the conflict and, instead of starting a second charge, either returns the in-progress status or blocks until the first attempt records its final result. This atomic claim is what defeats the double-charge race; a fuzzy match on amount plus user plus timestamp does not, because two legitimate identical purchases look the same and a single mistimed retry looks different.

Store the final response keyed by the idempotency key, so a retry arriving after success returns the identical result (same charge id, same amount, same status) without re-executing the charge. A 24-hour TTL is longer than any sane client retry window and keeps the keyed store small (about 100 GB, from the estimate above). Underneath, model each payment as an explicit state machine: created, intent recorded, authorizing, authorized or failed, captured, settled, with refunded and disputed as later branches. Every transition is keyed on (payment_id, current_state) and is itself idempotent. A worker that crashes halfway through authorizing re-reads the state on redelivery and resumes; it never double-authorizes, because the call it makes to the PSP also carries an idempotency token, so even the bank deduplicates. This is the difference between a system that survives retries and one that quietly double-charges during every incident.

Exactly-once = at-least-once delivery + idempotent effects. Pure exactly-once delivery is not achievable over a network.
Client sends a unique idempotency key per charge; the gateway requires it, the service acts on it.
Atomic claim via a unique constraint in a strongly-consistent store wins the race between concurrent retries.
Cache the final response under the key so a post-success retry replays the same result without re-charging.
Model the payment as an idempotent state machine; forward the idempotency token to the PSP so the bank dedupes too.

Deep dive 2 — Ledger, the async PSP path, and reconciliation

Record money as a double-entry ledger, never as a mutable balance counter. Every movement is a balanced pair of entries (a debit and a credit) that sum to zero, and account balances are derived by summing entries, not stored and incremented. This is non-negotiable: an eventually-consistent 'increment a balance column' design is exactly how you lose or duplicate money under concurrency, while a double-entry ledger balances by construction and is auditable line by line. The ledger is the strongly-consistent source of truth, typically a single-leader SQL system or a Spanner-style store, committing one transaction per payment. That strong-consistency requirement is also the throughput ceiling you sized for in Step 2.

The subtle correctness bug here is the dual-write problem. The payment service must do two things when it accepts a charge: record the intent and enqueue the PSP job. If it commits the intent and then crashes before enqueuing, you have a recorded payment that will never be authorized, which is money in limbo. The fix is the transactional outbox: inside the same database transaction that records the intent, write a row to an outbox table. A relay process then reads outbox rows, publishes them to the PSP queue, and marks them sent. Now the enqueue is atomic with the commit and guaranteed at-least-once, and the consumer dedupes on the payment id. This is why you cannot just 'write the DB, then call the queue' and hope both succeed.

Workers pull jobs and call the PSP with bounded retries, exponential backoff, and jitter. On a transient failure they retry; on a permanent decline they mark the intent failed and notify the client. Because the job is idempotent, a redelivered message re-reads the state and skips work already done. But retries alone are not enough, because of the worst case in payments: the bank call times out and you genuinely do not know whether the charge went through. You must never blindly assume failure (the customer may have been charged) or success (they may not have been). That is what reconciliation is for: query the PSP by the idempotency key to learn the true outcome, and nightly, reconcile the processor's settlement file against your ledger and repair any drift. The async result then reaches the client through a webhook, and the client can also poll the payment intent. Asynchronous PSP calls, retry-with-backoff plus reconciliation, and an idempotent state machine are precisely the three things the rubric rewards, and together they are why the system never drops or double-charges.

Double-entry: every movement is two balanced entries; balances are derived, never an incremented counter.
Ledger is strongly consistent (single-leader SQL / Spanner-style), one commit per payment — this is the throughput ceiling.
Transactional outbox makes 'record intent' and 'enqueue PSP job' atomic, closing the dual-write hole.
Workers retry with exponential backoff + jitter, bounded attempts, and are idempotent on redelivery.
Reconciliation resolves the 'don't know' timeout case: query the PSP by idempotency key, and reconcile settlement files nightly.

Bottlenecks under load and how to scale them

The simulator's chaos timeline is a checklist of the four ways this system fails, so design against each. The PSP-caller fleet is the fragile tier because it depends on a slow external service, and the simulator kills a worker at 12 seconds and again at 26 seconds. The defense is over-provisioning the fleet to N+1: the reference runs 5 workers at 3,000 rps each (15,000 total), so losing one at the 12,000 peak still leaves 4 times 3,000 = 12,000 — the live fleet alone exactly covers offered load, and nothing drops. This is the engine's central trap: workers have no buffer of their own, so the instant their live capacity falls below offered load they drop the excess immediately, and the queue cannot save them because it sits upstream and forwards everything it drains. The queue only buffers load that exceeds its own 20,000/s drain rate, which is a different failure mode. So you survive a worker death by sizing the worker fleet itself to cover peak with one replica down, not by leaning on the queue.

The ledger is the classic bottleneck, and the simulator hits it with an 8x latency spike at 18 seconds to mimic lock contention on hot accounts. A latency spike multiplies latency but not capacity, so it does not drop requests; it makes commits slower, which is why the reference gives the ledger 1.5x headroom (18,000 versus 12,000 peak) — at lower utilization the queueing penalty stays small, so a slow blip stays a blip. To scale the ledger for real you keep transactions tiny and short-lived, use append-only entries so there are no update-in-place row locks, and partition by account or tenant so unrelated payments never contend on the same rows. Strong consistency means you buy throughput with partitioning, not by relaxing correctness.

The PSP latency spike at 32 seconds (another 8x) is the scenario Little's Law warned about. Because the bank call is asynchronous, the user already has a 'pending' response, so a slow PSP shows up as deeper in-flight work, not user-facing latency or dropped payments; workers apply timeouts and backoff and keep draining at the same rate. A synchronous hold-open design would instead pile up tens of thousands of open connections and topple. Finally, the flash sale itself (2x to 12,000) is handled by the simple rule that drives all the sizing: every tier should survive peak load with one instance down. The gateway is over-provisioned because it is cheap; the service and worker fleets are sized to peak; the ledger carries 1.5x headroom. The dominant lever is the worker fleet, because availability is weighted most heavily and the workers are what die.

Worker deaths: over-provision N+1 (5 x 3,000 = 15,000 covers 12,000 even down one). Workers have no self-buffer and sit downstream of the queue, so the queue can't cushion a worker shortfall — size the fleet itself to cover peak with one replica down.
Ledger contention: 1.5x headroom, tiny short transactions, append-only entries, partition by account to kill lock contention.
Slow PSP: a latency spike raises in-flight work but not dropped payments (capacity is unchanged); async means the user already got a 'pending', so it never becomes user-facing latency — the whole reason sync fails.
Flash sale: size every tier to survive peak with one instance down; spend the cost budget on the worker fleet first.

Key trade-offs

Synchronous versus asynchronous confirmation is the defining choice. Synchronous gives a simpler client experience but ties a request resource to an uncontrollable bank call; under load, Little's Law turns that into tens of thousands of open connections and the system collapses. Asynchronous via a queue scales, absorbs spikes, and survives slow processors, at the cost of a 'pending' state plus webhook or polling complexity on the client. For money at scale, async wins every time, and the rubric scores the synchronous hold-open answer as wrong.

Strong versus eventual consistency on the ledger is not actually a trade you get to make. Eventually-consistent balances mean lost or duplicated money, so strong consistency is mandatory; what you trade is throughput, which strong consistency caps, and you recover scale by partitioning the ledger rather than weakening it. Relatedly, 'exactly-once' is a property you engineer, not one you buy: you implement at-least-once delivery plus idempotent effects, because exactly-once delivery across a network is impossible.

Cost versus headroom is the trade the simulator makes explicit. The reference runs the service and worker fleets near full utilization at peak to hold the footprint at 12 instances and stay under the cost ceiling of 13 — which is why its saturation score is not perfect: the service tier sits at roughly 100 percent at the 12,000 peak. Spending the last budgeted instance on a fourth service replica would buy that saturation headroom, but with availability weighted 4 and saturation only 1, the right call is to spend on the worker fleet that protects availability before chasing a cleaner saturation number. And the deepest trade-off is the one that ties it all together: aggressive retries improve availability only because idempotency makes them safe. Without the idempotency key, retries are not a resilience feature, they are the mechanism by which you double-charge customers.

Sync vs async: async scales and survives slow banks; sync collapses under Little's Law. Async is the correct answer.
Strong vs eventual ledger: strong consistency is non-negotiable; buy scale with partitioning, not by relaxing it.
Exactly-once is engineered (at-least-once + idempotency), not a delivery guarantee you can purchase.
Cost vs headroom: with availability weighted 4x, spend the budget on workers before chasing a cleaner saturation number.
Retries are safe only because of idempotency — that single key links availability and correctness.

Mapping to the simulator

The gym grades five SLOs, and their weights tell you where to aim. Availability of 99 percent of payments completing carries weight 4 and dominates the score; throughput of at least 6,000 rps sustained carries weight 2; p99 latency under 220 ms, peak saturation under 90 percent, and a footprint of at most 13 instances each carry weight 1; a perfect run is worth 130 points. The generous 220 ms latency target and the heavy availability weight encode the core lesson: payments may take a beat, but they must not be lost. The latency metric is the median of per-tick p99 samples, so a brief contention or PSP spike barely moves it — another reason latency is the relaxable SLO here. Read the 99 percent availability target carefully too: of all checkout attempts during a chaotic flash sale, 99-plus percent complete, and the ones that do not are safe to retry precisely because of the idempotency key, which is how availability and correctness reinforce each other.

The reference design that scores in the A-to-S band is exactly the topology derived above: API gateway 18,000 rps times 2 replicas; payment service 4,000 rps times 3; PSP queue draining 20,000 rps with an 80,000-job buffer; PSP callers 3,000 rps times 5; ledger 18,000 rps; total footprint 12, under the cost ceiling. Every number traces back to the 12,000-per-second peak: gateway at roughly 3x, service at exactly 1x, workers at 1.25x so a death still covers peak, and ledger at 1.5x for contention headroom. It deliberately accepts a near-100-percent service-tier saturation (a small ding on the weight-1 saturation SLO) to keep the footprint lean and bank the heavily weighted availability and cost points.

It is just as instructive to see why the starter topology fails. Its worker fleet is only 1,500 rps times 3, which is 4,500 rps total, below even the 6,000 baseline, and its queue drains at just 8,000 rps. The moment load climbs to 9,000 or 12,000 and a worker dies, the workers cannot keep up, payments drop the instant offered load exceeds live worker capacity, and the heavily weighted availability SLO tanks. The fix is the one this whole guide builds toward: grow the worker fleet so it survives a death at peak, give the queue enough drain rate and buffer to absorb upstream bursts, hand the ledger real headroom, and never, ever couple the bank call to the user's request.

SLOs: availability >= 99% (weight 4), throughput >= 6,000 rps (2), p99 <= 220 ms (1), saturation <= 90% (1), cost <= 13 (1); reward 130.
Reference graph: gateway 18k x 2, service 4k x 3, queue drain 20k / buffer 80k, workers 3k x 5, ledger 18k; footprint 12.
Why the starter fails: workers 1.5k x 3 = 4,500 (below baseline) and queue drain 8k — workers drop the moment load exceeds their live capacity, so availability collapses.
The winning move: size the worker fleet to survive a death at peak, and keep the bank call asynchronous.

Key takeaways

Exactly-once is at-least-once delivery plus idempotent effects: enforce it with a client idempotency key claimed atomically (unique constraint) in a strongly-consistent store, and cache the final response under that key so retries replay instead of re-charging.
Record money in a double-entry ledger with strong consistency, where balances are derived from balanced debit/credit entries, never an incremented counter; eventually-consistent balances are how you lose money.
Never hold the user's request open on a synchronous bank call. Little's Law shows 12,000 rps at a 4-second slowdown means 48,000 in-flight calls; enqueue instead and let replicated workers call the PSP with retry, backoff, and reconciliation.
Close the dual-write hole with a transactional outbox so recording the intent and enqueuing the PSP job commit atomically.
Size the worker fleet to N+1 so a PSP-caller death at peak still covers offered load (reference: 5 x 3,000 = 15,000, down one = 12,000 = peak). Workers have no buffer and sit downstream of the queue, so the queue cannot backstop a worker shortfall — the fleet's own headroom is what saves you.
The ledger is the scaling constraint: keep transactions tiny, append-only, and partitioned by account, and give it about 1.5x headroom (18,000 vs 12,000) so contention slows commits rather than dropping payments.
Availability is weighted 4x in the gym; spend the cost budget on the worker fleet that protects it before chasing a cleaner saturation number, and remember retries are only safe because idempotency makes them so.

Now build it and watch it survive the chaos run.

Build it