Primary concepts and the hard part
Requirements
Transactions: 10,000/sec peak Ledger entries: 2+ per transaction (double-entry) → 20,000 writes/sec Retention: 7+ years, immutable (regulatory)
API / Model
{order_id, amount, currency, payment_method_token, capture: true|false}Two details worth stating unprompted: amounts are integers in minor units (cents), never floats — 0.1 + 0.2 != 0.3 is a real bug that costs real money. And ledger entries are immutable; a correction is a new compensating entry, never an UPDATE.
High-level architecture
Every payment is committed to the Primary DB before anything talks to the processor. From there, an outbox feeds Kafka, a PSP worker calls the PSP, and webhooks carry the result back into the same database.
- Checkout sends
POST /paymentswith anIdempotency-Keythrough the API Gateway, and the Payment Service claims the key with an INSERT on(merchant, key). If the key has been seen and the payment completed, the service returns the stored response and does nothing else. - For a new key, one ACID transaction inserts the payment as
PENDING, its balanced ledger entries and an outbox row, and commits them to the Primary DB together. - The outbox publisher polls for unpublished rows and publishes them to Kafka
payment.events. - The PSP worker consumes the event and calls the PSP to authorize or capture, with timeouts, backoff with jitter and a circuit breaker per PSP. It sends our idempotency key along, and after a timeout it queries the PSP for that key instead of guessing.
- The PSP reports the outcome to the webhook receiver, which verifies the HMAC signature and dedupes on
psp_event_idbefore updating the Primary DB.
Card details never enter this flow. Checkout sends them straight to the PSP, and the payment request carries only payment_method_token. Other consumers of payment.events handle fulfilment, receipts and the warehouse, independently of the PSP worker. Once a day, reconciliation compares the PSP settlement file with the ledger and sends any discrepancy to an exceptions queue.
Same system at 10x. 100k transactions/sec is around the peak the largest wallets report on their biggest shopping day; 100x would be beyond any payment platform's recorded peak, so 10x is the ceiling worth designing for. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Transactions at peak | 10k/sec | 100k/sec |
| Ledger writes at peak | 20k/sec | 200k/sec |
| Ledger entries per day | ~600M | ~6B |
| Databases | 1 primary + standby | ~32 merchant shards, each primary + standby |
| Payment processors | 1–2 | several, chosen per transaction |
What changes, and the number that forces it
- One primary → shards keyed by merchant. 200k ledger writes/sec with synchronous replication is past a single Postgres primary. Shard by
merchant_id, the follow-up's answer, with each shard a primary plus synchronous standby. The idempotency key is already(merchant_id, key), so the claim lands on the same shard as the payment it protects. - Platform accounts are split across shards. This is the trap in sharding a ledger: every payment also moves money into platform accounts (fees, PSP clearing), which would make every transaction cross-shard. Instead, each shard holds its own sub-accounts for fees and clearing, so the payment, both sides of every entry and the outbox row commit in one local ACID transaction. The platform's balance is the sum of its sub-accounts. Moving money between sub-accounts on different shards is rare and scheduled, and runs as a saga with balanced entries on each side.
- Giant merchants get their own shards. On a big sale day, one marketplace can exceed a whole shard's capacity. The router pins the largest merchants to dedicated shards.
- Outbox polling → CDC. Polling for unpublished rows on ~32 shards at this rate is constant load on the primaries. Logical decoding streams each shard's outbox into Kafka with no polling queries at all.
- Several processors, routed per transaction. 100k transactions/sec is too much revenue to hang on one processor's rate limits, pricing and outages. A PSP router chooses a provider for each transaction by card network, region, cost and live authorization success rate, with a circuit breaker per provider. Our idempotency key still travels to whichever PSP is chosen, and a retry after a timeout goes back to the same PSP and queries it first: failing over mid-payment to a different processor is exactly how double charges happen.
- Reconciliation becomes streaming. A daily line-by-line comparison over ~6B entries finds problems a day late. Processor events are matched against the ledger as they arrive, and the daily settlement file now confirms what's already been matched rather than discovering discrepancies.
- Old ledger entries leave Postgres. Seven years of ~6B entries a day doesn't belong on OLTP primaries. Entries older than ~90 days move to write-once columnar storage in hash-chained batches, so tampering is detectable, and balances are served from snapshots plus recent entries.
What stays the same
Consistency over availability. The idempotency claim is still a unique insert, a timeout is still an unknown state resolved by querying, every transaction still sums to zero, sagas still beat two-phase commit, webhooks are still verified and deduped, amounts are still integers in minor units, and card numbers still never touch our servers. Sharding is added exactly where the 10k/sec version said it would be, and not a transaction earlier.
Trade-offs and deep dives
The client sends a key; the server claims it with a unique-constrained INSERT before doing any work. A retry finds the existing row and returns the stored response without re-charging. The unique constraint is what makes this concurrency-safe — two simultaneous retries race on the INSERT and exactly one wins. Also hash the request body and compare: the same key with a different amount is a client bug and should return an error, not silently replay.
You call the PSP and get no response. Three possibilities: the request never arrived, it arrived and succeeded but the response was lost, or it arrived and failed. You cannot distinguish them. Treating a timeout as a failure and retrying blindly is how double-charges happen. The correct handling: pass your idempotency key to the PSP as well (every major processor supports this), so a retry is safe on their side too, and reconcile the unknown state by querying the PSP for that key rather than guessing.
Every transaction writes at least two entries that sum to zero — debit one account, credit another. This isn't accounting ceremony; it's an invariant you can check. If the sum of all ledger entries isn't zero, you have a bug, and you'll find it in seconds rather than discovering it in a quarterly audit. Balances are derived by summing entries (cached and periodically recomputed), never stored as a mutable field that can drift.
During a partition, refusing the payment and returning an error is correct. The user retries; nothing is lost. Accepting the payment on both sides of a partition and reconciling later means double-charging real customers and real regulatory exposure. This is the clearest case in all of system design where availability loses, and stating it confidently is exactly what they want to hear.
Two-phase commit across an inventory service, a payment service, and a shipping service would hold locks across network boundaries, and a coordinator crash strands everything. A saga runs local transactions with compensating actions — refund the charge, release the inventory. You give up isolation (there are observable intermediate states where money is taken but nothing has shipped) in exchange for availability and no distributed locking. Name the trade, don't hide it.
You must update the database and publish an event. Doing both directly is a dual write: either can fail independently. The outbox writes the event row inside the same transaction as the payment, and a separate publisher drains it. Now there is one transaction, atomicity holds, and publishing is at-least-once — hence idempotent consumers downstream. This is the standard answer to "what if the DB write succeeds but Kafka fails?" and it applies well beyond payments.
Authorization places a hold on funds without moving them; capture moves them. Splitting the two lets you verify stock or ship before taking money, and it's the industry norm. Authorizations expire (typically ~7 days), so you need a job to capture or void before expiry — an easy detail to forget and a good one to mention.
Card details go from the browser directly to the PSP's hosted fields or SDK, which returns a token. Your servers only ever see the token. This removes almost your entire codebase from PCI-DSS audit scope. If you accept raw card numbers anywhere, every service that touches them is in scope, and that is enormously expensive.
Every day, fetch the PSP's settlement report and compare it line by line against your ledger. Discrepancies go to an exceptions queue for human review. Distributed systems drift — webhooks get lost, retries land twice, the PSP has its own bugs. Reconciliation is the control that catches what the code missed, and mentioning it unprompted is a strong domain signal.
Verify the HMAC signature (otherwise anyone can mark payments as succeeded). Dedupe on the processor's event ID. Tolerate out-of-order arrival: model payment state as a state machine that ignores transitions that would move backwards, so a late authorized webhook doesn't undo a captured state.
Store minor units as integers with an explicit currency code. Never mix currencies in an arithmetic operation. For multi-currency, record the FX rate used on the transaction itself so the historical record is reproducible.
Possible follow-up questions
merchant_id or account_id so a transaction's entries stay within one shard and remain in a single ACID transaction. Cross-shard transfers become sagas. Avoid sharding as long as possible — 10k/sec doesn't need it.