Design a payment system

In 2 minutes

I would design for correctness first, since the volume is modest at about 120 payments a second. The checkout sends the amount, a card token and an idempotency key to the payment API. The API stores the key, so a repeat returns the first result and no one is charged twice. The payment service saves the payment in a created state before calling the provider, and passes the same key on, so a crash always leaves something to follow up. Each payment is a state machine with conditional updates. If a call times out, it stays pending, and a job asks the provider for the real status. Cards are tokenised by the provider, so card numbers never reach us. Results that arrive later come through signed webhooks, which we verify, deduplicate and apply only if they move the payment forward. Money is recorded in an append-only double-entry ledger written in the same transaction as the state change, with debits equal to credits and corrections made by new entries. A daily job reconciles the ledger with the provider's settlement report and flags every difference.

The question

Clarifying questions

• Do we talk to a payment provider, or move money between banks ourselves? Using a provider keeps card data and bank connections out of our systems, and makes our own job one of bookkeeping and coordination.
• Which payment methods and currencies must we support? Methods with extra steps, such as bank redirects or customer authentication, make some payments finish later and asynchronously.
• Do we charge in one step, or authorise first and capture later? Authorise-then-capture adds states, and an authorisation can expire if it is not captured in time.
• Do we need refunds, partial refunds and payouts to sellers? Each is another movement of money, so each needs its own idempotency and its own ledger entries.
• What audit and reporting duties apply? Auditors expect that every cent can be traced and that history is never rewritten.

Requirements

Functional

• Charge a customer's payment method for an order, once.
• Track every payment through its states until it is final.
• Refund a payment in full or in part.
• Record every movement of money and reconcile it with the provider's reports.

Non-functional

• A payment is charged at most once, however many times the request is repeated.
• No money is ever unaccounted for: every payment matches a balanced ledger entry.
• Payment API: p99 under 3 seconds including the provider call.
• Raw card numbers never touch our servers, and the system stays available at 99.99%.

Back-of-the-envelope estimates

Assumptions for this exercise:

• These numbers are assumptions for the exercise, not facts about any real business.
• 10 million payments a day, with a peak 10 times the average. Each payment writes 4 ledger entries of about 200 bytes.
• The provider sends about 3 webhook events per payment. Idempotency records are kept 24 hours at about 100 bytes each.
• About 1 in 1,000 provider calls times out with an unknown result.

The arithmetic:

• Payments per second: 10M a day ÷ 86,400 s ≈ 116 → ≈ 120/s on average, about 1,200/s at a 10x peak
• Ledger data: 10M × 4 entries = 40M a day × 200 bytes = 8 GB; × 365 → 8 GB a day, ≈ 2.9 TB a year
• Webhook events per second: 10M × 3 = 30M a day ÷ 86,400 s ≈ 347 → ≈ 350/s on average, about 3,500/s at a 10x peak
• Idempotency store: 10M keys a day × 100 bytes, kept 24 hours → ≈ 1 GB
• Unknown outcomes to resolve: 10M payments × 1 in 1,000 → ≈ 10,000 a day, which must be settled automatically

High-level design

The shop's Checkout sends a payment request to the Payment API. The request carries an amount, a currency, a card token instead of the card number, and an idempotency key. The API looks the key up in the Idempotency store. A repeat gets the first answer back, and a new key goes on to the Payment service.

The Payment service keeps the payment as a small state machine in the Payments DB. It saves the payment before it calls the Payment provider, so a crash always leaves a record to follow up. When the provider confirms, the service writes balanced entries to the Ledger.

Some results arrive later. The provider sends signed events to the Webhook receiver, which updates the payment. A daily Settlement check compares the provider's settlement report with the ledger, to catch anything that slipped through.

How a request flows

Pay for an order

1. The checkout sends the amount, the currency, a card token and a unique idempotency key.
2. The API records the key as in progress. A repeated key would return the stored result instead.
3. A new key goes on to the Payment service.
4. The service saves the payment as created before calling anyone, so a crash leaves a record.
5. It calls the provider with the token and the same idempotency key, and the payment becomes pending.
6. The provider answers that the payment succeeded.
7. In one transaction the service marks the payment succeeded and writes balanced ledger entries.
8. The API stores the result under the key and replies to the checkout. A retry gets this same reply.

Confirm by webhook

1. Later the provider sends a signed event saying the payment succeeded. It may send the same event twice.
2. The receiver checks the signature, then moves the payment to succeeded, only if it is still pending.
3. One row changed, so this event was new. A repeat would change none and be ignored.
4. The receiver answers 200 quickly, so the provider stops retrying.

Reconcile with the provider

1. Each day the provider publishes a settlement report listing every transaction and payout.
2. The job matches each report line to ledger entries by the provider's transaction ID.
3. The ledger returns what it holds. Lines with no entry, extra entries and different amounts are flagged.
4. Confirmed differences are posted as new adjusting entries. Old entries are never edited.

Deep dive: idempotency keys and the payment state machine

Networks fail in the middle. A checkout sends a payment, the connection drops, and it cannot tell whether the charge happened. If it retries blindly, the customer may pay twice. The fix is an idempotency key: a unique value the client creates for one intended payment, such as one tied to the order, and sends on every attempt.

The API stores the key with a hash of the request and, once known, the response, with a unique constraint on the key. Then:

• Same key, same request, finished: return the stored response without doing anything.
• Same key, same request, still running: answer 409 or wait, never start a second one.
• Same key, different request: reject it, because the client has a bug.

Pass the same key to the provider, which also remembers it, so even a retry from our side cannot charge twice. Keep keys for at least 24 hours.

A payment is a state machine: created, pending, authorised or succeeded, then perhaps refunded, with failed and cancelled as other ends. Each change is a conditional update that names the state it expects, so concurrent attempts cannot both apply. Always save the payment as created before calling the provider. If the process dies after the call, the record exists and a recovery job can resolve it.

When a call times out, the outcome is unknown. Never guess, and never retry with a new key. Leave the payment pending, and have a job ask the provider for the status using the original key or payment ID. Store amounts as whole minor units, like cents, with a currency code, never as floating-point numbers.

Deep dive: the provider: tokens, webhooks and retries

Tokenisation. The customer types card details into a form hosted by the provider, or into the provider's mobile or web component. The provider returns a token, a stand-in value that is useless to anyone else. Our servers only ever handle the token. This keeps raw card numbers out of our systems, which greatly shrinks our PCI DSS scope, the security rules for handling card data. Never log request bodies that could hold card data.

Many payments finish later. A bank may ask the customer to authenticate in a separate step, and some methods take hours or days to confirm. So a call can return 'requires action' or 'processing', and the final result comes by webhook, a message the provider sends to our endpoint.

Treat webhooks as untrusted, repeated and unordered.

• Verify the signature with the shared secret, and reject old timestamps to block replays.
• Deduplicate by event ID, and apply an event only if it moves the payment forward. A late 'pending' must not undo 'succeeded'.
• Answer fast. Return 200 after saving the event, then process it from a queue, or the provider will retry.
• Fetch when in doubt. If an event looks odd, ask the provider's API for the current state. It is the source of truth.

Retry provider calls on timeouts and 5xx errors, with backoff and jitter, always using the same idempotency key. Stop calling a provider that is failing hard, using a circuit breaker, so your threads do not pile up. If you have a second provider, route new payments to it while the first recovers, but a payment that started with one provider finishes with it.

Deep dive: the double-entry ledger and reconciliation

A payment's status is not enough as an accounting record. Keep a ledger: an append-only list of entries, each moving an amount from one account to another. In double-entry bookkeeping, every transaction has at least two entries, and the debits and credits add up to the same amount. For example, a 40.00 payment with a 1.46 fee could be:

• Debit the provider clearing account 4,000 (in cents).
• Credit the seller's balance 3,854.
• Credit fee revenue 146.

Debits 4,000 equal credits 3,854 + 146 = 4,000. Money cannot appear from nowhere: if the sums differ, something is wrong, and a simple daily check can find it. A refund is not an edit. It is a new transaction with opposite entries.

Rules that keep the ledger honest:

• Entries are never updated or deleted. A mistake is fixed by adding a correcting transaction.
• Write the payment's state change and its ledger entries in one database transaction, so they succeed or fail together.
• Add a unique constraint on payment ID and entry type, so a retried write cannot post twice.
• A balance is the sum of an account's entries. Keep a cached running total for speed, and verify it against the sum regularly.

Reconciliation. Our records and the provider's can still disagree: a lost webhook, a payment marked failed that actually succeeded, a fee that differs. Each day, fetch the provider's settlement report and match its lines to ledger entries by transaction ID. Sort the results into matched, missing on our side, missing on theirs, and amount mismatches. Alert on anything unmatched, and track each one until it is resolved. Finally, compare the total payout with the bank statement.

Trade-offs

• Duplicate protection: An idempotency key on our API and on the provider call over Hope that clients never retry. A repeated request can never charge twice. The cost is a store of keys and strict rules about reusing them.
• Learning the result: Webhooks, with a status check as a fallback over Wait for the provider's synchronous reply only. Slow or authenticated payments complete correctly, and lost replies are repaired. The cost is an extra endpoint and a recovery job.
• Recording money: An append-only double-entry ledger over A balance column that is updated in place. Every cent has a trail and errors show up as imbalances. The cost is more rows and the discipline of correcting by new entries.
• Card data: Tokenisation by the provider over Store cards ourselves. Card numbers never reach our systems, which cuts the security burden. The cost is dependence on the provider's token format and limits.

Common follow-up questions

How do refunds and partial refunds work?
A refund is its own object with its own idempotency key, linked to the payment. Before sending it, check in the database that the total refunded stays within the amount captured, using a conditional update. Post reversing ledger entries when it succeeds.

What if the database commit succeeds but the response is lost?
The client retries with the same idempotency key. The API finds the stored result and returns it, with no second charge. This is the main reason the key exists.

How do you handle several currencies?
Store each amount with its currency and keep separate ledger accounts per currency. A conversion is an explicit transaction with entries on both sides and a recorded rate. Never add amounts in different currencies together.

How would you scale it?
Shard the payments by merchant or payment ID. A few ledger accounts, such as the fee account, receive an entry from every payment and become hot, so split them into sub-accounts and add them up when you need the total.