Designing a Payment System
High Level Design·advanced·~22 min read
- system-design
- payments
- distributed-systems
- consistency
- hld
What you'll learn
1. Problem Statement and Requirements
What Are We Designing?
A payment processing system that handles real-time financial transactions between merchants and customers. The system must guarantee absolute data integrity -- no money should be lost, duplicated, or misattributed. Every transaction must…
Functional Requirements
- Initiate payment transactions (charge, refund, void) - Process card payments through external payment networks - Track payment status in real time (CREATED, PENDING, PROCESSING, SUCCESS, FAILED) - Support idempotent retry semantics for…
Non-Functional Requirements
- Throughput: Handle 10,000 transactions per second at peak - Consistency: Zero data loss -- every cent must be accounted for - Compliance: PCI DSS Level 1 compliant -- raw card numbers never stored - Auditability: Complete, immutable au…
Transition Rules
1. Atomic: Each transition is a single database transaction (compare-and-swap on current state) 2. Logged: Every transition creates an immutable audit log entry BEFORE the state is updated 3. Guarded: Invalid transitions are rejected (e.…
Database Implementation
This compare-and-swap pattern ensures that even under concurrent requests, only one transition succeeds. The losing request gets zero affected rows and must retry or fail gracefully.
3. Idempotency -- Preventing Double Charges
The Problem
Network failures are inevitable. When a client sends a payment request and receives a timeout, they cannot know if the payment was processed. They must retry -- but a naive retry could charge the customer twice.
Solution: Idempotency Keys
Every payment request includes a client-generated idempotency key. The server guarantees that processing the same key multiple times produces the same result.
Implementation
Key Generation and Storage
The hash of (merchantid + idempotencykey) serves as the deduplication key. This ensures: - Different merchants can use the same idempotency key independently - Keys are fixed-length for efficient storage and lookup - Lookups are O(1) in…
Edge Cases
| Scenario | Behavior | |----------|----------| | Same key, same payload | Return cached result (HTTP 200) | | Same key, different payload | Reject with HTTP 422 (conflict) | | Key expired (TTL passed) | Treat as new request | | In-progr…
4. PCI Compliance and Tokenization
Why Tokenization?
PCI DSS (Payment Card Industry Data Security Standard) mandates strict controls over cardholder data. Storing raw card numbers exposes the system to massive liability and compliance costs. The solution: never let raw card data touch your…
Client-Side Tokenization Flow
Security Layers
| Layer | Protection | |-------|-----------| | Tokenization | Card numbers never reach your servers | | TLS 1.3 | Encryption in transit for all communications | | AES-256 | Encryption at rest for stored tokens and metadata | | Access con…
What You Store vs. What You Never Store
| Stored (encrypted) | Never Stored | |-------------------|--------------| | Payment token (tokxxx) | Full card number (PAN) | | Last 4 digits (for display) | CVV/CVC | | Card brand (Visa, Mastercard) | Magnetic stripe data | | Expiratio…
5. End-to-End Payment Flow
Complete Request Lifecycle
Step-by-Step Flow
1. Merchant calls POST /v1/payments with amount, currency, payment token, and idempotency key 2. API Gateway authenticates the merchant (API key + HMAC signature), terminates TLS 3. Payment Service checks idempotency key in Redis - If fo…
Why Kafka Between Steps?
Kafka decouples the synchronous API response from the asynchronous network processing: - Merchant gets a quick acknowledgment (PENDING) without waiting for network roundtrip - If a processor worker crashes, the message remains in Kafka f…
6. Webhook Delivery System
Requirements
Merchants need real-time notifications about payment status changes. The webhook system must be: - Reliable: Every event delivered at least once - Authentic: Merchants can verify messages came from us - Ordered: Events arrive in state-tr…
Delivery Architecture
HMAC Signature Verification
Every webhook includes a signature header so merchants can verify authenticity:
Retry Strategy
| Attempt | Delay | Total Elapsed | |---------|-------|---------------| | 1 | Immediate | 0 | | 2 | 1 minute | 1 min | | 3 | 5 minutes | 6 min | | 4 | 30 minutes | 36 min | | 5 | 2 hours | ~2.5 hours | | 6 | 8 hours | ~10.5 hours | | 7 |…
7. Exactly-Once Processing
The Three Pillars
Achieving exactly-once semantics in a distributed payment system requires combining multiple mechanisms:
How They Work Together
Consider a payment that succeeds but the acknowledgment is lost: 1. Client retries with same idempotency key 2. Idempotency layer checks Redis -- if result cached, return it (done) 3. If Redis key expired: request reaches payment service…
8. Reconciliation
Why Reconciliation Is Critical
External payment networks and your internal system are separate data stores. Discrepancies can arise from: - Network timeouts (you think it failed, network processed it) - Webhook delivery failures (network succeeded, you never got the c…
Reconciliation Process
Types of Discrepancies
| Type | Internal State | External State | Resolution | |------|---------------|----------------|------------| | Stuck pending | PROCESSING (>30 min) | Not found | Query network, update state | | Phantom success | FAILED | Settled/approv…
Reconciliation Schedule
- Real-time: Alert on payments stuck in PROCESSING for >30 minutes - Hourly: Compare recent completions with network status API - Daily: Full reconciliation against settlement files (T+1 or T+2) - Monthly: Aggregate financial reconciliat…
9. Retry Logic and Circuit Breaking
Retry Strategy for External Networks
Not all failures are retryable. The system must distinguish between transient and permanent errors. | Error Type | Retryable? | Action | |-----------|-----------|--------| | Network timeout | Yes | Retry with exponential backoff | | HTTP…
Exponential Backoff Implementation
Circuit Breaker Pattern
When a payment network is down, continuing to send requests wastes resources and increases latency. A circuit breaker stops the bleeding: Configuration for payment networks: - Failure threshold: 5 failures within 60 seconds - Open durati…
10. Event Sourcing and Audit Trail
Append-Only Transaction Log
Every state change is persisted as an immutable event BEFORE the state update is acknowledged. This provides: - Complete audit trail for regulatory compliance - Ability to reconstruct any payment's history - Debugging capability for disp…
Event Schema
Write Pattern
The event is written first so that even if the state update fails, the audit trail captures the attempt. On replay or investigation, the event log is the source of truth.
11. Scaling the Payment System
Horizontal Scaling Architecture
Scaling Decisions
| Component | Strategy | Reasoning | |-----------|----------|-----------| | API servers | Horizontal + auto-scale | Stateless; add/remove based on TPS | | Kafka | Partition by paymentid | Ordering guarantee per payment; parallel across p…
Read Replicas for Status Queries
Merchants polling /v1/payments/{id}/status hit read replicas. The small replication lag is acceptable because: - Webhooks deliver real-time updates - Status polling is a fallback, not primary notification - Lag is typically What breaks n…
Stage 3 -- Bottleneck: idempotency and double-charge prevention at scale
At real scale (thousands of TPS with aggressive client retries), exactly-once semantics is the hardest problem this system has. It's not a hot-key problem or a fan-out problem -- it's a distributed-systems correctness problem where the c…
Cost vs Complexity -- When to Pick What
| Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Compute | Single VM running monolith | K8s deployment, 3-10 API pods | Multi-region K8s, HPA on Kafka lag + CPU |…
Migration Path (MVP → Enterprise)
This is the order I'd actually ship these changes to a live payment system without downtime. The dependencies matter -- doing them out of order creates avoidable outages. 1. Add idempotency keys with DB UNIQUE constraint (day 1, before y…
12. Full System Architecture