Foundations: The Assembly Line — Flow, Caching & Throughput

What you'll learn

• The Building Blocks

• 1. Load Balancer

  A load balancer distributes incoming traffic across multiple servers. Strategies: - Round Robin — Equal distribution across servers cyclically. - Least Connections — Route to the server with the fewest active connections. - Least Respons…

• 3. Database Scaling

  - Vertical Scaling - Bigger machine (limited) - Horizontal Scaling - More machines (sharding) - Read Replicas - Separate read/write workloads - Sharding - Partition data across databases

• 4. CAP Theorem

  In a distributed system, you can only guarantee 2 out of 3: - Consistency - All nodes see the same data - Availability - Every request gets a response - Partition Tolerance - System works despite network failures

• Message Queues and Pub/Sub

• Message Queue (Point-to-Point)

  - Delivery: Exactly-once (or at-least-once with dedup) - Pattern: One producer, one consumer per message - Ordering: FIFO guaranteed (within a partition) - Use case: Task distribution, job processing, action-oriented work

• Pub/Sub (Publish-Subscribe)

  - Delivery: At-least-once to each subscriber - Pattern: One publisher, many subscribers per message - Ordering: Often not guaranteed across subscribers - Use case: Event notification, broadcasting state changes

• Queue vs Pub/Sub Decision

  | Criterion | Message Queue | Pub/Sub | |-----------|--------------|---------| | Consumers per message | One | Many | | Message lifetime | Removed on consume | Retained for TTL | | Semantics | "Do this task" | "This happened" | | Orderin…

• AWS SQS (Simple Queue Service)

  Distributed queuing system with two queue types: | Feature | Standard Queue | FIFO Queue | |---------|---------------|------------| | Throughput | Unlimited | 3,000 msg/sec (with batching) | | Ordering | Best-effort (may reorder) | Stric…

• Request Coalescing (Request Collapsing)

  When many clients simultaneously request the same resource (e.g., cache miss on a popular key), without coalescing, ALL requests hit the backend simultaneously (thundering herd). How it works: Batch multiple identical requests within a t…

• Graceful Degradation

  When a system component fails, degrade functionality rather than failing entirely: | Strategy | Example | |----------|---------| | Feature flags | Disable recommendation engine; show default content | | Fallback responses | Cache serves…

• Concurrency Fundamentals

• Process vs Thread

  | Aspect | Process | Thread | |--------|---------|--------| | Memory | Own address space (isolated) | Shares heap with other threads in same process | | Communication | IPC required (sockets, pipes, shared memory) | Direct shared memory…

• Concurrency vs Parallelism

  - Parallelism: Actually doing multiple things simultaneously (requires multiple CPU cores) - Concurrency: Managing multiple tasks that can make progress (illusion of simultaneity via time-slicing) Single-core machine: concurrency via con…

• Inter-Process Communication (IPC)

  Processes have isolated memory spaces. They communicate via: | Method | Mechanism | Characteristics | |--------|-----------|-----------------| | File / Memory-mapped file | Both processes read/write shared file on disk | Slowest; not rec…

• Thread Anatomy

  Why threads are lighter than processes: Thread creation = allocate a new stack (~1MB in JVM). Process creation = copy entire address space (or use copy-on-write). Thread communication = direct shared heap access (no IPC needed, but requi…

• Thread-Per-Request Model

  Naive approach: spawn a new thread for every incoming request. Problems: - Thread creation is expensive (OS syscall each time) - Each thread consumes memory (~1MB default stack in JVM) - OS imposes hard limits on thread count - Context s…

• Thread Pools

  Instead of creating/destroying threads per request, maintain a pool: Sizing trade-off: Too few threads = requests queue up (high latency). Too many threads = excessive context switching + memory waste. Rule of thumb: - CPU-bound work: po…

• Thread Contention

  When multiple threads compete for shared resources: - Locks/Mutexes — serialize access (safe but limits parallelism) - Lock-free structures — use atomic operations (CAS) — higher throughput but complex - Deadlock — two threads each hold…

• Redis Deep Dive — Data Structures, Persistence, and Production Patterns

  Redis is far more than a simple cache. Understanding its internal data structures and persistence mechanisms enables using it as a primary data store for specific use cases.

• Redis Architecture

  Single-Threaded Event Loop: Redis uses a single main thread to process all commands. This simplifies implementation (no locks needed) and ensures atomicity per command. However, one slow command blocks everything. Async I/O: Despite bein…

• Redis Data Types and Their Internals

  | Data Type | Internal Structure | Key Commands | Production Use Case | |---|---|---|---| | Strings | Simple dynamic string (SDS) | SET, GET, INCR, DECR, SETNX, EXPIRE | Counters, session tokens, rate limit counters | | Lists | Quicklist…

• Sorted Sets — The Most Powerful Redis Structure

  A sorted set stores members with associated scores, maintaining members sorted by score at all times. Internal implementation: Skip List + Hash Map Why skip lists (not balanced trees)? - Simpler to implement correctly with concurrent acc…

• Redis as Rate Limiter (Sliding Window Pattern)

  The sliding window rate limiter uses sorted sets where: - Member = unique request identifier (e.g., request UUID or timestamp + random) - Score = timestamp of the request Why sorted sets beat alternatives for sliding windows: - Fixed-win…

• Redis Persistence — Understanding the Trade-offs

  | Mode | How It Works | Data Safety | Performance | File Size | |---|---|---|---|---| | RDB (Snapshots) | fork() process, child writes all data to disk as a compact binary file | Data loss = time since last snapshot (typically 5-15 min)…

• Redis Production Patterns

  Pattern 1 — Distributed Lock (Redlock): - NX — only set if not exists (atomic acquire) - EX 30 — auto-expire after 30s (prevents deadlock if holder crashes) - Release: only if current value matches your token (Lua script for atomicity) P…

• Redis Memory Management

  | Eviction Policy | Behavior | Use When | |---|---|---| | noeviction | Return errors when memory full | Cache must not lose data (use as primary store) | | allkeys-lru | Evict least recently used | General caching (most common) | | volat…

• Redis High Availability — Sentinel

  Redis Sentinel is a separate system designed to manage Redis instances. Its primary purpose is providing a high-availability system through monitoring, notification, and automatic failover — without data sharding. What Sentinel does: 1.…

• Redis Cluster — Sharding + HA

  Redis Cluster provides automatic data sharding across multiple nodes with built-in failover — combining horizontal scalability with high availability. Key differences from Sentinel: | Aspect | Sentinel | Cluster | |---|---|---| | Data mo…

• Redis Transactions

  Redis transactions differ fundamentally from RDBMS transactions — they provide atomicity (commands execute without interleaving) but NO rollback on individual command failures. MULTI/EXEC flow: DISCARD — cancel the transaction (discard a…