Designing a Distributed Job Scheduler

What you'll learn

• Problem Statement

  Design a system that executes jobs at specified times or intervals. The system must support scheduling jobs for immediate execution, future execution at a specific time, and recurring execution based on cron-like expressions. At scale, i…

• Key Distinction: Task vs Job

  Understanding the difference between a Task and a Job is critical to the design: | Concept | Definition | Example | |---------|-----------|---------| | Task | A reusable definition of work to be done (the template) | "send email" | | Job…

• Requirements

• Functional Requirements

  1. Schedule jobs for immediate execution 2. Schedule jobs for future execution at a specific time 3. Schedule recurring jobs (cron expressions, fixed intervals) 4. Monitor job status (pending, running, succeeded, failed) 5. Cancel or pau…

• API Design

  Job Submission: Job Status Query:

• Non-Functional Requirements

  1. Throughput: Handle 10,000 jobs/sec submission and execution 2. Accuracy: Execute jobs within 2 seconds of their scheduled time 3. At-least-once guarantee: Every scheduled job must execute at least once (may execute more than once in f…

• Core Entities

  Job States: PENDING -> QUEUED -> RUNNING -> SUCCEEDED / FAILED ---

• High-Level Architecture

• Component Responsibilities

  | Component | Role | |-----------|------| | API Service (Submission Service) | Receives job definitions, stores binary in Object Store, metadata in DB | | Job Store (Postgres) | Durable storage for tasks, jobs, and execution history | |…

• Storage Separation Principle

  A key architectural insight is separating binary storage from metadata: - Object Store (S3/GCS): Stores job binary code — binaries can be large and don't need relational querying - Metadata DB: Stores job definitions, scheduling info, an…

• Design Evolution & Trade-offs

  How this system grew from an MVP to what a production version looks like — and the choices you'd defend in an interview. A distributed job scheduler is deceptively simple on day one — a table of "when to run" rows, a cron loop, a worker.…

• Stage 1 — MVP: One box, one database

  The simplest thing that could possibly work for a scheduler. - A single API server (monolith) accepts POST /jobs. - A single Postgres instance stores jobs(nextexecutionat, state, payload). - A background thread inside the same process ru…

• Stage 2 — Scale-out: Multiple pods behind a load balancer

  - API layer becomes stateless; N replicas behind an L7 load balancer accept submissions. - The scheduler loop is pulled out of the API pods into a separate deployment — you don't want a slow poll to steal CPU from the submission path. -…

• Stage 3 — Bottleneck: Hot-instant timer clustering

  This is the characteristic problem of a scheduler at scale, and it's caused by humans, not machines. Every marketing team schedules the campaign for 09:00:00 UTC. Every cron says 0 . Every batch reminder lands at midnight. When 100k j…

• Stage 4 — Second bottleneck: Fault-tolerant execution and worker failure detection

  You've solved finding due jobs at scale. Now the second wall: workers die mid-execution, and the at-least-once guarantee demands that the system notice and re-dispatch. Naively, this is trivial — a SELECT WHERE state='RUNNING' AND starte…

• Stage 5 — Mature architecture

  Everything from Stages 2–4 assembled into one picture: Approximately 12 nodes: load balancer, API tier, primary Postgres (sharded), read replicas, Redis ZSET cache, scheduler tier (partitioned via Zookeeper), Kafka fan-out, worker pool,…

• Cost vs Complexity — When to Pick What

  | Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Compute | Single VM running API + scheduler | K8s deployment: API tier + scheduler tier + workers | Multi-region K…

• Migration Path (MVP → Enterprise)

  The order matters. Each step is independently valuable, each is safe to roll back, and none of them requires the whole system to be reworked at once. 1. Shard the jobs table by hash(jobid) % N. No behavioural change; pollers just get sha…

• API Contract

  Key entity — Callback (specialisation of Job): payloadref points to blob storage if payload is large; small payloads inline. shardkey = hash(callbackid) % N — decouples routing from fireat so temporal clustering can't create hot shards (…

• Design A — Second Precision (fireat bucketed to whole seconds)

  This is the easier variant. Anything within ±1s of the target is acceptable. Dispatcher loop (one per shard): Why shard-per-loop: every dispatcher polls only its own shard partition, so 200ms tick × N shards gives you effectively N paral…

• Design B — Millisecond Precision (fireat bucketed to whole ms)

  The interesting change: your polling interval must be shorter than your tolerance. If you promise ±5ms, you cannot poll every 200ms. Two viable approaches:

• The Hot-Instant Overlap Problem

  > "Seriously a lot of requests overlapping at the specific millisecond or second." Real workloads cluster. Every marketing team schedules the email at 09:00:00.000 UTC. Every cron says 0 . Every batch of 100k reminders lands at 00:00:…

• Delivery Semantics

  At-least-once + idempotent is still the correct choice for callbacks — but the receiver is the one that must be idempotent, not the scheduler's own task code. Push this into the API contract: Receivers use X-Deduplication-Key to make ide…

• Second vs Millisecond — Comparison

  | Dimension | Second precision (±1s) | Millisecond precision (±5ms) | |------------------------|--------------------------------------|-------------------------------------| | Storage bucket size | 1000ms | 1ms | | Storage rows/sec | ~1×…

• When Not to Do Millisecond Precision

  If any of these are true, downgrade to second precision and jitter: - Callback receivers cannot process ≥ your millisecond fire rate → the queue backs up and precision is illusory. - Network hop to the receiver is > your tolerance (a 30m…

• Summary: Key Design Decisions

  | Decision | What It Solves | |----------|---------------| | Redis sorted set (score = timestamp) | O(log N) retrieval of due jobs; sub-second polling | | Partitioned scheduling | No duplicate scheduling; horizontal scaling | | Kafka as…