Designing a Top-K / Trending System
High Level Design·advanced·~25 min read
- system-design
- streaming
- aggregation
- top-k
- hld
What you'll learn
Problem Statement
Design a system that determines the top K most viewed (or liked, shared, etc.) items across multiple time windows — hourly, daily, monthly, and all-time. The system ingests a massive event stream (~700K events/second) from user interacti…
Requirements
Functional Requirements
1. Query top K items for any supported time window (last hour, last day, last month, all-time) 2. Items are ranked by a single metric (views, likes, etc.) within each window 3. K is configurable per query (typically 10-100) 4. Support tu…
Non-Functional Requirements
1. Freshness — results reflect events up to
Why It Fails
| Issue | Impact | |-------|--------| | 700K writes/sec to single DB | Exceeds Postgres capacity (~10K writes/s) | | GROUP BY + ORDER BY on billions of rows | Minutes to compute, not
Watermarks and Late Events
Events may arrive out of order due to network delays. Flink uses watermarks to handle this:
Tumbling Window Aggregation in Flink
Instead of writing every raw event (700K/s), Flink aggregates within windows:
Fault Tolerance via Checkpoints
If Flink fails, it resumes from the last checkpoint. Kafka retains messages, so no events are lost. This gives exactly-once processing semantics. ---
Scaling Writes
The Sharding Challenge
At 700K TPS of raw events: | Approach | Shards Needed | Reasoning | |----------|---------------|-----------| | Raw events to DB | ~70 shards | Each shard handles ~10K writes/s | | Flink pre-aggregated | 5-10 shards | Flink reduces 700K →…
Scaling Reads
The Problem with Direct DB Queries
Even with pre-aggregated tables, computing top-K at query time means: - Scanning an index on the count column across all items in a window - At 4B items, even an indexed ORDER BY is not
Key Design Decision
The Top-K service NEVER queries the database in the hot path. It only reads from the precomputed Redis cache. This guarantees = true count (upper bound) | | Accuracy | With w = e/epsilon, d = ln(1/delta): error = ? GROUP BY itemid ORDER…
Stage 3 — First bottleneck: Aggregating a 700K/sec firehose without melting the DB
This is the characteristic top-K problem: you don't actually need every raw event durably stored — you need item-level counts per window. The trick is collapsing 700K raw events/sec into a much smaller stream of (itemid, window, count) u…
Stage 4 — Second bottleneck: Serving top-K at
Option comparison — how to serve top-K reads: | Option | How it works (1 line) | p99 read latency | Freshness | Cost | Complexity | Fails at | |--------|-----------------------|------------------|-----------|------|------------|---------…
Stage 5 — Mature architecture
Combining Stages 2–4: stateless API on one side, Kafka + Flink chewing the firehose, Redis ZSETs as the hot serving surface, Postgres aggregate tables as the durable fallback, and S3 as the checkpoint substrate. ---
Cost vs Complexity — When to Pick What
| Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Ingest | Direct HTTP → DB | Kafka + naive consumer | Kafka multi-region + schema registry | | Counting | Postgres…
Migration Path (MVP → Enterprise)
Stepped migration for a live system, in dependency order: 1. Introduce Kafka in front of the DB writer — writers publish to Kafka and a consumer replays into Postgres. No read behavior changes; you now have replay and buffering. No downt…
Full In-Memory Flink Architecture
Eliminating the Database Entirely
For maximum performance, run everything in Flink's managed state:
Fault Tolerance
Why This Works
- No database in the critical path - Flink manages all state with automatic fault tolerance - Redis is just a serving cache (not source of truth) - Horizontal scaling by adding Flink tasks + Kafka partitions - Exactly-once via Kafka tran…
End-to-End Summary
Design Decision Summary
| Decision | Choice | Rationale | |----------|--------|-----------| | Stream processor | Apache Flink | Native windowing, exactly-once, checkpoints | | Buffering | Kafka | Handles 700K/s, replay on failure, partitioned | | Write reductio…