Designing a Metrics Monitoring System
High Level Design·advanced·~25 min read
- system-design
- monitoring
- time-series
- alerting
- observability
- hld
What you'll learn
1. Problem Statement
Design a metrics monitoring and observability platform capable of ingesting 5 million metrics per second from 500,000 servers. The platform must support real-time dashboards, flexible metric queries, and alerting with less than 1 minute…
Scale Parameters
| Parameter | Value | |-----------|-------| | Metrics ingestion rate | 5 million/sec | | Number of servers/hosts | 500,000 | | Unique time series | ~50 million | | Dashboard query latency | 90% for 5 minutes). Support aggregations across…
Non-Functional Requirements
- High availability: No single point of failure. Ingestion must not drop data during partial outages. - Horizontal scalability: Every component scales independently. - Durability: Once acknowledged, metrics must not be lost. - Low alert…
Design Evolution & Trade-offs
How this system grew from an MVP to what a production version looks like — and the choices you'd defend in an interview. Metrics monitoring is a write-heavy, append-only workload. That single fact drives almost every architectural choice…
Stage 1 — MVP: One box, one database
The simplest thing that could possibly work for a metrics monitoring platform. - A single API server accepting HTTP POST /metrics from a handful of agents. - A single relational database (Postgres) with a metrics(name, timestamp, value,…
Stage 2 — Scale-out: Multiple pods behind a load balancer
- The API layer becomes stateless: N pods behind an L7 load balancer. - Batching is introduced on the agent side (send every 10 seconds instead of every point) and on the server side (bulk INSERT with 1000-row batches). - The alert cron…
Stage 3 — Bottleneck: Ingest pipeline for a write-heavy stream
At 5M metrics/sec the ingest path is the system. Writing 5 million small rows per second into any single-primary database is a losing battle regardless of tuning. The real question is how you decouple the producer (agents) from the consu…
Stage 3.5 — The cardinality problem (a sibling bottleneck)
Cardinality is enough of a hard problem in metrics systems that it deserves its own comparison. If a well-meaning developer adds requestid as a tag to a latency metric, they instantly create one time series per request — potentially mill…
Stage 5 — Mature architecture
The end state pulls together every choice above: durable Kafka buffer, tiered storage, dedicated alert path off the same Kafka topic, cardinality controls at ingest, and dashboards fed by a caching query engine. Every arrow in this diagr…
Cost vs Complexity — When to Pick What
| Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Compute (ingest) | Single VM running the API | K8s deployment, HPA on CPU | Multi-region K8s with regional Kafka m…
Migration Path (MVP → Enterprise)
A production metrics system is almost never built greenfield at target scale. It's grown, one migration at a time, from a much simpler starting point. The order matters — some steps unblock others, and some steps are safe to defer indefi…
5. Ingestion Layer
Design
The ingestion layer is a fleet of stateless HTTP/gRPC servers sitting behind a load balancer. Their responsibilities: - Accept batched metric payloads from agents - Validate metric names, tag formats, and value ranges - Normalize timesta…
Sizing
At 5M metrics/sec with each ingestion server handling ~50-100K metrics/sec: - 50-100 ingestion servers required - Horizontal scaling: simply add more servers behind the load balancer
Back-Pressure Handling
When Kafka or downstream systems are overwhelmed: 1. Ingestion servers return HTTP 429 to agents 2. Agents buffer metrics locally (ring buffer, typically 5-10 minutes) 3. Agents retry with exponential backoff 4. This prevents cascading f…
6. Kafka as Buffer Layer
Why Kafka?
Kafka serves as the central nervous system, decoupling producers (ingestion) from consumers (storage, alerting): | Benefit | Explanation | |---------|-------------| | Decoupling | Ingestion, storage, and alerting scale independently | |…
Partitioning Strategy
- Partition key: hash(metricname + sortedtags) - This ensures all data points for the same time series land on the same partition - Maintains temporal ordering per series (critical for alert evaluation) - 500+ partitions for parallelism…
Consumer Groups
7. Time-Series Database (TSDB)
Storage Architecture -- Tiered Design
Compression Techniques
Time-series data is highly compressible because consecutive timestamps and values are similar: Timestamp compression (Delta-of-Delta): - Timestamps arrive at regular intervals (e.g., every 10 seconds) - Store the delta-of-delta: if times…
8. Downsampling and Rollups
Purpose
Full-resolution data (every 10 seconds) is only needed for recent time ranges. For historical queries spanning days or months, pre-aggregated rollups provide fast query performance with dramatically reduced storage.
Rollup Aggregates
For each time bucket (1-minute, 1-hour), pre-compute: | Aggregate | Use Case | |-----------|----------| | min | Detect lowest values (e.g., minimum free memory) | | max | Detect peaks (e.g., max CPU spike) | | avg | General trend analysi…
Storage Savings
| Tier | Resolution | Points/series/day | Relative Size | |------|-----------|-------------------|---------------| | Hot (raw) | 10 sec | 8,640 | 1x | | Warm (1-min) | 60 sec | 1,440 | ~0.17x (with 6 aggregates: ~1x) | | Cold (1-hour) |…
Implementation
Background rollup jobs run periodically: 1. Read raw data points for the previous completed time bucket 2. Compute all aggregate functions 3. Write rollup records to the warm/cold tier 4. Mark raw data as eligible for eviction after TTL
9. Query Engine
Query Flow
Tier Selection
| Query Time Range | Data Source | |-----------------|-------------| | Last 6 hours | Hot tier (full resolution) | | Last 7-30 days | Warm tier (1-min rollups) | | 30+ days | Cold tier (1-hour rollups) |
Caching Strategy
- Redis cache for hot dashboard queries with TTL of 10-30 seconds - Cache key: hash(query + timerangebucket) - Cache invalidation: time-based (short TTL) since data is append-only - Reduces TSDB load by 60-80% for popular dashboards
Query Optimizations
- Pushdown filters: Send tag filters to TSDB shards so they return only matching series - Parallel fan-out: Query all relevant shards simultaneously - Early termination: For top-N queries, stop once sufficient data is collected - Pre-mat…
10. Alert Evaluation Engine
Design Principles
The alert evaluator is the most latency-sensitive component. Key design decisions: 1. Consume directly from Kafka -- bypasses TSDB write latency entirely 2. In-memory sliding windows -- no disk I/O for recent data 3. Per-partition assign…
Alert Rule Example
State Machine
Why PENDING State?
The PENDING state prevents flapping alerts. A brief CPU spike lasting 10 seconds should not trigger an alert configured with for: 5m. The metric must continuously exceed the threshold for the full specified duration before firing.
Preventing Duplicate Notifications
- State machine ensures notification is sent exactly once on transition to FIRING - A resolved notification is sent once when transitioning to RESOLVED - Deduplication IDs prevent re-notification if evaluator restarts
High Availability
- Alert evaluators run in active-passive pairs per partition range - Standby evaluator maintains warm state by consuming Kafka (slightly behind) - On primary failure, standby promotes within seconds - Worst case: one evaluation cycle del…
11. Notification Service
Responsibilities
- Route alerts to configured channels (email, Slack, PagerDuty, webhooks) - Manage notification preferences per team/alert severity - Retry failed deliveries with exponential backoff - Maintain notification history for audit and debugging
Architecture
Delivery Guarantees
- At-least-once delivery with idempotency keys - Retry schedule: 1s, 5s, 30s, 2m, 10m (then dead-letter queue) - Notification history table for audit trail
12. Handling Cardinality Explosion
The Problem
High-cardinality tags (like userid, requestid, or traceid) can cause the number of unique time series to explode. If a metric has a tag with 1 million unique values, that single metric produces 1 million time series, overwhelming the TSD…
Mitigations
| Strategy | Description | |----------|-------------| | Ingestion-time validation | Reject or drop tags from a known blocklist (userid, etc.) | | Rate limiting per metric | Cap the number of new series a single metric can create per hour…
Detection
- Track cardinality per metricname in a HyperLogLog sketch - Alert operators when a metric's cardinality exceeds warning thresholds - Dashboard showing top-10 highest cardinality metrics
13. Late and Out-of-Order Data
Challenge
In distributed systems, data can arrive out of order due to network delays, agent buffering, or clock skew between hosts.
Solution
- Accept data within a configurable late window (default: 5 minutes) - Data arriving within the window is inserted normally (TSDB supports out-of-order writes within the window) - Data arriving after the window is dropped or sent to a de…
14. Scaling Considerations
Horizontal Scaling at Every Layer
| Component | Scaling Mechanism | Target Scale | |-----------|-------------------|--------------| | Ingestion servers | Add more stateless servers behind LB | 50-100 servers | | Kafka | Add partitions, add brokers | 500+ partitions, 20+…
Bottlenecks and Solutions
TSDB Write Throughput: - Batch writes (buffer 1-5 seconds per writer) - Write-ahead log for durability during batching - Parallel writes across shards Query Fan-out Latency: - Limit fan-out degree (query only relevant shards) - Aggressiv…
15. End-to-End Latency Analysis
Tracing the path from metric emission to alert notification: The <1 minute SLA is achievable for alerts configured with for: 0 (instant alerts). For alerts requiring sustained threshold violation, the "for" duration dominates.
16. Summary of Key Design Decisions
| Decision | Rationale | |----------|-----------| | Kafka between ingestion and storage | Decoupling, durability, fan-out to multiple consumers | | Alert evaluator reads from Kafka (not TSDB) | Reduces alert latency by bypassing write +…