Designing a Social Media Search System

What you'll learn

• Requirements (What to Clarify)

  Functional Requirements: - Store and index hundreds of millions of short text posts daily - Full-text search across all posts using keyword queries - Support multi-word queries with AND/OR logic - Return results sorted by recency, releva…

• Stage 3 — Bottleneck: Full-text search over hundreds of billions of posts

  This is THE defining problem of social-media-search. A traditional DB full-text index (Postgres GIN, MySQL FULLTEXT) works up to hundreds of millions of rows, then falls apart. Beyond that you need a dedicated inverted-index engine, and…

• Stage 4 — Second bottleneck: Ingestion write-amplification and index freshness

  Even with sharding solved on the read side, we still have to keep the index current. At 400M posts/day, each with ~15 indexable words, we're pushing ~6B posting-list updates every 24h — ~70K updates/sec sustained. If we index synchronous…

• Stage 5 — Third bottleneck: Trending query fan-out and tail latency

  Once ingestion and index shape are solved, the last recurring pain is query-side. Every search fans out to all 152 shards, so the p99 latency is set by the slowest shard — and one slow GC pause or one hot node poisons every query. Meanwh…

• Stage 6 — Mature architecture

  The full picture with everything from stages 2–5 wired together: stateless API tier, sharded inverted index over PostID with primary + replica, async ingestion via Kafka with per-shard RAM buffers for freshness, and a query-side cache +…

• Cost vs Complexity — When to Pick What

  | Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Compute | Single VM | K8s deployment, HPA on QPS | Multi-region K8s + regional routing | | Post store | Managed Po…

• Migration Path (MVP → Enterprise)

  Stepped list — HOW you'd migrate a live system, in what order, and how to avoid downtime. Steps are numbered by dependency, not by calendar month. 1. Add Postgres GIN full-text index — before you do anything else, at least stop scanning…

• The Core Insight: Inverted Index as Distributed Hash Table

  The key data structure is an inverted index — a mapping from every word to the list of post IDs containing that word.

• Index Structure

• Index Size Calculation

  ---

• Data Sharding Strategies

• Option 1: Shard by Word

  Pros: - Single-word queries hit exactly ONE server (optimal) - No scatter-gather for simple queries Cons: - Hot words ("breaking", "news") create hot servers - Uneven distribution — common words accumulate millions of PostIDs - Difficult…

• Option 2: Shard by PostID (Preferred)

  Pros: - Even distribution (PostIDs are uniformly distributed) - No hot-word problem (every word is split across all servers) - Easy to add servers (just redistribute PostIDs) Cons: - Every query hits ALL servers (scatter-gather) - Higher…

• Fault Tolerance

• Primary-Secondary Replication

• Index Rebuild (Both Replicas Die)

  Problem: If both primary and secondary for a shard die simultaneously, how do we rebuild? Solution: Index-Builder Server maintains a reverse mapping: ---

• Caching Layer

• Hot Posts Cache

  ---

• Load Balancing

  ---

• Search Result Ranking

• Beyond Simple Keyword Matching

• Popularity Score Storage

  ---

• Multi-Word Query Processing

  ---

• Key Architecture Decisions

  | Decision | Reasoning | |----------|-----------| | In-memory index | 21 TB across 152 servers = 144 GB/server (fits in RAM) | | Shard by PostID | Even distribution, no hot-word servers | | Scatter-gather queries | Acceptable at 38 QPS/s…

• Key Numbers Summary

  | Metric | Value | |--------|-------| | Total users | 1.5 Billion | | DAU | 800 Million | | Posts/day | 400 Million | | Searches/day | 500 Million | | Search QPS | ~5,800 | | Post size | 300 bytes | | Daily storage | 120 GB | | 5-year st…