Designing a Photo Sharing Service
High Level Design·intermediate·~25 min read
- system-design
- photo-sharing
- distributed-systems
- hld
What you'll learn
Problem Statement
Design a social photo-sharing platform where users can upload photos, follow other users, and view a personalized News Feed of latest photos from people they follow. The system must handle hundreds of millions of users while keeping phot…
Requirements
Functional Requirements
1. Upload/Download/View photos 2. Search photos by title/description 3. Follow other users 4. News Feed — personalized feed of top photos from followed users 5. Large files — support photos up to several MB
Non-Functional Requirements
1. High availability — system should always be accessible 2. Low latency — News Feed generation What breaks next — reads dominate this system 100:1. Every feed impression fans out to fetch N thumbnails, so download bandwidth from the ori…
Stage 3 — Bottleneck: Photo delivery latency and origin bandwidth
The FIRST hard, domain-specific problem for a photo-sharing service is serving hundreds of MB/s of images to millions of geographically distributed viewers without melting the origin or the wallet. A single viral photo can be requested 5…
Stage 4 — Second bottleneck: News feed fan-out for celebrity users
Once photo delivery is solved, the next domain-specific wall is how you assemble a user's News Feed — top 100 recent photos from the 500 accounts they follow, in under 200 ms, across 500 M users. The naive read-time approach (scatter-gat…
Stage 5 — Mature architecture
The full production system with everything from stages 2–4: separated upload/download paths, CDN edge, cached metadata, wide-column store, sharded object storage, async fan-out, and a background Feed Generator. ---
Cost vs Complexity — When to Pick What
| Concern | MVP choice | Growth choice | Enterprise choice | |---------|-----------|---------------|-------------------| | Compute | Single VM API + DB | K8s deployment, upload/download pools split | Multi-region K8s with regional failov…
Migration Path (MVP → Enterprise)
Ordered, safe migration steps. Each step is independently deployable and reversible. 1. Move photo bytes out of the DB into object storage. Write path: API writes bytes to S3, stores URL in DB. Read path: return the S3 URL. This unblocks…
Database Design (Detailed)
Schema
Why NoSQL / Wide-Column Store?
| Consideration | RDBMS | Wide-Column (Chosen) | |---------------|-------|---------------------| | Horizontal scaling | Hard (sharding is complex) | Built-in, linear scaling | | Availability | Master-slave has failover lag | Multi-master…
Index Design
- Primary index on PhotoID — enables O(1) photo lookup - Index on (PhotoID, CreationDate) — enables "latest photos" queries - Since PhotoID embeds epoch time, primary index IS the time-sorted index ---
Key Generation Service
The Problem
With PhotoID-based sharding (Shard = PhotoID % N), we need the PhotoID BEFORE we know which shard to write to. We can't use per-shard auto-increment.
Solution: Dedicated Key Generation DB
Fault tolerance: If one key-gen server dies, the other continues serving (IDs will have gaps — odd-only or even-only — but that's fine; uniqueness is preserved).
Epoch + Sequence Structure
Why this structure? - Primary key queries for "latest photos" work via index scan (IDs are time-ordered) - 512 photos/sec capacity vs 23/sec average = 22x headroom - Sequence resets every second - No coordination needed between shards ---
Data Sharding (Deep Analysis)
Strategy A: Shard by UserID
| Advantage | Disadvantage | |-----------|-------------| | All user's photos on one shard | Hot users create hotspots | | Simple user-centric queries | Non-uniform storage distribution | | No scatter-gather for "my photos" | Single shard…
Strategy B: Shard by PhotoID (Chosen)
| Advantage | Disadvantage | |-----------|-------------| | Even distribution across shards | "All photos by User X" requires scatter-gather | | No hot-shard from popular users | Feed generation touches all shards | | PhotoID is globally…
Planning for Growth: Logical Partitions
- Start with many logical partitions on few physical servers - As data grows, migrate logical partitions to new servers - Only update a config file mapping partitions → servers - No data restructuring needed ---
News Feed Generation
The Problem
For User X who follows 500 people, generate a feed of top 100 latest photos from all followed users. Naive approach: 1. Get list of 500 followed users 2. For each, fetch latest 100 photos (scatter to all shards) 3. Rank 50,000 photos, re…
Solution: Pre-Generation
UserNewsFeed table structure: ---
Feed Delivery: Pull vs Push vs Hybrid
Pull Model
- Client initiates request when opening app or scrolling - Pro: Simple, no server-side state for push connections - Con: Stale until next pull; most pulls return "no new data"
Push Model
- Server pushes new feed entries to followers immediately - Pro: Real-time updates - Con: Celebrity with 5M followers → 5M pushes per upload = fan-out explosion
Hybrid Model (Chosen)
Why this works: - 99%+ of users have < 10K followers → push is cheap (write to a few thousand feed entries) - The rare celebrity post doesn't trigger millions of writes - Celebrity followers get feed on next open (seconds of staleness, a…
Caching Strategy
Multi-Layer Cache
80-20 Caching Rule
- 20% of photos generate 80% of traffic - Daily read volume: 2,300 reads/sec × 86,400 sec = ~200M reads/day - Cache 20%: ~40M photo references - At 200 KB each: 40M × 200 KB = 8 TB cache (distributed across servers) - With metadata only:…
Cache Invalidation
- Photo data: Never changes once uploaded → infinite cache validity - Metadata: TTL-based (e.g., 5 minutes) or event-driven invalidation on edit - Feed cache: Invalidated when Feed Generator updates the UserNewsFeed table ---
Reliability and Redundancy
Photo Storage Redundancy
- Minimum 3 copies of every photo across different servers/racks - Object storage systems handle this natively (replication factor = 3) - Losing one server → serve from replica with zero downtime
Service Redundancy
Metadata DB Redundancy
- Wide-column stores (Cassandra) have built-in multi-node replication - Configurable consistency: write to quorum (2 of 3 replicas) before ack - Read from any replica (eventual consistency acceptable for photos) ---
Load Balancing
Three Layers of Load Balancing
Algorithm Selection
| Stage | Algorithm | Why | |-------|-----------|-----| | Initial | Round Robin | Simple, works when all servers are similar | | Production | Weighted Round Robin | Account for different server capacities | | Mature | Least Connections /…
Health Monitoring
- LB monitors server health (response time, error rate, connection count) - Unhealthy servers removed from rotation within seconds - Recovered servers re-added after passing health checks ---
Security Considerations
- Upload validation: Check file type (reject non-image), size limits, malware scanning - Access control: Public photos (anyone), private (owner + specified users) - Authentication: Token-based auth for all API calls - Rate limiting: Prev…
Summary: Why Each Decision Matters
| Decision | What it Solves | |----------|---------------| | Read/Write separation | Connection starvation from slow uploads | | Wide-column DB | Scale to petabytes without complex sharding | | PhotoID-based sharding | Even distribution,…