Large Media Upload & Processing Pipeline
High Level Design·advanced·~50 min read
- system-design
- media
- s3
- signed-url
- multipart
- fan-out
- transcoding
- hld
- 1. Direct-to-Storage Uploads via Signed URLs - 2. Client Event vs. S3 Event — Who Triggers Processing? - 3. Multipart Uploads for Large Media (>10 MB) - 4. Client-Side Resilience - 5. Securing S3 Access for Internal Workers - 6. Consum…
What you'll learn
1. Direct-to-Storage Uploads via Signed URLs
The API server does not relay media bytes. The client uploads directly to object storage (S3, GCS, Azure Blob) using a short-lived pre-signed URL scoped to a specific object key and HTTP verb. - Why: your API tier serves 10 KB JSON. Rout…
1a. One Signed URL per File — Not per Bucket
Pre-signed URLs are intrinsically tied to one object key. You cannot generate a URL that lets the client upload arbitrary filenames into your bucket. For a 5-file carousel: - Client issues one API request: GET /posts/upload-urls?count=5.…
2. Client Event vs. S3 Event — Who Triggers Processing?
A tempting shortcut: let S3 fire an ObjectCreated webhook into your processing pipeline so the backend has nothing to do. This is wrong for the primary publishing flow. | Problem | Why S3 events fail | |----------------------------|-----…
2a. The Recommended Flow
1. Client requests pre-signed URLs → uploads directly to S3. 2. Client sends an explicit POST /posts/{id}/commit to the Post Service. 3. Post Service verifies the files exist (HeadObject on each key). 4. Post Service drops MediaReadyForP…
3. Multipart Uploads for Large Media (>10 MB)
A standard PUT of a 50 MB video that fails at 49 MB is a UX catastrophe. For anything over ~10 MB, use S3 Multipart Uploads: the file is sliced into 5 MB chunks and uploaded in parallel. Only failing chunks retry. The backend orchestrate…
Phase 1 — Initiate
Backend calls CreateMultipartUpload on S3, receives an UploadId, picks a secure key, decides the chunk size.
Phase 2 — Request Part URLs
Client slices locally, calculates it needs e.g. 30 parts, asks for signed URLs in batches: Backend returns one signed UploadPart URL per part.
Phase 3 — Client Uploads Chunks
Client PUTs each chunk directly to S3. S3 returns an ETag header on each successful part — the client must capture it. Later S3 uses the ETag manifest to verify the final file. > CORS gotcha for web clients: your bucket's CORS policy mus…
Phase 4 — Complete
Backend calls CompleteMultipartUpload with the manifest. S3 stitches, verifies hashes, makes the object available. Backend drops MediaReadyForProcessing. ---
4. Client-Side Resilience
Mobile networks drop packets without closing TCP. Backgrounded apps lose RAM. Users force-kill apps mid-upload. Build for all of it.
4a. Retry Logic
- Exponential backoff with jitter. On a 5xx or network drop: wait 1 s → 2 s → 4 s → 8 s, plus a random millisecond offset. Jitter prevents a thundering herd when a cell tower comes back online. - Stalled-connection timeouts. Mobile uploa…
4b. Resumable State Machine
In-memory retry loops die when the OS kills the app. Treat uploads as persistent state in a local DB (Room / CoreData / IndexedDB): | Step | Action | |------|-------------------------------------------------------------------------------…
4c. Backend Garbage Collection
Users will always abandon uploads. Don't write a cron job — use S3 Lifecycle Rules: - AbortIncompleteMultipartUpload after 3 days (reclaims chunks that never completed). - Delete uncommitted objects in the uploads/ bucket after 7 days. -…
5. Securing S3 Access for Internal Workers
Your transcoder consumer needs to download the raw video and upload the processed outputs. Hardcoding IAM Access Keys in each service is the classic anti-pattern: on a breach you must rotate across every service, and leaks are easy.
Option A — IAM Workload Identity (Best Practice)
Workers run on AWS compute (EC2 / EKS / ECS / Lambda). - Attach an IAM Role to the instance / pod / function. - The AWS SDK auto-fetches temporary credentials from the instance metadata service, rotates them every 15–60 minutes. - No key…
Option B — AWS STS AssumeRoleWithWebIdentity
Workers run outside AWS (on-prem, GCP, Azure). - Worker authenticates to a central OIDC provider. - Worker calls STS AssumeRoleWithWebIdentity with the OIDC token. - STS returns short-lived credentials. Equivalent on GCP: Workload Identi…
Option C — Pre-Signed URLs in the Event Payload
Treat workers like clients. The Post Service pre-computes read + write URLs and ships them inside the message: Worker needs zero AWS SDK. Pure HTTP. Leaked URLs expire in an hour. Limitation — HLS/DASH streaming: adaptive-bitrate transco…
Decision Table
| Scenario | Pick | |---------------------------------------------------|---------| | Workers on AWS, static output count | A or C | | Workers on AWS, HLS/DASH or unknown output count | A | | Workers off-AWS, you own the identity provide…
6. Consumer Architecture — Fan-Out / Fan-In
A single MediaReadyForProcessing event spawns multiple parallel tasks with wildly different cost profiles. The event bus fans out one event to many specialised queues.
6a. Why Separate Queues per Task Type
- Thumbnail: 2 s, 512 MB RAM, CPU — cheap. - Transcode 1080p: 60 s, hardware encoder preferred. - AI caption (Whisper): 45 s, dedicated GPU, PyTorch runtime. - Virus scan: 10 s, specialised engine. Put them all in one queue and a GPU bac…
6b. Streaming the Byte Pipe
Do not download 2 GB to local disk → process → upload. Disk I/O and container storage costs scale badly. Use Unix pipes / streaming SDKs: S3 GetObject → stdin → ffmpeg → stdout → S3 Multipart Upload. The worker never materialises the ful…
6c. Specialised Hardware
| Task | Hardware | Why | |--------------------------------|--------------------------------------|------------------------------------------------| | Thumbnails, audio extraction | General-purpose CPU (Graviton / M6g) | Lightweight; che…
6d. Queue-Depth Autoscaling (KEDA)
Media uploads are bursty — New Year's Eve, concert, product launch. Do not scale on CPU — by the time CPU is high your users are already waiting. Scale on queue depth (KEDA on Kubernetes, Keda-Core on ECS, Lambda event source scaling). -…
7. Fan-In — Collating Results Without Race Conditions
As workers finish they emit TaskCompleted. The Post Service aggregates. The non-trivial part is that two workers can finish in the same millisecond.
7a. The Job Manifest
Keep a relational row per task on the parent post: See [[scaling-relational-databases]] on how this table survives at 1 B+ posts.
7b. The Fan-In Transaction
Both workers independently read the manifest, both see Count(PENDING) == 0, both try to "publish and notify." Classic race. Fix with a row-level lock on the parent post: Postgres serialises the two transactions on the FOR UPDATE lock. Ex…
7c. Edge Cases
- Idempotent task completion. At-least-once delivery means a worker may process the same message twice. UPDATE … SET status='SUCCESS' WHERE status='SUCCESS' must be a no-op (which it naturally is). - Partial failures. AI captioning on a…
What to Say in an Interview
1. Signed URLs for direct-to-storage uploads — your API never touches media bytes. One URL per object key; batch the request. 2. Client sends an explicit /commit — S3 events are for storage-level work, not product state transitions. 3. M…