Primary concepts and the hard part
Requirements
Uploads: 500 hours of video/minute
Views: 5B/day → ~60,000 concurrent streams/sec initiated
Bandwidth: this is the real number —
5M concurrent viewers × 5 Mbps = 25 Tbps
Storage: 1 hour of source ≈ 5 GB; × ~6 renditions ≈ 15 GB stored
500 hr/min × 15 GB = huge → petabyte scale, tieredAPI / Model
Note what is not in the database: the video bytes. Metadata in the database, bytes in object storage, delivery via CDN. That split is the whole design.
High-level architecture
Metadata and bytes take different routes. The Upload Service handles only metadata and upload URLs, while the video itself goes from the creator to object storage, through a transcoding DAG, and out to viewers from CDN edges.
- The creator posts the video's metadata, and nothing else, to the Upload Service.
- The Upload Service returns a presigned multipart URL.
- The creator uploads the file directly to object storage under
raw/, in parts that can each be retried and resumed. - The completed upload emits an event to Kafka
video.uploaded, and the Orchestrator runs the transcoding pipeline as a job DAG state machine:- Inspect the codec, duration and tracks.
- Split the source into chunks of about 10 seconds on keyframe boundaries.
- Transcode each chunk into each rendition, from 240p to 4K in H.264 and AV1, on spot instances.
- Run the side jobs: thumbnails, audio, captions and moderation.
- Package HLS / DASH segments and a master manifest.
- The packaged output is written to object storage under
processed/, and the video's status becomesREADY.
Playback runs the other way, and it doesn't touch the app servers either. The player requests segments from a CDN edge such as London, Tokyo or São Paulo, and the edges serve over 95% of all bytes. On a miss, an edge pulls the segment from the Origin, which exists only to serve those misses from processed/. The player measures throughput and buffer depth to choose the bitrate of each next segment, and sends view events asynchronously to Kafka and Flink for view counts, watch time and QoE.
Same product at 10x. At 100x the service would push 2.5 Pbps of video, which stops being a system design and becomes a telecom build-out, so this tab uses 10x. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Upload rate | 500 hours/min | 5,000 hours/min |
| Concurrent viewers | 5M | 50M |
| Egress | 25 Tbps | 250 Tbps |
| New stored video, ~15 GB per hour | ~11 PB/day | ~110 PB/day |
What changes, and the number that forces it
- The edge moves inside ISPs. At 250 Tbps, paying a commercial CDN per gigabyte becomes the dominant cost of the business. Cache appliances are placed inside ISP networks, which host them happily because the traffic no longer crosses their own transit links. Bytes travel from a box in the viewer's ISP, never across the backbone. It's the Open Connect endpoint that today's CDN trade-off already points at.
- Appliances are filled overnight, by prediction. Appliance disks are limited, and pulling every miss during prime time hammers the shield. A fill job pushes tomorrow's predicted popular titles into each appliance during off-peak hours, while the long tail still pulls on miss.
- A steering service chooses the edge. DNS-based routing can't tell that an appliance is full, unhealthy or missing a title. The player asks a steering service, which picks an appliance for each session by health, load and what that appliance actually holds.
- An origin shield tier becomes mandatory. With thousands of appliances, a new release's first wave of misses would multiply into origin fetches. A regional shield collapses them, so many edges produce one origin fetch per segment, as the viral-video follow-up describes.
- The encoding ladder depends on expected views. AV1 saves a lot of bandwidth but costs far more CPU to encode. At 5,000 hours uploaded per minute, encoding everything in AV1 at a full ladder is a compute bill that never pays back for videos nobody watches. A popularity predictor decides: likely hits get AV1 plus H.264 at the full ladder, and the long tail gets H.264 at a few renditions, re-encoded if it starts to trend.
- Storage is erasure-coded and pruned. ~110 PB/day of new renditions can't be kept at 3x replication. Processed renditions are erasure-coded, originals move to an archive tier, and the renditions of videos that went cold are deleted and regenerated from the original on demand.
What stays the same
Video bytes never touch the application servers. Uploads are still presigned, multipart and resumable, transcoding is still chunked on keyframes and run on spot capacity, players still step down fast and up slowly, and view counts still come from a stream aggregation rather than database increments.
Trade-offs and deep dives
If uploads route through your application servers, those servers become a bandwidth bottleneck and you pay for the traffic twice. A presigned URL is a time-limited, permission-scoped credential that lets the client write directly to object storage. Your service authorizes, then gets out of the way. Same idea in reverse for playback: never proxy bytes.
A 5 GB upload over a mobile connection will fail. Multipart splits it into parts uploaded independently; a failed part is retried alone rather than restarting from zero. The client also gets parallelism. This is table stakes for any large-file system, not a video-specific trick.
Transcoding a 2-hour film serially takes hours. Splitting into ~10-second chunks lets 1,000 workers transcode in parallel, cutting wall-clock time to minutes. It also makes failure cheap: a crashed worker loses 10 seconds of work, not the whole job. This is MapReduce applied to video, and the parallelism argument is the main insight the interviewer wants.
Chunks must split on keyframe boundaries, or the pieces won't decode independently and won't reassemble cleanly. Good detail to drop.
Transcoding is batch, idempotent, retryable, and not latency-sensitive. That profile is exactly what preemptible/spot capacity is for, at a large discount. Saying this shows cost awareness, which senior interviews do reward.
The video is encoded at several quality levels, each cut into aligned segments. A manifest lists them. The player measures throughput and buffer depth and picks the next segment's quality. Because segments are aligned and independently decodable, it can switch mid-playback without interrupting.
The asymmetry worth mentioning: players step down aggressively and up conservatively, because a rebuffer is far more damaging to perceived quality than a few seconds of lower resolution. That's a product-informed engineering decision.
HLS is Apple's, universally supported, historically .ts segments. DASH is the open standard, codec-agnostic. Real services ship both, or use CMAF so one set of segments serves both. Know the names; don't spend time here.
Push popular content proactively to edges before demand (a new season release is predictable); pull for the long tail. Netflix goes further with Open Connect — appliances placed inside ISP networks, so bytes never cross the public internet backbone. Mention it as the logical endpoint of "put the data near the user."
View distribution is extreme power-law: a tiny fraction of videos get almost all views. Keep hot content in standard storage and on CDN, move cold content to infrequent-access or archive tiers automatically via lifecycle policies. Also consider storing fewer renditions for cold content — you can re-transcode on demand if a five-year-old video suddenly trends.
5B/day of UPDATE videos SET views = views + 1 would melt any database, and every view is a hot row on a popular video. Instead: fire an event to Kafka, aggregate in a stream processor over tumbling windows, and write periodic aggregates. The displayed count is approximate and delayed by seconds. That's fine, and saying it's fine (rather than trying to make it exact) is the correct answer.
Same chunking and ABR, but transcoding must happen in real time with a latency budget of seconds, there's no batch parallelism (chunks arrive as they're recorded), and you need low-latency protocols (LL-HLS, WebRTC) plus origin-shield layers to protect against thundering herds when a stream starts.