Primary concepts and the hard part
Requirements
Users: 500M, avg 3 devices
Files: 100B files
Daily uploads: 1 PB/day
Dedupe savings: ~50% cross-user (same PDFs, installers, media everywhere)
Chunk size: ~4 MB average (variable, content-defined)
Metadata ops: far higher QPS than byte transfers — the metadata service
is the real high-traffic serviceAPI / Model
The chunks table being global and keyed by content hash is the whole dedupe story: if any user anywhere has already uploaded a chunk with that hash, nobody uploads it again.
High-level architecture
Sync is split three ways: the client turns file changes into hashed chunks, the Metadata Service decides which chunks are missing and records versions, and object storage holds the bytes. Other devices find out about changes through the user's journal and a notification poke.
- The filesystem watcher sees a change. The content-defined chunker splits the file into chunks of about 4 MB, the client hashes each one with SHA-256 and updates its local index, and then it sends the hashes with
POST /prepare. The Metadata Service checks auth, quota and the path, and returns only the hashes it doesn't already have. - For those missing chunks it hands out presigned URLs, and the client uploads the bytes straight to object storage, where each chunk is stored under its
chunk_hash. - The client commits the ordered chunk list. The Metadata Service writes the new version to the Metadata DB, on the user's shard, and appends an entry with the next
seqto the user's journal. - The journal entry triggers the notification service, which pokes the user's other devices over long-poll or WebSocket without sending any payload.
- Each device calls
GET /delta?cursor=last_seq, compares the changes with its local index, downloads only the chunks it lacks, reassembles the file and advances its cursor.
The conflict branch applies to a device that edited the same file while offline. Its commit carries the base version it started from. If that is still the current version, the change is applied. If not, both files are kept: report.docx and report (conflicted copy).docx. In the background, object storage keeps a refcount on every chunk and deletes unreferenced chunks lazily.
Same product at 10x the data. 100x would mean more users than there are people, so this tab uses 10x the bytes and about 4x the users, with far larger shared workspaces. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Users | 500M | ~2B |
| Devices | ~1.5B | ~6B |
| Files | 100B | ~1T |
| Uploads | 1 PB/day | 10 PB/day |
| Largest shared folder | a team | a company of 100k+ people |
What changes, and the number that forces it
- Metadata shards by namespace, not by user. Sharding by
user_idassumed a user's files live with that user. At 10x, much of the data sits in shared folders used by tens of thousands of people at one company, and the "reference plus subscription" approach turns every commit into a fan-out across huge member lists. A namespace (a user's private root, or one shared folder) becomes the unit: its files, versions and journal live on one shard, a commit is still a single-shard transaction, and each device keeps one cursor per namespace it can see. - The chunk index gets its own shards and a filter. The global
chunkstable grows to trillions of entries and sits behind every/prepare. It's sharded by hash prefix (hashes are uniform, so there are no hot spots), and a Bloom filter on each shard answers "definitely new" for fresh content, so a brand-new video upload skips thousands of index lookups. - Bytes move onto owned storage. At 10 PB/day, renting object storage becomes one of the largest bills in the company. Chunks go to owned block storage with erasure coding (far less overhead than 3x replication), and chunks untouched for a year move to denser, higher-ratio codes; this is roughly the path Dropbox took with its own storage system. Uploads land in the nearest region, erasure-coded across its zones, and replicate to a second region asynchronously.
- Refcounts give way to mark-and-sweep. Updating a global refcount on every commit and delete means a cross-shard write for every shared chunk, and one lost decrement leaks space while one doubled decrement deletes live data. Instead, a collector scans live chunk lists shard by shard, builds the live set, and deletes chunks that are absent from it and older than a generous grace period. It reclaims space slower, but it can only err toward keeping garbage.
- Notification becomes its own fleet. ~6B devices holding long-polls means millions of idle connections per server. A dedicated notification fleet tracks which devices wait on which namespaces and coalesces pokes, so a burst of 500 commits to one team folder still produces a single poke per device.
What stays the same
Content-defined chunking so an edit only re-uploads the chunk it touches, content addressing by SHA-256, presigned direct uploads of missing chunks only, delta sync from a cursor, a notification that pokes rather than carries data, "keep both" on conflict, and soft deletes with a restore window. Metadata is still the hot path and bytes are still the volume problem; each just got its own dedicated scaling story.
Trade-offs and deep dives
With fixed-size chunks, inserting a single byte at the start of a file shifts every subsequent boundary, so every chunk hash changes and you re-upload the entire file. Content-defined chunking (a rolling Rabin fingerprint that declares a boundary when the hash of a sliding window matches a pattern) makes boundaries depend on content, so an insertion only disturbs the chunk it lands in. One edited paragraph in a 2 GB file means uploading ~4 MB. This is the difference between a usable product and an unusable one, and it's the single detail most worth getting right in this interview.
- Within a file: repeated blocks stored once.
- Across a user's versions: v5 and v6 of a document share nearly all chunks, so versioning is nearly free.
- Across all users: the same Ubuntu ISO uploaded by 10,000 people is stored once, because the key is the content hash.
Global dedupe typically halves storage. The privacy caveat worth raising: content-addressed global dedupe leaks information — an attacker who can observe "this chunk already exists" can confirm whether a specific file exists in the system. Mitigate by scoping dedupe per user or per organization, or by adding a per-tenant salt to the hash. Raising this unprompted is a strong signal, because it shows you think about the security consequences of an optimization.
The metadata service handles far more QPS than the byte pipeline (every client polls for deltas constantly, but most polls find nothing). Shard metadata by user_id so all of a user's files, versions, and journal entries live on one shard, making a commit a single-shard transaction and the delta query a single-shard range scan. Bytes go to object storage, which scales without your involvement.
Rather than diffing whole file trees, each user has a monotonically increasing sequence of changes. A client stores its cursor and asks "what's happened since N?" This is cheap, incremental, and resumable — a client that's been offline for a month makes one request, not a full tree walk. It's the same idea as a replication log.
The notification service tells devices "something changed"; the device then pulls the delta. This keeps the notification path tiny and stateless, and means a missed notification is harmless — the next poll catches up. Long-polling is usually sufficient and cheaper than WebSockets here, since the update rate per user is low. Choosing long-poll over WebSockets with that reasoning is a good judgment signal.
You cannot merge two versions of a binary file, and silently picking a winner destroys someone's work. Detect conflicts by having the client send the base version it edited from; if the server's current version differs, it's a conflict. Then create a conflicted copy with both preserved and let the human decide. Last-write-wins is the wrong answer here and saying why is worth more than proposing a clever merge algorithm that can't exist.
Presigned URLs so bytes never traverse your servers, parallel chunk uploads to saturate bandwidth, and per-chunk retry so a dropped connection costs one chunk rather than the whole file. The client can also resume by re-running /prepare — the server tells it which chunks are still missing, which is inherently idempotent.
Chunks are shared, so you cannot delete a chunk when one file referencing it is deleted. Maintain a refcount, decrement on delete, and run a background GC for zero-refcount chunks after a grace period. Do the deletion lazily and conservatively — deleting a still-referenced chunk is unrecoverable data loss, so err toward retaining garbage.
Mark deleted, keep for 30 days, then purge. Users restore files constantly, and an immediate hard delete of a chunk that's still referenced by an old version is catastrophic.
A file shared from user A to user B lives on A's shard but must appear in B's tree. Store a lightweight reference in B's namespace pointing at A's file, and include shared-file changes in B's journal by having the share create a subscription. This is a fan-out problem again, bounded by the number of collaborators.
Compress chunks before upload where the content is compressible (skip already-compressed formats — detect by entropy or extension). Use a CDN for downloads of widely-shared files. Throttle background sync so it doesn't saturate a user's connection while they're working.