Primary concepts and the hard part
Requirements
Docs: 1B
Concurrent edit sessions: 1M
Ops per active editor: ~5/sec while typing
Peak op rate: a few M/sec across the fleet, but each doc
is independent → shard by doc_id
Op size: ~100B; a doc's op log can reach millions of entries
→ SNAPSHOTS are mandatoryAPI / Model
→ client sends: {type:"op", doc_id, base_version, op, client_id, seq}
← server sends: {type:"op", version, op, origin_client}
{type:"ack", client_seq, version}
{type:"presence", user_id, cursor, selection}High-level architecture
Each client edits its own copy immediately and syncs through the one Document Session Server that owns the document. WS Gateways carry ops in and out, and the operations log is the durable record everything else is rebuilt from.
- Client A applies the keystroke locally at once and keeps the op, with its
base_version, in its pending buffer. The op travels over the WebSocket to the WS Gateway, which routes it bydoc_idto the Document Session Server that owns the document, and the sequencer there assigns it the next monotonic version. - If the op was based on an older version, the transform step adjusts it against the ops applied since, shifting its positions so it still lands where the user meant.
- The session server persists the op to the append-only operations log, keyed by
doc_idandversion. - It broadcasts the transformed op through the WS Gateway, whose connection registry maps
doc_idto connected clients, and acks Client A, which drops the op from its pending buffer. - Client B transforms the incoming op against its own pending buffer and applies it locally.
Two background flows keep the log usable. Every N ops, a snapshot job writes the folded document to object storage, so a load reads the latest snapshot plus the ops after it. If the owner dies, a new owner is elected, rebuilds from snapshot and ops, and clients resend their unacked ops. Presence takes its own path: cursors and selections live in Redis with a TTL and reach collaborators through the gateway, never through the operations log.
Same product at 10x. Documents are still independent, so raw session count is the easy part; what 10x really adds is users in every region and documents with audiences today's design never planned for. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Concurrent edit sessions | 1M | 10M |
| Documents | 1B | ~10B |
| Op rate at peak | a few M/sec | tens of M/sec |
| Largest live audience on one doc | ~50 editors | ~100 editors + 10k viewers |
| Regions | 1 | several |
What changes, and the number that forces it
- Viewers stop costing the owner. A company-wide doc with 10k people watching would make the owning server write every op to 10k sockets. Editors still receive every op directly, but viewers read from a per-document pub/sub topic that gateways subscribe to, receiving ops batched every ~200ms. The owner's work now grows with the number of editors, not the size of the audience.
- Ownership lives near the editors. With users in every region, an owner on another continent adds a long round trip to every remote edit and ack. Optimistic local application hides it while typing, but collaborators' changes arrive late. The owner runs in the region closest to most active editors and hands off when that majority moves: flush, release the lease, and the new owner loads snapshot plus ops. It's the existing failover path, run on purpose.
- Leases replace per-document elections. Running leader election for each of millions of open documents is a lot of coordination. A regional lease service maps
doc_idto a session server; ownership is taken on first open and dropped once the doc goes idle, so the ~10B mostly idle documents cost nothing. A fencing epoch on the lease keeps a paused old owner from writing after a handoff. - Op writes are group-committed. Tens of millions of ops/sec as individual appends would be tens of millions of log writes. Each session server batches ops from every document it owns into one write every few milliseconds, then acks, well inside the 200ms remote-visibility budget.
- Snapshots follow activity. Snapshotting every N ops still applies, plus a snapshot whenever a doc goes idle, so the next open, possibly on a different owner in a different region, loads quickly.
- Presence is sampled. Ten thousand viewer cursors is noise, not collaboration. Large audiences show editors' cursors plus a count ("and 9,400 others"), and presence stays in each region's Redis without ever crossing regions.
What stays the same
Edits apply locally at 0ms. There is still one authoritative sequencer per document, convergence still comes from OT (or a CRDT), the op log is still append-only and never garbage-collected, snapshots remain an optimization on top of it, and presence stays deliberately lossy. The correctness core is untouched; 10x only changes who pays for the audience and where the owner sits.
Trade-offs and deep dives
If two users type into the same paragraph and you resolve by timestamp, one person's sentence silently vanishes while they're looking at it. In a document editor that's not an edge case, it's the normal flow. The requirement is that both edits survive and both users see the same result. That rules out every simple conflict-resolution strategy and is why OT and CRDTs exist.
If a keystroke waits for a server round trip, typing feels broken at even 50ms of latency. So the client applies immediately, buffers the op as unacknowledged, and reconciles when the server responds. Everything else in the design exists to make that optimism safe.
Each operation carries the document version it was based on. When an op arrives based on a stale version, the server transforms it against every op that has been applied since, adjusting positions so the intent is preserved. An insert before your position shifts you right; a delete before your position shifts you left. The client performs the mirror-image transformation on incoming ops against its own pending buffer.
The honest caveat, worth volunteering: OT is notoriously difficult to implement correctly. The transformation functions must satisfy convergence properties (TP1/TP2) that are easy to get subtly wrong, and the number of cases grows with the richness of the operation set. Google Wave's OT implementation was famously hard. Acknowledging this is more credible than presenting OT as straightforward.
Conflict-free Replicated Data Types assign each character a unique, densely-ordered identifier (a fractional index or a path in a tree), so operations are commutative by construction. Apply them in any order and you converge — no central sequencer, no transformation.
| OT | CRDT | |
|---|---|---|
| Needs central server | yes (sequencer) | no — works peer-to-peer |
| Metadata overhead | low (ops are small) | high (per-character IDs; tombstones for deletes never fully go away) |
| Implementation | complex transform matrix | complex data structure, but composable |
| Used by | Google Docs, Etherpad | Figma, Automerge, Yjs, most newer tools |
The recommendation to state: OT when a central server already exists and document size/memory matters; CRDT when you want offline-first, peer-to-peer, or simply want to avoid the transformation minefield. For a Google Docs clone with a server in the loop, either is defensible — what matters is that you can articulate the trade rather than declaring one universally better. Newer systems trend toward CRDTs because the tooling has matured.
Route all connections for a document to a single session server via consistent hashing on doc_id, with leader election so exactly one node owns it at a time. This gives a single point that assigns the total order. Without it, two servers could sequence conflicting ops independently and you'd need a much harder distributed agreement.
The failure question follows immediately: if that node dies, a new owner is elected and rebuilds state from the persisted op log (snapshot plus subsequent ops). Clients reconnect, send their unacked ops with base versions, and the new owner transforms and applies them. Brief unavailability for one document, and no data loss because the log is durable. Sharding by document means the blast radius is one document, not the platform.
A long-lived document accumulates millions of ops. Replaying all of them to open the file would take forever. Snapshot the materialized content every N ops (or on a timer), store it in object storage, and load as snapshot + ops since snapshot. This is standard event-sourcing practice, and forgetting it is a common gap.
Cursor positions update many times per second and are worthless a second later. Keep them in Redis with a TTL, broadcast at a throttled rate (a few times per second, not per keystroke), and never persist or order them. Treating presence with the same rigor as document ops would multiply your load for zero benefit — and saying so shows you're allocating engineering effort by value.
The client queues ops locally with their base version. On reconnect it sends the queue; the server transforms each against everything that happened meanwhile. This works well for short absences. For long ones the transformation chain gets expensive and the result can be semantically surprising even if technically convergent — at which point offering the user a "review changes" step is more honest than silent auto-merge. CRDTs handle this case more gracefully, which is a fair reason to prefer them if offline-first is a core requirement.
Undo must be local — undoing your own last edit, not whoever edited most recently. That means maintaining a per-user undo stack whose entries are themselves transformed as other people's ops arrive, since the thing you want to undo may have moved. Worth raising as a known complexity rather than assuming it's free.
Falls out of the append-only log for free. Restoring a version is not a rewind — it's appending the ops that transform current state back to the target state, preserving history. Never mutate or truncate the log.
Possible follow-up questions
doc_id and scale horizontally. The per-document workload is small; the fleet just needs enough capacity to own many documents.