Primary concepts and the hard part
Requirements
500M DAU · 50M concurrent connections Messages: 100B/day → ~1.2M/sec avg, 4M/sec peak Msg row ~200B → 20 TB/day Connections per gateway node: ~50k → need ~1,000 gateway nodes
API / Model
→ client sends: {type:"send", client_msg_id, conv_id, body}
← server sends: {type:"message"|"receipt"|"presence", ...}High-level architecture
The design puts a stateless Chat Service between two stateful gateways: one holds the sender's socket and one holds the recipient's. A Redis connection registry links them, and every message is stored before anyone is told about it.
- Client A sends the message over WSS to WS Gateway #17, which holds A's socket and forwards it to the Chat Service. The Chat Service validates and authorizes it, assigns a Snowflake
message_id, dedupes onclient_msg_id, and writes it to the messages store, where it lands in theconversation_idpartition. - Only after that write succeeds does the Chat Service ack the sender.
- It then resolves the recipient in the connection registry, which maps
user_idto the gateway node holding that user's socket and expires stale entries through TTL and heartbeats. - If the recipient is online, the Chat Service publishes to that node's pub/sub channel,
gw:42, so only WS Gateway #42 receives the message. - WS Gateway #42 pushes it to Client B over B's socket.
When the recipient is offline, the message is written to inbox_queue as an undelivered row instead, and APNs / FCM sends a mobile push. When B reconnects, the gateway drains that backlog to the client, B acks, and the rows are deleted.
Same product at 10x the traffic. At 100x, 50M concurrent sockets would become 5B, more than the number of smartphones in use, so 10x (a WhatsApp-sized service) is the meaningful next tier. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Daily active users | 500M | ~3B |
| Concurrent connections | 50M | 500M |
| Messages at peak | 4M/sec | 40M/sec |
| Message storage | 20 TB/day | 200 TB/day |
| Gateway nodes | ~1,000 | ~10,000 |
| Regions | 1 | ~6 |
What changes, and the number that forces it
- Gateways: ~1,000 → ~10,000, in the user's nearest region. A socket held across an ocean adds ~150ms to every frame, and a reconnect storm there crosses continents. Each region runs its own gateway fleet behind L4 load balancers, sized for the reconnect storm after a regional failure rather than for steady traffic.
- Registry heartbeats move from users to gateways. With 500M entries on a 30-second TTL, per-user heartbeats alone would be ~17M registry writes/sec, a large fraction of the message rate. Each entry instead records its gateway's lease epoch. A gateway renews one lease every few seconds; when it dies, its epoch expires and every entry pointing at it is treated as stale at lookup. The registry is written only on connect and disconnect, and it's sharded per region.
- Every conversation gets a home region. Ordering needs a single place that assigns IDs for a conversation, so the Snowflake
message_idand the durable write happen in its home region, usually where it was created. Senders elsewhere forward to it and pay one inter-region hop before the ack, about 100ms, still inside the 500ms budget. Delivery to recipients in other regions goes over an inter-region bus, ordered per conversation. inbox_queueis replaced by cursor sync. At any moment most of 3B users are offline, and writing an inbox row for every undelivered message nearly doubles writes on the hottest path. Clients already track their highestmessage_idper conversation, so on reconnect they page the gap straight from the messages store, and a per-user unread pointer decides what to push. This is the "sync from cursor" follow-up promoted to the only offline path.- Message storage is tiered. At 200 TB/day, keeping every time bucket on hot Cassandra nodes is the storage bill. Buckets older than ~30 days move to cheaper columnar files in object storage, because nearly all reads are recent. Scrolling far back in history gets slower.
- Mobile push becomes its own pipeline. Hundreds of millions of offline recipients hit APNs and FCM rate limits. Pushes are collapsed per conversation ("12 new messages") and bulkheaded per provider, borrowing the notification design's delivery tier.
What stays the same
Durable before ack, client_msg_id idempotency, and ordering by server-assigned ID within a conversation. Gateways still hold only sockets, pub/sub still targets the one gateway holding a recipient rather than broadcasting, and very large channels are still pulled on open rather than pushed to every member.
Trade-offs and deep dives
Message order is defined by the server-assigned Snowflake ID, not by client timestamps (clocks lie, networks reorder). Because all messages in a conversation share a partition key, they are stored in one ordered log. Clients sort by message_id, which is monotonic per conversation. Ordering across conversations doesn't matter and isn't guaranteed.
The sender gets "sent" only after the message is committed to storage. If you ack on receipt at the gateway and the gateway dies, the message vanishes with a checkmark showing. Never ack before durable.
Client generates client_msg_id (a UUID). Retries on flaky mobile networks reuse it. Server dedupes, so a retry returns the original message_id rather than creating a duplicate. This is the single most important detail for mobile chat.
Broadcasting every message to 1,000 gateways is 1,000x write amplification on the pub/sub layer. The registry turns a broadcast into a targeted publish. Cost: an extra Redis lookup and a consistency window when a user reconnects to a different node (handled by TTL + the fact that a stale publish simply finds no socket and falls through to the offline path).
This violates the stateless ideal from Module 1. Contain it by keeping only the socket on the node; all durable state lives elsewhere. A gateway dying just means clients reconnect and re-register.
Slack channels with 50,000 members are the celebrity problem again. For small groups, push to each member. For very large channels, don't push to everyone — clients that have the channel open subscribe to a channel-level pub/sub topic, and everyone else pulls on open. Same hybrid shape as the feed.
Naive presence (broadcast every status change to everyone who might care) is a fan-out explosion. Mitigations: only compute presence for conversations the user currently has open, batch updates on a few-second interval, and treat presence as best-effort/lossy. Say out loud that presence is the most over-engineered part of most chat designs.
Store last_read_msg_id per (user, conversation) rather than a per-message read flag. One row updated instead of N. Unread count = count of messages with id > last_read_msg_id, which is a bounded range scan, or a cached counter.
A busy channel is a hot partition. Bound partition size by bucketing the key: PK = (conversation_id, time_bucket). Reads for recent history hit the newest bucket.
If required, the server stores ciphertext only and cannot do server-side search, previews, or moderation. Say this trade explicitly — it removes features, not just adds security.
Possible follow-up questions
message_id client-side. At-least-once + idempotent rendering.message_id per conversation and sends it on reconnect; server replies with everything after it. This "sync from cursor" model is more robust than relying on the inbox queue alone.user_id → [list of connections], not one. Deliver to all; read state syncs server-side so all devices converge.