Primary concepts and the hard part
Requirements
Events: 5,000/sec avg, 50,000/sec peak (a big repo goes viral, CI storm) Fan-out: ~10 recipients avg; "celebrity repos" up to 500k watchers Deliveries: 50k-500k/sec at peak Records: ~0.5 KB → ~2 TB/day
API / Model
High-level architecture
Events pass through two Kafka topics: activity.events feeds the Fan-out service, and delivery.jobs feeds delivery workers that are bulkheaded per provider. The notifications store and the Read API sit beside that pipeline as each user's inbox.
- The PR Service, CI Service, Comments and Issues publish to Kafka
activity.events, partitioned byentity_id. The Fan-out service consumes it and resolves the audience: watchers, mentions and participants. - It checks whether the event belongs to a celebrity repo with more than 100k watchers. If it does, the service writes nothing per user and marks the event for read-time merge.
- For every other event, it filters the audience by mutes and preferences.
- It collapses duplicates, so ten likes become one "10 people liked your post" notification.
- It makes one batched, idempotent write keyed by
user_idandevent_idto the notifications store, and puts one job per channel on Kafkadelivery.jobs. - The In-app worker, Push worker and Email worker each take their own jobs. The in-app worker sends through Redis pub/sub to the WS gateway, the push worker checks quiet hours before calling APNs / FCM, and the email worker buffers digests before calling the SMTP provider.
Failed provider calls go to the retry stage, which backs off exponentially with jitter behind a circuit breaker per provider. Jobs that run out of attempts land in the dead letter queue, which alerts on depth and is replayed after a fix. On the read side, the Read API pages through the notifications store by cursor, merges celebrity-repo events at read time, and takes badge counts from the Redis unread counters.
Same product at 10x the traffic. At 100x the peak would be 50M deliveries/sec, far beyond what email and push providers accept from anyone, so 10x is the tier where the design is still recognisably this one. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Events at peak | 50k/sec | 500k/sec |
| Deliveries at peak | 500k/sec | 5M/sec |
| Notification records | ~2 TB/day | ~20 TB/day |
| Biggest repo | 500k watchers | 5M watchers |
| Read-time merge threshold | 100k watchers | 20k watchers |
What changes, and the number that forces it
- One topic becomes two, split by urgency. At 500k events/sec, a CI storm or a viral repo can put minutes of backlog on a single topic, and a direct @mention waits behind it. The ingest router tags each event at the source: mentions and review requests go to a small, latency-sensitive topic with its own fan-out pool, and watch activity goes to a large topic that is allowed to lag. Priority isolation becomes part of the topology.
- Fan-out splits by audience size. Small audiences go straight through. Mid-sized ones (1k to 20k) are chunked into tasks of at most 10k recipients, so no single job holds a partition hostage. And the read-time merge threshold drops from 100k watchers to 20k: at 5M deliveries/sec, eager fan-out for mid-size repos is what saturates the workers first, just as the 10x follow-up predicts.
- Collapse windows stretch under backlog. When consumer lag rises, the collapse window for low-priority types grows from minutes toward an hour, so a storm produces "38 new comments on PR #412" rather than 38 notifications and 38 emails. Lag is the input signal, and direct notifications never stretch.
- Email goes through a mail router. Tens of millions of emails a day exceed what one provider will accept, and mailbox providers throttle per sending domain and IP reputation. A mail router spreads traffic across several providers and warmed dedicated IP pools, applies per-destination-domain throttles, and switches heavy recipients to digests by default. One provider's outage or throttling now shifts traffic instead of filling the DLQ.
- The inbox gets bounded. 20 TB/day kept forever is a storage bill for notifications nobody opens. Partitions become
(user_id, month), so a heavy user's inbox isn't one enormous partition, and rows expire after 90 days via TTL. - In-app delivery needs a registry. With millions of open tabs, one Redis pub/sub can no longer carry every user's channel. A connection registry maps each user to their gateway, the same shape as the chat design, and unread counters move to a sharded Redis Cluster reconciled shard by shard.
What stays the same
At-least-once delivery with the (user_id, event_id) idempotency key, preferences checked at delivery time rather than fan-out time, bulkheads and circuit breakers per provider, retries with backoff into an alerting DLQ, and Kafka replay after a consumer bug. The pipeline shape is unchanged; it just gains lanes.
Trade-offs and deep dives
Replay. If the fan-out consumer ships a bug that drops a category of notification, you reset the offset and reprocess. If you add a fifth delivery channel next quarter, it reads history from day one. A queue that deletes on consume gives you none of that.
Say it precisely: at-least-once delivery with idempotent consumers, producing exactly-once effects. The dedupe key is (user_id, event_id) as the storage primary key for the notification, and (notification_id, channel) in the delivery log so a retried job that already sent an email doesn't send a second one.
If a user mutes a thread after the fan-out ran but before the email worker fires, the mute should take effect. Filtering at write time bakes in a stale decision. Small detail, real product consequence.
Ten likes should be one notification. Implement as a short time-windowed aggregation keyed by (recipient, type, entity): hold for N minutes, merge arrivals, then emit. This is a windowed stream operation, and it also protects the email provider from a burst.
APNs, FCM, and SMTP each get their own worker pool, their own queue, and their own circuit breaker. When SMTP is degraded, push notifications must keep flowing. Sharing a thread pool across providers means one bad provider stalls everything — this is the bulkhead pattern and it's worth naming.
A CI storm generating 100k email jobs will get you throttled or blocked by your email provider. Two mitigations: batch into digests (one email covering many events) and apply a leaky bucket in front of the provider so output is smooth regardless of input burstiness.
Buffer events per user, and a scheduler fires hourly/daily to render and send one email. Requires the user's timezone so "daily at 9am" means their 9am.
Never push at 3am local; cap pushes per user per day. Needs per-user timezone and a counter. Cheap to implement, big product win, and shows you're thinking about the human on the other end.
Counting rows on every page load is a range scan per request. Maintain a Redis counter with atomic INCR/DECR, accept that it drifts (missed decrements, race conditions), and run a periodic reconciliation job against the store. Say the drift is acceptable because this is an AP system — the failure mode is a badge showing 3 instead of 2.
Same hybrid as the feed: above a watcher threshold, don't fan out. Store the event once, and merge it in at read time for users who watch that repo. One cached "recent events for repo X" list serves all 500k watchers.
A 500k-recipient job sitting in a partition starves ordinary notifications behind it. Either chunk large jobs into sub-tasks or route them to a separate low-priority queue. Priority isolation.
Kafka consumer lag (the leading indicator), DLQ depth (alert, not dashboard), per-provider delivery success rate and latency, fan-out worker error rate, and end-to-end p99 from event emitted to in-app delivery.
Possible follow-up questions
event_id; if both paths generate a notification, the unique constraint on (user_id, event_id) collapses them. Prefer the higher-priority reason when choosing the display text.delivery.jobs and a preference field. Nothing upstream changes — that's the payoff of event-driven architecture.