Primary concepts and the hard part
Requirements
Creates: 100M/day → 100M / 10^5 = 1,000/sec (peak 3x = 3,000/sec) Reads: 100:1 ratio → 100,000/sec Storage: ~500B/record × 100M/day = 50 GB/day → 18 TB/yr → ~90 TB over 5 yrs Bandwidth: 100k/sec × 500B = 50 MB/sec Key space: base62^7 = 3.5 × 10^12 → 100M/day for ~95 years. 7 chars is enough.
API / Model
Note the partition key: short_code is effectively random (base62 of a hashed/encoded counter), so it distributes perfectly with no hot-partition risk. That's a rare gift — say so.
High-level architecture
The shortener is two paths joined by one Redis cluster: a write path at about 1k requests per second that creates links, and a read path at about 100k that redirects them. A separate analytics pipeline hangs off the redirect.
- A client's
POST /v1/urlspasses the API Gateway, which applies auth and an abuse rate limit, and reaches the Shorten Service. - ID generation takes the next number from a range of 10,000 that the ticket server handed out, scrambles it, and encodes it as a base62
short_code. - The urls store saves the row with a conditional write on
short_code, and write-through copies the mapping from code tolong_urlinto the Redis cluster. - Later, a browser's
GET /{code}goes through the Load balancer to the Redirect Service, which is stateless and autoscaled and looks the code up in the Redis cluster. - About 95% of lookups hit and return 302 Found straight away. The other 5% read the urls store read replicas, populate the cache, and then return the 302.
- The browser follows the redirect to the destination site.
Every 302 also fires a click event to Kafka and returns without waiting for it. A stream processor groups those events into tumbling windows and writes to two places: clicks_agg, which serves the stats API, and the data warehouse, which keeps raw analytics.
Same product at 100x the traffic. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 100x | |
|---|---|---|
| Creates at peak | 3,000/sec | 300,000/sec |
| Redirects | 100k/sec | 10M/sec |
| Storage over 5 years | ~90 TB | ~9 PB |
| Codes created | 100M/day | 10B/day |
| 7-char key space lasts | ~95 years | under 1 year |
| Where redirects are answered | a few regions | ~300 edge locations |
What changes, and the number that forces it
- Codes grow from 7 to 8 characters. At ~10B creates/day, the 3.5 trillion 7-char codes run out in under a year. 62^8 is ~218 trillion, about 60 years at this rate. Existing 7-char links keep working, because a code's length is part of the code.
- Redirects move to the edge. 10M redirects/sec served from a few regions can't hold a 50ms p99 for users far from them. An edge worker in each CDN point of presence answers from a small edge key-value store of hot links. Unlike caching the 302 response itself, the worker runs on every click, so analytics survive: this resolves the "should the CDN cache redirects?" trade-off instead of giving something up.
- Clicks are batched at the edge. 10M events/sec sent one at a time would double the edge's own request volume. Workers buffer ~100ms of clicks and ship batches to the regional Kafka, dropping them if the region is unreachable, because redirect availability still outranks analytics completeness. Viral codes get salted partition keys so one link can't pin one partition.
- ID ranges are carved per region. Creates go to the nearest region, and a round trip to one global ticket server would put an ocean on the write path. Each region's ticket server hands out ranges from its own prefix, so codes stay unique with no cross-region coordination.
- Storage becomes a sharded KV store with a cold tier. ~9 PB over five years is well past "plan for partitioning". The urls table lives in a multi-region key-value store hash-partitioned on
short_code, which is still a perfectly uniform key. Links nobody has clicked in a year move to object storage and are restored on their first miss. - Not-found answers stop at the edge. At this volume, scanners guessing codes are real load. Caching "not found" for a short TTL at the edge, plus per-IP limits on 404s, keeps enumeration from reaching the regions at all.
What stays the same
302 over 301, a scrambled counter so codes can't be enumerated, analytics that never block a redirect, a conditional write for custom aliases, and TTL expiry returning 410. The 10x follow-up already says this design scales almost linearly; at 100x each piece just moves closer to the user, and the key space gets one more character.
Trade-offs and deep dives
| Approach | How | Why / why not |
|---|---|---|
| Hash the URL (MD5 → take 7 chars) | deterministic, no coordination | Collisions are inevitable at 10^11 records; requires collision check + retry loop, and identical URLs map to the same code (sometimes desired, sometimes a privacy leak) |
| Counter + base62 | central or ranged counter, encode | No collisions ever, shortest possible codes. But sequential IDs are enumerable — scrape every link by counting up |
| Snowflake + base62 | 64-bit local ID | No coordination, no collisions, but 64 bits → 11 base62 chars, longer than needed |
The pragmatic answer: ticket-server ranges (each app node pre-fetches 10,000 IDs and increments locally, so coordination happens once per 10,000 creates, not per create), then scramble the counter before encoding — XOR with a secret, or apply a Feistel permutation — so codes are unguessable while collisions remain impossible. Gaps from unused ranges when a node dies are harmless.
[0-9a-zA-Z] = 62 symbols, all URL-safe, no escaping. 62^7 ≈ 3.5 trillion. Consider excluding visually ambiguous characters (0/O, 1/l/I) if links are ever typed by hand — that drops you to base58 and you should mention the trade.
- 301 Permanent: browsers cache it aggressively, so repeat clicks never touch your servers. Great for load, but you lose all analytics after the first click, and you can never change the destination.
- 302 Found: every click hits you. You keep analytics and can update or expire links.
Choose 302 for an analytics product. Say why.
Only with short TTLs, and only if you accept losing per-click analytics for cached hits. Most shorteners skip CDN caching of the redirect itself for exactly this reason — the analytics are the product. This is a nice place to show that a technically-better-performing option can be the wrong product choice.
Fire the click event to Kafka and return the 302 immediately. The user is waiting on a page load; they must not wait on an analytics write. If Kafka is down, drop the event and serve the redirect — availability of the redirect outranks completeness of analytics. That's a deliberate, statable priority.
Cache-aside with LRU + TTL. Write-through on create so a freshly-created link is warm (people click their own link immediately after shortening). Hit rate is the primary metric; it should sit above 90%.
are the one place you need strong consistency: two users must not both claim /sale. Use a conditional write (IF NOT EXISTS) rather than read-then-write, which races.
Don't scan for expired rows. Set a TTL on the storage row (Cassandra/DynamoDB do this natively) and check expires_at at read time. Return 410 Gone rather than 404 so it's distinguishable.
Possible follow-up questions
short_code — already uniform, no hot partitions, and every lookup is a point query carrying the partition key. This is the easy case; contrast it with a design where the natural key is skewed.(domain, code); store the domain→tenant mapping and validate TLS via a certificate management service.