Primary concepts and the hard part
Requirements
robots.txt and crawl-delayPages to crawl: 10B
Target rate: 10,000 pages/sec → 10B / 10k = ~11 days for a full pass
Avg page: ~100 KB HTML → 1 GB/sec ingest → ~85 TB/day raw
URLs seen (dedupe): 100B+ URLs → storing them as strings ≈ 10+ TB
→ Bloom filter at 10 bits/URL ≈ 125 GB, fits in RAM across a clusterAPI / Model
Internal system, so the "API" is the queue contracts:
High-level architecture
The crawler is a loop around the URL Frontier: URLs leave it, pages are fetched and parsed, and new links go back in. Politeness lives inside the frontier, and duplicates are caught twice, once by content and once by URL.
- Seed URLs enter the front queues, which order work by priority (Q1 high for news and homepages, Q2 medium, Q3 low) through a weighted selector.
- URLs then move to the back queues, one per host. A heap keyed by
next_allowed_fetch_timelets a worker pop a URL only when its host is ready. - Consistent hashing by domain sends all of a host's URLs to the same node in the fetcher fleet, which reads the robots.txt cache for rules and crawl-delay, and the DNS cache for addresses.
- The fetcher makes the HTTP request with a timeout, a size cap and a redirect limit, and sends ETag / If-Modified-Since so an unchanged page costs only a 304.
- Content dedupe checks a checksum for exact duplicates and a simhash for near duplicates, and the page is written to the page store as compressed HTML.
- The parser / link extractor pulls out links and normalizes each URL: lowercase host, strip fragment, sort query params, resolve relative paths.
- The Bloom filter checks every normalized URL. A URL that is definitely not seen goes back into the front queues, and one that is probably seen is discarded.
Two jobs feed the frontier from the side. Trap detection enforces a maximum depth and a per-domain cap and catches calendar and session-id patterns, so runaway URL spaces don't flood the queues. The recrawl scheduler estimates how often each page changes and puts it back into the front queues at its next_crawl_at.
Same crawler at 100x the fetch rate: a full pass over 100B known pages in about a day instead of eleven days for 10B. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 100x | |
|---|---|---|
| Fetch rate | 10k pages/sec | 1M pages/sec |
| Known pages | 10B | 100B |
| URLs seen | 100B+ | ~1T |
| Bloom filter at 10 bits/URL | ~125 GB | ~1.25 TB |
| Raw ingest | ~85 TB/day | ~8.6 PB/day |
| Full pass | ~11 days | ~1 day |
What changes, and the number that forces it
- The frontier is sharded along with its hosts. One global heap popping 1M URLs/sec becomes a contention point before anything else, as the 10x follow-up warns. Each crawler node owns a slice of hosts by consistent hashing and holds their frontier queues, politeness state and seen-set together. Popping a URL, checking politeness and checking "seen" are all local memory operations.
- The seen-set shards with the frontier. ~1T URLs at 10 bits each is ~1.25 TB, too much to sit behind a network call for every extracted link. Partitioning the Bloom filter by host puts ~1 GB on each node, right next to the frontier that needs it.
- Discovered links are shuffled in batches. Most links point at other hosts, which belong to other nodes. The parser groups discovered URLs by owning node and ships them in batches, and the owner does the Bloom check and enqueue. It's a MapReduce-style shuffle, not a network call per link.
- Politeness counts IPs, not just hostnames. At 1M pages/sec, thousands of small sites on one shared-hosting IP would each get their own "one request at a time" budget and together overwhelm the machine. Budgets apply per host and per IP, and error rates trigger backoff per ASN.
- Crawling runs in several regions. Fetching a Brazilian site from Virginia adds latency to every request and sees the wrong geo-served content. Hosts are assigned to the region closest to their servers, and each region runs its own recursive resolvers that resolve ahead of fetch time. DNS is otherwise the first thing to break.
- Only changed content is stored. 8.6 PB/day of raw HTML is mostly pages that didn't change since the last crawl. When a recrawl's simhash matches the stored version, only a pointer and the fetch time are recorded; changed pages are written as compressed WARC files to object storage.
What stays the same
The two-level frontier that makes politeness structural, consistent hashing by domain, a Bloom filter that never re-crawls a seen URL, URL normalization, trap detection, conditional requests that make unchanged pages nearly free, and JavaScript rendering on its own smaller fleet. The loop is identical; it's just partitioned so that almost nothing in it crosses the network per URL.
Trade-offs and deep dives
A naive FIFO queue will happily hand ten workers ten URLs from the same domain simultaneously, which is a denial-of-service attack against that site and gets you blocked. The two-level structure fixes this: front queues encode priority, back queues encode politeness (one per host), and a heap ordered by next_allowed_fetch_time means a worker can only ever pop a URL whose host is ready. Politeness becomes a structural property rather than a check you might forget.
Route all URLs for a host to the same worker node. Now politeness state (last fetch time, crawl delay, robots rules) is local — no distributed coordination per fetch. Consistent hashing means adding or removing a node moves only ~1/N of domains rather than reshuffling everything.
You need "have I seen this URL?" over 100B URLs. Storing them costs 10+ TB. A Bloom filter at ~10 bits per element costs ~125 GB and answers in O(1) from RAM. Its guarantee is asymmetric: no false negatives, ~1% false positives. Translated to this problem: you will never re-crawl a page you've already crawled (which would be a correctness/politeness problem), but you will occasionally skip a page you haven't (which costs you ~1% coverage). That's the right direction for the error to point, and saying so demonstrates you understand why the structure fits rather than just naming it.
The limitation to mention: you can't delete from a Bloom filter, so a URL can never be "un-seen". Handle recrawls through the scheduler, not by removing from the filter.
Example.com/Page?b=2&a=1#section and example.com/page?a=1&b=2 are the same resource. Without normalization (lowercase host, drop fragment, sort query params, resolve relative paths, strip known tracking params), you'll enqueue millions of duplicates and your Bloom filter fills with noise.
Different URLs frequently serve identical content (mirrors, print views, session IDs in the path). Exact checksums catch identical bytes; simhash catches near-duplicates where only boilerplate differs. Detecting near-duplicates is what keeps the downstream index from being 40% junk.
Infinite calendars (/calendar?date=2099-12-31 linking to the next day forever), session IDs generating unlimited unique URLs, and deliberately deep link structures. Defenses: cap URL depth and length, cap pages per domain, detect repeating path patterns, and monitor the ratio of new-URLs-discovered to useful-content-found per domain. A domain generating a million URLs and no new content is a trap.
Resolution is a synchronous network call that can take 50-200ms, and at 10,000 fetches/sec a naive crawler spends most of its time in DNS. Cache aggressively (respecting TTLs), run your own resolvers, and resolve asynchronously ahead of fetch time.
Send If-Modified-Since/If-None-Match on recrawls. A 304 Not Modified costs almost no bandwidth and confirms freshness. For a recrawl-heavy workload this is a large efficiency win.
Not all pages change at the same rate. Estimate change frequency from observed history (a news homepage changes hourly; a 2009 forum post never will) and set next_crawl_at accordingly. Budget crawl capacity between discovery (new pages) and freshness (recrawls) — they compete for the same fetchers, and how you split is a product decision.
Respect robots.txt and its Crawl-delay. Send a descriptive User-Agent with a contact URL. Back off automatically on 429/503 responses and rising error rates from a host. Being a well-behaved crawler is not just etiquette — sites block badly-behaved ones, which costs you coverage.
Many modern pages are empty without executing JS. Rendering requires a headless browser, which is roughly 10-100x the CPU and memory of a plain fetch. The practical answer is a two-tier approach: plain fetch for everything, and route only pages detected as JS-dependent (empty body, known SPA frameworks) to a smaller rendering fleet. Flag the cost explicitly rather than hand-waving it.
If parsers fall behind fetchers, the frontier grows unboundedly. Monitor queue depth and throttle fetchers when it exceeds a threshold. The frontier is also persistent — it must survive restarts, so it's backed by durable storage, not just memory.