Primary concepts and the hard part
Requirements
Drivers online: 5M Location updates: every 4s → 5M/4 = 1.25M writes/sec ← the headline number Ride requests: 10,000/sec Location payload: ~100B → 125 MB/sec sustained
API / Model
High-level architecture
Two workloads meet in this design: a constant stream of driver positions through the Location Gateway, and rider requests that go through matching and become trips. The Redis geo index is where they connect.
- A rider's
POST /ridescreates a trip in the Ride Service with stateREQUESTEDand hands it to the Matching Service, which runs GEOSEARCH on the Redis geo index within 3 km, covering the target cell and its 8 neighbours. - It filters candidates by availability, vehicle type, rating and heartbeat.
- It ranks the rest by routed ETA from the Routing / ETA Service rather than by straight-line distance.
- It acquires a per-driver lock with
SET NX PXand a 30-second expiry, plus a fencing token, so the driver can't be assigned twice. - It offers the trip and waits 15 seconds. A reject or timeout moves it to the next candidate.
- Once a driver accepts, the Trip Service runs the state machine from
MATCHEDthroughARRIVINGandIN_PROGRESStoCOMPLETED, records each change intripsandtrip_events, and starts the saga on completion: charge, pay the driver, send the receipt.
The geo index is fed by the location paths. Every 4 seconds a driver's position reaches the Location Gateway, which overwrites that driver's entry synchronously, with a 30-second TTL that doubles as liveness, and sends the ping asynchronously to Kafka location.stream. Flink / Spark streaming turns that stream into ETA models and analytics and writes history to location_hist in S3. For drivers on an active trip only, the gateway also publishes to trip:{id}, throttled to one update every 2 seconds, which the rider's live map subscribes to.
Same product at 10x. At 100x there would be 500M drivers online, more than the world's taxi and courier workforce, so 10x (rides, food and parcels on one platform) is the realistic tier. Dashed outlines mark what's new or reshaped compared with today's design.
| Today | At 10x | |
|---|---|---|
| Drivers online | 5M | 50M |
| Location pings at a flat 4s | 1.25M/sec | 12.5M/sec |
| Ride requests | 10k/sec | 100k/sec |
| Raw location payload | 125 MB/sec | 1.25 GB/sec |
| Drivers in the densest metro | tens of thousands | hundreds of thousands |
What changes, and the number that forces it
- Ping rate adapts to what the driver is doing. At a flat 4-second interval, most of 12.5M writes/sec would come from parked drivers. Drivers on a trip or heading to a pickup ping every 2 seconds, idle and stationary drivers every 15, and the app sends immediately after a large position change. Most online drivers are idle at any moment, so this removes a large share of the writes without making any match worse.
- Persistent connections replace a POST per ping. At this rate, per-request overhead (TLS, headers, load-balancer work) costs more than the 100-byte payload. Drivers hold one connection to regional ingest, which also pushes trip offers back down the same socket.
- City shards split into S2 cells. The densest metros now have hundreds of thousands of drivers online, more writes than one Redis sorted set on one node can take. A cell router maps S2 cells to geo shards and splits a cell when it gets too dense, so a search over a cell and its neighbours may touch a few shards. Regional fault isolation still holds, because cells never span regions.
- Matching goes batched, one matcher per cell. At 100k ride requests/sec, overlapping candidate lists become the norm and most lock attempts collide. Each cell's matcher collects requests for ~2 seconds and solves the batch as one bipartite assignment, the approach from the batched-matching follow-up. That gives better global matches and removes contention inside the cell, since one matcher owns it. Locks are only needed for drivers a neighbouring cell's batch also wants. The cost is up to ~2 seconds added to time-to-match.
- The conditional write is still the arbiter.
WHERE driver_id IS NULLon the trip row stays the final word, so a race at a cell border costs a retry and never a double assignment. - Location history is downsampled. 1.25 GB/sec of raw pings mostly records cars that aren't moving. On-trip pings are kept at full rate for fares, disputes and ETA training; idle pings are thinned to about one per minute before landing in columnar files in object storage.
What stays the same
Current position lives in memory and is overwritten, never appended, with TTL as liveness. Candidates are still ranked by routed ETA rather than straight-line distance, only on-trip drivers publish live position to trip:{id}, and trip completion is still a saga with compensations.
Trade-offs and deep dives
Geohash encodes lat/lng into a string where nearby points share a prefix, turning a 2D proximity query into a 1D prefix scan any index can serve. The trap: two points can be 10 metres apart but sit either side of a cell boundary and share no prefix. Always query the target cell plus its 8 neighbours, then filter by true distance. Forgetting this produces a system that mysteriously can't find the driver parked across the street.
- Geohash: dead simple, works with any string-prefix index, fixed grid so dense cities and empty ocean get the same treatment.
- Quadtree: subdivides only where density is high, adapting to uneven distribution. Better for wildly varying density, more complex to maintain under constant updates.
- S2: projects the sphere onto a cube with a Hilbert curve; better locality, handles poles and the antimeridian correctly. What Uber and Google actually use.
For an interview, geohash with an explicit mention of S2 and the density caveat is a complete answer.
These writes are overwrites of ephemeral state with a useful lifetime of seconds. Durability is not required — if Redis loses a position, the driver publishes a new one four seconds later. Writing 1.25M/sec to a disk-backed store to hold data you'll throw away is the wrong trade. Persist to Kafka asynchronously for analytics, and keep the two paths separate.
Partition Redis by city or region. All matching queries are local to a region, so no cross-shard scatter-gather. This also gives you regional fault isolation for free: San Francisco going down doesn't touch London. Say this explicitly — it's a strong availability argument.
This is the part most candidates miss. Two rider requests in the same neighbourhood will surface overlapping candidate lists. Without a mutex, both get offered the same driver. Use a short-TTL distributed lock (SET lock:driver:{id} {token} NX PX 30000) around the offer window, release it on accept-elsewhere, reject, or timeout. The TTL is the safety net so a crashed matcher doesn't strand a driver forever.
Be ready for the honest caveat: Redis locks are not a correctness guarantee under pause-and-resume scenarios (a matcher can be GC-paused past its TTL while believing it holds the lock). The fix is a fencing token — a monotonically increasing number issued with the lock that the trip service checks, rejecting any assignment carrying a stale token. Alternatively, make the final assignment a conditional write on the trip row (UPDATE ... WHERE driver_id IS NULL), so the database is the arbiter of record. That conditional write is the simplest correct answer and worth offering.
Almost everything here is AP — driver positions, ETAs, nearby-driver lists can all be a few seconds stale with no harm. Exactly one operation needs strong consistency: the driver assignment. Isolating strong consistency to a single narrow operation, instead of applying it system-wide, is the mature design instinct.
A driver 500m away across a river is 15 minutes away. Rank candidates by routed ETA from a routing service, not by geohash distance. Use geo distance only to generate the candidate set cheaply, then rank properly on a small set. That two-stage pattern (cheap retrieval, expensive ranking) is the same shape as search.
Publishing all 5M drivers' positions to a pub/sub layer would be catastrophic. Only drivers on an active trip publish to a trip:{id} channel, and only the one rider subscribes. Throttle to ~1 update per 2 seconds; the map interpolates between points for smoothness. This turns an impossible fan-out into a fan-out of 1.
Charging the rider, paying the driver, and issuing a receipt span multiple services. Use a saga with compensating actions (refund on payout failure) rather than a distributed transaction, and make each step idempotent.
A TTL on the Redis entry means a driver whose app crashed silently disappears from the index within 30 seconds. No cleanup job, no stale offers to phones that aren't listening. TTL-as-liveness is an elegant detail worth calling out.