Index/Flink

SponsorGitHub
Key technologyStream processor3 min

Flink

Stateful computation over unbounded streams: windows, joins and aggregates that survive a crash.

At a glance

Shape
A graph of operators: source → keyBy → window → sink
State
Managed, keyed, RocksDB-backed — terabytes if needed
Fault tolerance
Checkpoints: operator state plus source offsets, restored on failure
Time
Event time, with watermarks deciding when a window is done
Guarantee
Exactly-once state; end to end only with a cooperating sink
Upgrades
Savepoints: stop, change the job or parallelism, restore

Key concepts and capabilities

The short listwhat it gives you
Event time, not processing time — count an event in the minute it happened, not the minute it arrived
Watermarks assert that nothing older is still expected, and are what make a window fire
Late data can be dropped, given a grace period, or routed to a side output for reconciliation
Windows — tumbling (per minute), sliding (last 5 minutes, every minute) and session (gaps of inactivity)
Keyed state — a running count or session per key, held in RocksDB on local disk, checkpointed to durable storage
Checkpoints snapshot every operator plus the source offsets, so recovery rewinds and replays
Savepoints are the manual version: stop, change parallelism or code, restore
Exactly-once needs a cooperating sink — transactional two-phase commit, or an idempotent write
Backpressure propagates — a slow sink slows the source and grows Kafka lag instead of dropping data

Use cases

Clicks per campaign per minute, counted honestly

The canonical job. Events are keyed by campaign, windowed on event time, and the window fires when the watermark passes its end — so a phone that was offline for ten minutes still lands in the minute it clicked, not the minute it reconnected.

DatabaseQueue / streamFocusClick a node for details

Attributing a click to its impression

A stream-to-stream join over a time window: hold impressions in keyed state for 30 minutes, and when a click arrives for the same ad and user, emit the pair. State that expires is why this is a stream processor's job and not a database query.

DatabaseQueue / streamFocusClick a node for details

Turning raw events into sessions

A session window groups a user's events until they go quiet for N minutes, which is how a raw click stream becomes "a browsing session" — the unit product analytics actually wants, and one that no fixed window can express.

DatabaseQueue / streamFocusClick a node for details

Surviving a crash, and a traffic doubling

Checkpoints are what make a stateful job restartable: a barrier flows through the graph and snapshots every operator's state with the offsets that produced it, so recovery is "restore and rewind". The same mechanism, triggered by hand, is how you rescale.

DatabaseFocusClick a node for details