Index/Kafka

SponsorGitHub
Key technologyEvent log4 min

Kafka

A durable, replayable, partitioned log — the backbone of nearly every asynchronous design.

At a glance

Model
Append-only log; reading does not remove anything
Unit of everything
The partition: ordering, parallelism and assignment
Ordering
Within a partition only — the key decides the partition
Durability
acks=all with min.insync.replicas=2
Position
A consumer group's committed offset, so restarts resume
Retention
Time or size based; compaction keeps the latest per key

Key concepts and capabilities

The short listwhat it gives you
The partition is the unit — of ordering, of parallelism, and of assignment within a consumer group
The key picks the partition, so everything for one user or order stays ordered together
Consumer groups are independent — one write, many readers, each with its own offsets
Replay — retained records mean a new consumer can read last week, and a fixed bug can reprocess a range
At-least-once by default — a crash between handling and committing replays the record, so consumers are idempotent
Exactly-once holds only inside Kafka; the moment you write elsewhere, you need an upsert or a dedup key
Consumer lag is the health metric — the distance between the head and your offset, and the first thing to alert on
Log compaction keeps the latest record per key, turning a topic into a changelog you can rebuild state from
Buffering is the point — a fast producer and a slow consumer build lag instead of failing

Use cases

One write, several independent readers

The property that separates a log from a queue. Notifications, analytics and a search indexer each read the same partitions at their own pace with their own offsets, and adding a fourth consumer later costs the producer nothing.

FocusClick a node for details

Publishing database changes without a dual write

An event written to Kafka by the application after the database commit can be lost; one written before can describe a transaction that rolled back. The outbox row is written in the same transaction as the data, and a relay publishes it afterwards, so the two can never disagree.

primary keyforeign key → referenced columnHover a table or column to trace its keys
DatabaseQueue / streamFocusClick a node for details

Absorbing a spike the downstream cannot take

A producer writing 50k events/s into a consumer that handles 10k/s does not fail: lag grows and drains later. That decoupling of write rate from processing rate is the structural reason to put a log between two services.

DatabaseQueue / streamFocusClick a node for details

Retries and a dead letter topic

A record that always fails blocks its partition, and everything behind it stops. Retry with backoff a bounded number of times, then move it aside and keep going — and alert on the dead letter topic, because it is where silent data loss hides.

Queue / streamFocusClick a node for details