Index/Elasticsearch

SponsorGitHub
Key technologySearch index3 min

Elasticsearch

An inverted index with relevance ranking — a derived view of your data, never the source of truth.

At a glance

Model
Inverted index: term → the documents containing it
Unit of scale
Shards, each a Lucene index, plus replica shards
Shard count
Fixed at creation; changing it means a reindex
Freshness
Near real time — searchable after a refresh, ~1s
Storage
Immutable segments; updates write a new doc and merge later
Role
Derived index. The source of truth lives somewhere else

Key concepts and capabilities

The short listwhat it gives you
Inverted index — every term maps to the documents holding it, so a match is a lookup and an intersection
The analysis chain — tokenise, lowercase, remove stop words, stem, at index and query time; mismatches return nothing
BM25 relevance — rare terms score higher, long documents lower; tune by boosting fields and decaying by recency
Aggregations — facets, histograms and percentiles over the matching set, which is what log dashboards are
Edge n-grams index every prefix at write time, which is how typeahead is a plain term lookup
Deep paging has a cliff — from: 10000 sorts 10,000 hits per shard; use search_after with a cursor
No joins — denormalise, and copy the customer name into the order document when it changes
Aliases — applications query an alias so a rebuilt index can be swapped in atomically
It is derived — fed from a change stream, rebuildable from scratch, and never the place a write lands first

Use cases

A search index kept in sync by a change stream

The architectural claim that earns marks: writes go to the database, and the index is fed from its change stream by an idempotent consumer. Dual-writing from the application is the anti-pattern being listened for, and a reindex job is what makes a bad mapping or a dropped message recoverable.

DatabaseQueue / streamFocusClick a node for details

Typeahead in fifty milliseconds

Edge n-grams index "sys", "syst", "syste" at write time, so a prefix query is a plain term lookup rather than a wildcard scan. Put a cache of popular prefixes in front and the common case never reaches the cluster at all.

CacheFocusClick a node for details

Faceted search over shards

A query fans out to every shard, each returns its top hits and its slice of the aggregations, and the coordinating node merges them. That is why facets are cheap and page 500 is not: every shard has to sort everything before the offset.

FocusClick a node for details

Changing a mapping without downtime

Mappings are close to immutable, so you do not change one: you build a new index beside the old one, backfill it from the source of truth, and flip the alias in a single atomic step. The application never learns the index name.

FocusClick a node for details