Index/Object storage

SponsorGitHub
Key technologyBlob store4 min

Object storage

S3-style storage for bytes: effectively unlimited, extremely durable, and never in the request path.

At a glance

Model
HTTP key-value store for immutable blobs; no directories
Durability
~11 nines, from erasure coding across facilities
Capacity
No ceiling to plan for; cost an order below block storage
Latency
Tens of milliseconds — never in a hot request path
Consistency
Read-after-write for new objects and overwrites
Queries
None. Listing a prefix is slow and paginated

Key concepts and capabilities

The short listwhat it gives you
Presigned URLs — the client uploads and downloads directly, so a 2 GB video never touches your API
Multipart upload — parts in parallel, retried individually, completed as one object; this is what makes resume work
Range requests — read a byte range without fetching the object: video seeking, one chunk of a file, resuming
Events on change — an upload emits a notification, which is the standard trigger for scanning and transcoding
Lifecycle rules move objects to colder tiers after N days and delete them after M — the whole cost answer
Versioning keeps every generation of a key, which gives undelete and file history almost free
Blob here, metadata there — the row holds id, owner, key, size and status; the bucket holds the bytes
A CDN goes in front of anything user-facing: object storage is an origin, not a delivery network
Not a database, not a cache — no queries, no transactions, and tens of milliseconds per request

Use cases

Uploads that never touch your servers

The one thing to say about object storage in an interview. The API signs a URL and records a pending row, the client PUTs the bytes directly, and the bucket's own event marks the row ready — so a client that dies mid-upload leaves an orphan for a lifecycle rule rather than a broken record.

DatabaseObject storageQueue / streamFocusClick a node for details

Metadata you can query, bytes you cannot

The split that shows up in every file design. Everything the application filters, sorts or authorises on lives in a row; the bytes live under a key that the row points at.

primary key

Syncing a large file by chunks

Hash fixed-size chunks on the client, ask which ones the server is missing, and upload only those. Deduplication across users, resume after a dropped connection and cheap versioning all fall out of the same content-addressed store.

DatabaseObject storageFocusClick a node for details

Serving the bytes, and ageing them out

Object storage is an origin. A CDN serves users, signed URLs keep private media private, and lifecycle rules move cold objects down the tiers — which is the one-line answer to what a petabyte of logs costs.

Object storageFocusClick a node for details