Indexing Parquet with sidecars

Add indexes to Parquet files you already have — no rewrite, no table format required — and when that actually pays off.

A sidecar index is a separate .parquet.idx file written next to a Parquet data file. The data file is never touched — you can index Parquet you already have, serve it to engines that know nothing about the index, and delete the sidecar at any time without losing data. Sidecars come from ematix-parquet, the Parquet codec inside ematix-flow.

Status: the ematix-parquet index builder + readers are shipped. The ematix-flow integration described below (automatic read-side pushdown, flow index build, Iceberg manifest stamping) is on the feat/sidecar-indexes branch — preview, next release. Numbers and behavior on this page are measured from that branch.

Creating indexes

Two ways, same artifact:

  • Backfill onto existing files (the headline). The flow index build CLI walks a directory of Parquet and emits a sidecar per file for the columns you name — index without rewriting a byte of data.
  • At write time. Flow-owned ingest can emit the sidecar with each Parquet write (opt-in config), atomically renamed so a reader never sees a half-written index.

Each sidecar carries a fingerprint of its data file. If the data file is rewritten, the index is detected as stale and ignored — a stale sidecar can slow nothing down and can never return wrong rows.

Read-side behavior (preview)

When a scan carries a supported predicate (col = literal today, ranges next) on an indexed column, the reader consults the sidecar instead of scanning — gated by selectivity. From the column’s min/max the reader estimates the match fraction; above EMAT_SIDECAR_MAX_SELECTIVITY (default 0.05) it skips the index and scans, because a low-selectivity “index lookup” is just a slower scan. Hits, misses, staleness fallbacks, and selectivity skips are all counted and observable.

When sidecars pay — honest guidance

We benchmarked point lookups on a 10M-row file on local NVMe, with and without a sidecar, across match fractions from 0.01% to 10%. On fast local disk the index does not win today: the full scan finishes in single-digit milliseconds while an index lookup pays an index-open + decode cost in the hundreds of milliseconds, at every match fraction. That’s why the selectivity gate defaults conservative.

Where sidecars are designed to pay:

  • Object storage (S3-class latency), where a full scan means re-downloading gigabytes but an index lookup touches a few ranges.
  • Very large tables (SF1000-class), especially combined with Iceberg manifest pruning — prune to the few files that can match, then let each file’s sidecar cut the rows.
  • Wide tables where the scan itself is expensive even locally.

We publish the crossover as we measure it rather than assuming it — the same discipline as the benchmarks.

Iceberg integration (preview)

With the off-by-default iceberg feature, ematix-flow combines sidecars with Apache Iceberg metadata:

  • Manifest pruning: a predicate prunes the Iceberg manifest first — files whose stamped min/max can’t match never produce work at all.
  • Write-side stamping: when flow writes an Iceberg table it stamps each data file’s manifest entry with the summary + sidecar path (derived from exact Parquet footer statistics only, so pruning can never produce a false negative).
  • Distributed: on the peer mesh, the coordinator prunes once and fans out only the surviving files — the mesh starts with a strictly smaller task set.

Lifecycle notes

  • Drop the data file → drop its sidecar (nothing enforces this yet; the stale-fingerprint check makes a forgotten sidecar harmless).
  • Sidecars are per-file, so partitioned layouts index part by part — a rewrite of one part invalidates one sidecar, not the table.