Why ematix-flow

Eight lines on the back of the box — why ematix-flow exists.

1. Fast — on one node and on a mesh.

On AWS (TPC-H, c7i.4xlarge boxes, July 2026, shipped pip defaults, sum of 22 query medians):

  • Single node: fastest engine at every scale we measured — SF=1 0.68 s vs DuckDB 1.45 s, SF=10 4.8 s vs 5.8 s, SF=100 51.8 s vs 59.0 s.
  • 4-node mesh: 5.6–8.4× faster than Trino and 6–19× faster than PySpark on the identical cluster.
  • One node vs their cluster: a single ematix-flow node outruns the 4-node Trino cluster ~12× at SF=10 and 9.6× at SF=100 — on a quarter of the hardware, that’s ~38× the price-performance.

And in the tightest-controlled setting — single Apple M4 Max, strict protocol, 2σ verdicts (v0.12.0) — at or ahead of every peer on all 22 queries at every scale, with SF=1 geomeans of 2.51× vs DuckDB, 4.27× vs Polars, 18.5× vs local-mode PySpark, and the lead holds under concurrent load. Full tables + reproducers in Benchmarks.

2. Auto-tunes per query — no knobs to set.

With Spark you tune shuffle.partitions, autoBroadcastJoinThreshold, adaptive.enabled, executor memory, and add /*+ BROADCAST(...) */ hints per query to land on a good plan. With ematix-flow you just write the SQL — the catalog is on by default, and the TPC-H table on Benchmarks doesn’t have a per-query tuning column because there’s nothing to tune.

How the auto-tuning works

A physical-optimizer shape catalog pattern-matches the plan and substitutes purpose-built fused operators automatically — filter + aggregate, filter + group-by, dictionary-keyed group-counts, and the other common analytical shapes hit a fused executor instead of the generic Arrow stream. Each catalog entry is a small declarative shape with a rewrite, so adding a new optimization is one entry — no plumbing in the SQL frontend or session API.

For the scan-side adaptive decisions — per-chunk bitmap-vs-values selectivity dispatch, page-streaming vs. eager decode, per-bit-width SIMD kernels — see §6.

3. Scheduling + DAG, no service to operate.

Pipelines carry their own cron schedule and depends_on= edges (with cycle detection and exponential-backoff retries). Run flow run-due from cron, systemd, a Kubernetes CronJob, GitHub Actions, or the bundled long-running scheduler — same code, same topological order, same retry semantics.

Already on Airflow / Dagster / Prefect? Call .sync() directly.

Want a visual operator view? flow web ships a local SPA with the live task DAG, run history, and one-click Restart from failed step / Resume from watermark / Pause / Resume on in-flight runs — see Web UI.

4. Batteries included.

Out-of-the-box backends:

  • Databases. Postgres, MySQL, SQLite, DuckDB.
  • Cloud warehouses. Snowflake, BigQuery, Redshift.
  • Lakehouse. Delta Lake (local + S3).
  • Object stores. Parquet, CSV, ORC, JSONL — local and S3.
  • Streaming. Kafka, RabbitMQ, GCP Pub/Sub, AWS Kinesis.
  • Schema Registry. Confluent SR and Apicurio for Avro / Protobuf.
  • CDC. Source mode dispatches per-op transactionally to your existing target backend.

5. Scales out — auto-detected distributed mode, no cluster service.

Most fast single-node engines (DuckDB, Polars) stop at one machine. ematix-flow doesn’t.

engine = "auto" is the default. Drop in a peers = [...] list and the same SQL fans out across a peer-to-peer mesh of flow-worker processes via Apache Arrow Flight; with no peers it stays in-process. mTLS for the mesh, cross-pod lookup broadcast for small dimension tables, no separate cluster service to run. Going single-node → distributed is a configuration change, not a code change — same @ematix.pipeline decorator, same pipelines, modes, watermarks, and run history.

How the mesh works
  • Library, not a service. ematix-flow-distributed is a Rust crate any ematix-flow process can link. Any process that links it can play coordinator or worker — symmetric mesh, no master node, no JVM, no driver/executor split.
  • Peer auto-detection. peers = [...] accepts three schemes: http://host:port (static), dns://host:port (resolves the A-record at startup and expands to every IP behind the name — good for headless services), and k8s://service.namespace:port (sugar for *.svc.cluster.local). Default engine = "auto" picks distributed when peers expand to ≥1 URL, otherwise falls back to in-process. No restart for new pods — just re-resolve.
  • Spark / DuckDB SQL portable in place. A dialect translator targets DataFusion’s parser under the hood, so existing Spark-flavored or DuckDB-flavored SQL ports across without a rewrite.

The mesh is measured, not promised: on 4 × c7i.4xlarge (July 2026), TPC-H SF=100 runs in 61.3 s vs Trino’s 497 s and PySpark’s 374.5 s on the identical cluster — and the per-query AUTO gate keeps small queries in-process so SF=1 stays at 1.47 s. Full tables in Benchmarks.

6. Hand-tuned Parquet scan path.

Most analytical engines lean on parquet-rs. ematix-flow ships with ematix-parquet — a hand-rolled Rust Parquet codec built for analytical workloads. That’s where most of the TPC-H wins come from: SIMD bit-unpacking that hits 76–96 GB/s, and predicate-fused decode that’s 3.7–6.3× faster than the usual materialize-then-filter path. The codec also ships standalone on crates.io as ematix-parquet-codec / ematix-parquet-io if you want it without the whole pipeline framework.

What’s inside the codec
  • SIMD bit-unpackers on NEON + AVX2. Every bit width 1–32 has a hand-tuned raw-indices kernel; bw=1–21 (the practical range) also has fused unpack + dict-gather kernels.
  • Predicate-fused decode. unpack + filter + bitmap-pack in one SIMD pass. Rows that fail the predicate never materialise.
  • Adaptive dispatch. Per-chunk selectivity probe decides whether to emit a bitmap (wins at low selectivity) or a values vector (wins at high).
  • Decode-into-caller-buffer. Late-materialization (*_masked_into) and Arrow-style (bytes, offsets) shape skip the alloc + zero-fill that dominates scan profiles.
  • Full spec coverage. Every physical type, every encoding (PLAIN, dict, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT), every mainstream codec (Snappy, Zstd, Gzip, Brotli, LZ4_RAW), V1 + V2 pages, page indexes, bloom filters, Parquet Modular Encryption.
  • Light footprint. Sync read/write stack has zero third-party deps beyond the chosen compression codec. Async, encryption, and parallel decode are opt-in features.

7. Data quality is part of the pipeline, not a bolt-on.

Declare expectations= and a freshness_sla= right on the @ematix.pipeline decorator (pip install "ematix-flow[quality]"): expectations run as a post-write stage on every run — warn or fail, your call — and freshness SLOs alert on the healthy→breached edge. The ManagedTable you already declared is the contract: primary keys become unique checks, NOT NULL columns become not_null checks, automatically.

Under the hood it’s ematix-probe, a sibling framework for declarative data-quality assertions and load testing that also stands alone — declare the schema once, get DDL and data-quality checks, with zero hard dependency between the two packages. Ships on PyPI as ematix-probe, Rust + tokio core.

What ematix-probe checks, and how
  • Data probes. @probe.data declares a target (Postgres, DuckDB, Parquet — local or S3) plus assertions: not_null, unique, between, regex, is_in, row_count, freshness, percentile_between, cardinality_between, schema_match. The adapter chooses pushdown SQL vs. Arrow scan internally.
  • Auto-derived from ematix-flow tables. probe_from_table(CustomerDim, source=...) reads your ManagedTable and derives not_null on every non-nullable column + unique on every primary key. extend= lets you layer extras (regex, ranges, freshness windows) via the same fluent API.
  • Loosely coupled. ematix-probe has zero hard dependency on ematix-flow — duck-typed protocol means any class with __tablename__ and an iterable columns attribute participates.
  • pytest plugin. Auto-loaded via pytest11 entry point — no pytest_plugins wiring required. Each assertion becomes one test node, so failures pinpoint the exact rule that fired.
  • Load probes. HTTP and Postgres SQL targets under constant-rate (open-model) or virtual-user (closed-model) schedulers. Shared verdict
    • run-history surface with data probes. (Python decorators land in v0.2; today the load engine is driven from the Rust API or pseudocode- ish Python.)
  • Run history. Opt-in SQLite log keeps verdict trends queryable across runs (ematix-probe runs ...). Designed as the substrate for v0.2 drift detection.

8. Operationally honest.

  • Restart-safe. Watermarks, run-history store, and offset commit ordering mean restarts don’t lose or duplicate data.
  • At-least-once. End-to-end, by default. Exactly-once available for Kafka → Kafka.
  • Credentials redacted. Typed connections strip secrets in repr().
  • Observable. Structured run history, Prometheus metrics + OpenTelemetry trace spans, alerters for Slack, generic webhook, email, and PagerDuty.
  • DLQ. App-level routing + broker-level (RabbitMQ DLX, Pub/Sub dead- letter policy).

Status

ematix-flow is STABLE — v1.0.0. The public API is under semantic versioning: no breaking changes without a major bump. Under the strict, provenance-stamped protocol (solo-engine passes, per-query process isolation, 2σ verdict bars), ematix-flow is fastest — or statistically tied for fastest — on every measured TPC-H configuration, at every scale from in-cache SF=1 through out-of-core SF=100, and holds that under concurrent load across all five stream-count configs. Peer-geomean speedups land at 2.51× / 1.47× / 1.37× vs DuckDB as the columnar reference. On AWS at SF=1000 (1 TB) one r7i.8xlarge runs all 22 in 384.2 s on the mesh AUTO path. See /reference/whats-shipped for the full surface.

Today on PyPI as ematix-flow. All four surfaces — declarative pipelines, multi-backend, streaming, stream processing — are functional end-to-end and benchmark-validated:

pip install ematix-flow

Bug reports, feedback, and design pushback are exactly what we want — open issues on GitHub.