Advantages

1. Fast.

TPC-H SF=1, 22 queries, single Apple M3 Pro:

1.69× faster than DuckDB
2.71× faster than Polars
12.9× faster than single-node PySpark

(All geomeans. 18 / 22 wins outright.) Full table + reproducer in Benchmarks.

2. Scheduling + DAG, no service to operate.

Pipelines carry their own cron schedule and depends_on= edges (with cycle detection and exponential-backoff retries). Run flow run-due from cron, systemd, a Kubernetes CronJob, GitHub Actions, or the bundled long-running scheduler — same code, same topological order, same retry semantics.

Already on Airflow / Dagster / Prefect? Call .sync() directly.

3. Batteries included.

Out-of-the-box backends:

Databases. Postgres, MySQL, SQLite, DuckDB.
Lakehouse. Delta Lake (local + S3).
Object stores. Parquet, CSV, ORC, JSONL — local and S3.
Streaming. Kafka, RabbitMQ, GCP Pub/Sub, AWS Kinesis.
Schema Registry. Confluent SR and Apicurio for Avro / Protobuf.
CDC. Source mode dispatches per-op transactionally to your existing target backend.

4. Hand-tuned Parquet scan path.

Most analytical engines lean on parquet-rs. ematix-flow ships with ematix-parquet — a hand-rolled Rust Parquet codec built for analytical workloads:

SIMD bit-unpackers on NEON + AVX2. Every bit width 1–32 has a hand-tuned raw-indices kernel; bw=1–21 (the practical range) also has fused unpack + dict-gather kernels. Output ceiling hits 76–96 GB/s on every specialised width.
Predicate-fused decode. unpack + filter + bitmap-pack in one SIMD pass — 3.7–6.3× faster than materialize-then-filter at low selectivity. Rows that fail the predicate never materialise.
Adaptive dispatch. Per-chunk selectivity probe decides whether to emit a bitmap (wins at low selectivity) or a values vector (wins at high).
Decode-into-caller-buffer. Late-materialization (*_masked_into) and Arrow-style (bytes, offsets) shape skip the alloc + zero-fill that dominates scan profiles.
Full spec coverage. Every physical type, every encoding (PLAIN, dict, DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT), every mainstream codec (Snappy, Zstd, Gzip, Brotli, LZ4_RAW), V1 + V2 pages, page indexes, bloom filters, Parquet Modular Encryption.
Light footprint. Sync read/write stack has zero third-party deps beyond the chosen compression codec. Async, encryption, and parallel decode are opt-in features.

That’s where most of the TPC-H wins come from. The codec also ships standalone on crates.io as ematix-parquet-codec / ematix-parquet-io if you want it without the whole pipeline framework.

ematix-probe is a sibling framework for declarative data-quality assertions and load testing. The ManagedTable you declared for the pipeline becomes a probe contract — declare the schema once, get DDL and data-quality checks.

Data probes. @probe.data declares a target (Postgres, DuckDB, Parquet — local or S3) plus assertions: not_null, unique, between, regex, is_in, row_count, freshness, percentile_between, cardinality_between, schema_match. The adapter chooses pushdown SQL vs. Arrow scan internally.
Auto-derived from ematix-flow tables. probe_from_table(CustomerDim, source=...) reads your ManagedTable and derives not_null on every non-nullable column + unique on every primary key. extend= lets you layer extras (regex, ranges, freshness windows) via the same fluent API.
Loosely coupled. ematix-probe has zero hard dependency on ematix-flow — duck-typed protocol means any class with __tablename__ and an iterable columns attribute participates.
pytest plugin. Auto-loaded via pytest11 entry point — no pytest_plugins wiring required. Each assertion becomes one test node, so failures pinpoint the exact rule that fired.
Load probes. HTTP and Postgres SQL targets under constant-rate (open-model) or virtual-user (closed-model) schedulers. Shared verdict
- run-history surface with data probes. (Python decorators land in v0.2; today the load engine is driven from the Rust API or pseudocode- ish Python.)
Run history. Opt-in SQLite log keeps verdict trends queryable across runs (ematix-probe runs ...). Designed as the substrate for v0.2 drift detection.

Ships on PyPI as ematix-probe. Rust + tokio core.

6. Operationally honest.

Restart-safe. Watermarks, run-history store, and offset commit ordering mean restarts don’t lose or duplicate data.
At-least-once. End-to-end, by default. Exactly-once available for Kafka → Kafka.
Credentials redacted. Typed connections strip secrets in repr().
Observable. Structured run history, Prometheus + OpenTelemetry metrics, Slack alerts.
DLQ. App-level routing + broker-level (RabbitMQ DLX, Pub/Sub dead- letter policy).

Status

ematix-flow is currently PRE-ALPHA. Beta release coming soon.

Today on PyPI as ematix-flow. All four surfaces — declarative pipelines, multi-backend, streaming, stream processing — are functional end-to-end and benchmark-validated, but the public surface (decorator names, config keys, CLI flags) may shift between now and the beta tag. If you’re trying it out, pin the exact version in your requirements:

pip install "ematix-flow==0.3.0"