A Python framework for data pipelines — batch or streaming,
scheduled or event-driven — with a fast query engine and an
operator Web UI built in. One pip install, no
cluster to run.
No tuning required. The fastest engine we've measured on single or multi node.
Sum of 22 query medians · single nodes are 1× c7i.4xlarge, clusters are 4× c7i.4xlarge
*completed 16/22 queries — the total covers only those 16 (the missing queries would only add time). Five exceed the 32 GB box with Polars's in-memory engine; one lacks a Polars SQL variant. PySpark completes all 22 after we gave it EBS shuffle scratch. The mesh total is the June build; a refresh with the July planner fix is in progress.
8× faster than Trino, 6× faster than PySpark — same 4-node cluster.
4× c7i.4xlarge · SF=10: 5.6× vs Trino, 6.5× vs PySpark · SF=1: 7.2× / 19×
One ematix node outruns a 4-node Trino cluster 12×.
Same queries, same S3 parquet — a quarter of the hardware, ~12–14× the speed
Fastest single-node engine at every scale we measured.
SF=100 totals: ematix 51.8 s, DuckDB 59.0 s · SF=1: 0.68 s vs 1.45 s
Measured on AWS, not a laptop — TPC-H on c7i.4xlarge (16 vCPU /
32 GB), July 2026, shipped pip install ematix-flow
defaults, no flags, results value-validated.
Full benchmarks →
Install ematix-flow, declare a connection, and ship your first job + workflow. About 10 minutes end-to-end.
How ematix-flow is put together. Why it exists, what's inside, and the mental model behind jobs, workflows, modes, and streaming.
Recipes for common tasks — schedule with composite triggers, run streaming pipelines, drive the Web UI from the operator's seat.
Authoritative tables — what's shipped this release, and the TPC-H numbers vs DuckDB / Polars / PySpark.
A workflow names a DAG of jobs and declares when it fires. Member jobs declare where they sit inside the DAG.
Cron, event, message, and boolean combinations — AND, OR, nested — evaluated against last successful run.
Long-running Kafka / RabbitMQ / Pub/Sub / Kinesis consumers with at-least-once delivery and DLQ support.
Workflows / Jobs / Runs / DAG tabs, restart-from-step, live throughput on streaming pipelines.
TPC-H at SF=1/10/100 on AWS — single node vs DuckDB / Polars, 4-node mesh vs Trino / PySpark — plus the strict M4 Max protocol and concurrent-stream throughput. Reproducer commands included.
Backend matrix and v0.11.0 surface. What's stable, what's still in motion.
A workflow with a composite trigger (event + cron) plus within-DAG ordering.
from ematix_flow import ematix, ManagedTable, Annotated, BigInt, Text, pk
@ematix.connection
class warehouse:
kind = "postgres"
url = "${WAREHOUSE_URL}"
class OrdersExtracted(ManagedTable):
__schema__ = "analytics"; __tablename__ = "orders_extracted"
order_id: Annotated[BigInt, pk()]
customer_id: BigInt
amount_cents: BigInt
class OrdersEnriched(ManagedTable):
__schema__ = "analytics"; __tablename__ = "orders_enriched"
order_id: Annotated[BigInt, pk()]
amount_bucket: Text
@ematix.job(name="extract_orders",
target=OrdersExtracted, target_connection="warehouse",
mode="merge", keys=("order_id",))
def extract_orders(conn):
return "SELECT order_id, customer_id, amount_cents FROM raw.orders"
@ematix.job(name="enrich_orders",
target=OrdersEnriched, target_connection="warehouse",
mode="merge", keys=("order_id",),
depends_on=["extract_orders"])
def enrich_orders(conn):
return "SELECT order_id, CASE WHEN amount_cents < 10000 THEN 'small' ELSE 'large' END AS amount_bucket FROM analytics.orders_extracted"
# Workflow declares the trigger; member jobs declare their DAG position.
ematix.workflow(
name="orders_etl",
triggered_by=["upstream_workflow"],
schedule="0 21 * * *",
timezone="America/New_York",
jobs=["extract_orders", "enrich_orders"],
)
On PyPI as ematix-flow. The core API is
settling toward 1.0; pinning a minor version is still
recommended. Bug reports and design pushback during the
beta window are exactly what we want — file issues on
GitHub.