Delivery semantics & recovery

What ematix-flow guarantees under failure — crashes, restarts, rebalances, and slow or unavailable targets — and where the operator carries responsibility.

This is the reference for what happens under failure: process crashes, restarts, partition rebalances, poison messages, and slow or unavailable targets. If you’re evaluating ematix-flow for a production stream, read this before you design around it — and validate the cases that matter to you against your own broker and target.

The semantics below reflect the v1.0.0 Kafka backend and streaming runtime. They are stable and intended, and the public API around them is under semantic versioning — no breaking changes without a major bump.

At a glance

At-least-once is the default

The streaming loop reads a batch, transforms it, writes to the target, and only then commits the source offset. The consequence is the standard at-least-once contract: if the process dies after the write but before the commit, those records are read again on restart. So duplicates are possible, and the operator makes a re-write harmless.

Make writes idempotent

Pair at-least-once delivery with an idempotent load mode so a re-delivery is a no-op:

• merge (SCD1) — upsert on the merge key; a re-delivered row overwrites itself. Net effect: exactly-once state.
• scd2 — versioned upsert with merge-key resolution and watermarks.
• Derive the merge key deterministically from the event, not from an auto-increment or wall clock, so the same event always resolves to the same row.

append mode does not de-duplicate — use it only when duplicates are acceptable or removed downstream.

Exactly-once

Two distinct paths exist; know which one you’re getting.

1. Kafka producer sink (write-side EOS). Each write batch to a Kafka topic can be wrapped in a Kafka transaction (begin → produce → commit, or abort on failure). Requires a unique transactional_id per process; the transactional producer is reused across batches, and two producers sharing an id are fenced by the broker. Cost: one extra broker round-trip per batch.
2. Kafka → Kafka read-process-write. The full flow — where the consumer offset commit is itself part of the producer transaction — is handled by the dedicated EOS pipeline. The offset advance and the output produce commit atomically, so a crash can’t leave the output written but the input un-consumed.

Exactly-once into a non-Kafka target (Postgres, Delta, S3) isn’t a Kafka-transaction property — for those, use at-least-once + an idempotent merge/scd2 write, which gives exactly-once state.

Failure & recovery

Crash between target write and offset commit — records re-delivered on restart (at-least-once). Idempotent merge/scd2 absorbs the duplicate.

Rebalance while a batch is in flight — handled through the negotiated protocol (cooperative incremental_assign / eager). Recovered offsets are injected into the assignment before the assign; if recovery can’t set one, the partition falls back to the broker-stored committed offset. Since commits only follow a successful write, a reassigned partition resumes from the last fully-written position — re-processing the in-flight batch rather than skipping it.

Poison message / partial batch — transform_on_error selects fail (stop / retry from last commit), drop (skip the record), or dlq (quarantine). The DLQ is a Kafka topic (dead_letter_topic) or a table (dlq_store); a record is quarantined after dlq_max_attempts (its original emission is attempt 1), so you get bounded retries, not an infinite poison loop.

Multi-target fan-out — every target is written before the source offset advances. A failure on any target leaves the offset un-committed and the whole batch retries, so the already-written targets see a re-write too — they must be idempotent as well.

Target slow or unavailable (backpressure) — reads are gated by the batch size/time limits, so a slow target throttles consumption instead of building an unbounded backlog. If the target is down, the write fails, the offset isn’t committed, and the batch retries from the last commit.

Pipeline stops running entirely — no batch means no error to alert on. That’s what freshness SLOs are for: freshness_sla="6h" is evaluated at run-end and on a schedule (flow freshness-check), so a stalled pipeline breaches its SLO and fires sla_breached. Freshness is computed from run history and is target-agnostic — it works for any connection kind.

Running multiple replicas

Replicas in the same consumer group distribute partitions via the rebalance protocol above — standard Kafka scaling. For the exactly-once producer path, each replica needs its own unique transactional_id; sharing one triggers broker fencing. Validate replica scaling under a forced rebalance before relying on it.

Validate it yourself

Documentation isn’t a substitute for testing against your broker, target, and load. Before trusting a critical stream, exercise: duplicate handling after a hard kill between write and commit; rebalance under load; poison-message routing and dlq_max_attempts; late / out-of-order events for your window type; window/state recovery after restart; backpressure when the target is throttled; and broker outage and reconnect.

The full, code-level reference lives with the engine in docs/DELIVERY_SEMANTICS.md. If behavior differs from what’s documented here on settled, idle hardware, open an issue with a reproduction.