You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iggy.apache.org by "kriti-sc (via GitHub)" <gi...@apache.org> on 2026/04/30 19:22:57 UTC
Manual Offset Commit
GitHub user kriti-sc created a discussion: Manual Offset Commit
The sink runtime currently configures consumers with `AutoCommit::When(AutoCommitWhen::PollingMessages)` ([runtime/src/sink.rs:421](runtime/src/sink.rs#L421)), meaning offsets are committed when messages are polled — before they are processed. A failed batch is silently lost.
Sink must have an option to decide exactly when it is safe to advance the offset, as only the sink knows when a write is truly durable from the target's perspective.
## Proposed Solution
Offset information is already available to the sink. `MessagesMetadata` (passed to `Sink::consume`) contains `partition_id` and `current_offset`. Each `ConsumedMessage` carries its own `offset`. What is missing is a mechanism to actually call commit.
The runtime passes a `CommitCallback` function pointer into the sink at `open` time, analogous to how `LogCallback` is already passed. The sink stores it and calls `commit_callback(partition_id, offset)` when its write to the target is durable. The runtime switches to `AutoCommit::Disabled` and trusts the sink to drive commits.
Note: The above mentioned `CommitCallback` mechanism is a breaking change but a non-breaking alternative is proposed below.
## Implementation
Three pieces of work are required.
### 1. Configurable Commit Mode (`OffsetCommitMode`)
Add a new field to `SinkConfig` ([runtime/src/configs/connectors.rs:106](runtime/src/configs/connectors.rs#L106)):
```rust
#[serde(default)]
pub offset_commit: OffsetCommitMode,
```
With the following variants:
- `Auto` — default; current behaviour, `AutoCommit::When(AutoCommitWhen::PollingMessages)`; existing sinks work unchanged
- `SinkControlled` — `AutoCommit::Disabled`; runtime passes a `CommitCallback` to the sink at `open` time, sink drives all offset commits
`setup_sink_consumers` ([runtime/src/sink.rs:379](runtime/src/sink.rs#L379)) reads this field and sets the consumer's `AutoCommit` accordingly:
```rust
let auto_commit = match config.offset_commit {
OffsetCommitMode::Auto => AutoCommit::When(AutoCommitWhen::PollingMessages),
OffsetCommitMode::SinkControlled => AutoCommit::Disabled,
};
```
### 2. `CommitCallback` in the ABI
Add to the SDK:
```rust
pub type CommitCallback = extern "C" fn(plugin_id: u32, stream_id: u32, topic_id: u32, partition_id: u32, offset: u64) -> i32;
```
The SDK's `SinkContainer` stores the callback and exposes it to the sink instance via a `CommitHandle` wrapper. Two options for how the callback is delivered:
**Option A: Extend `iggy_sink_open` (breaking)**
`iggy_sink_open` gains a `commit_callback: CommitCallback` parameter. All existing plugins break at the ABI level and must be recompiled. The runtime always passes either the real callback (when `SinkControlled`) or a no-op stub (when `Auto`).
**Option B: Separate `iggy_sink_set_commit_callback` symbol (non-breaking)**
A new optional symbol is added:
```rust
extern "C" fn iggy_sink_set_commit_callback(plugin_id: u32, callback: CommitCallback) -> i32;
```
The runtime calls it after `iggy_sink_open`, only when `SinkControlled`. Existing plugins that do not export this symbol continue to load and run unchanged — the runtime detects the symbol's absence via `dlopen2` and treats it as `Auto` mode. Only plugins that opt into `SinkControlled` in config need to export the new symbol.
Option B is preferred as it preserves backwards compatibility with all existing plugins.
### 3. Global Offset Sender Registry
The `CommitCallback` is `extern "C"` — a bare function pointer that cannot capture state. It needs to route a commit call from the plugin back to the correct `IggyConsumer` instance in the runtime. The solution follows the same pattern as `SOURCE_SENDERS` ([runtime/src/source.rs:52](runtime/src/source.rs#L52)), which solves the identical problem for sources:
```rust
pub static SINK_COMMIT_SENDERS: Lazy<DashMap<(u32, u32, u32, u32), flume::Sender<u64>>> =
Lazy::new(DashMap::new);
```
The registry is keyed by `(plugin_id, stream_id, topic_id, partition_id)`. Each part is necessary: a plugin can have multiple consumers across streams and topics, and partition IDs are only unique within a topic. The sink has all four values at hand — `stream_id` and `topic_id` from `TopicMetadata`, `partition_id` from `MessagesMetadata`, and `plugin_id` baked into the `CommitHandle`.
The runtime, for each consumer created under `SinkControlled`:
1. Creates the consumer
2. Inserts its `store_offset_sender` into `SINK_COMMIT_SENDERS` keyed by `(plugin_id, stream_id, topic_id, partition_id)`
3. Passes the static `commit_callback` into `iggy_sink_open`
**Considerations:**
- **Cleanup** — entries must be removed on sink shutdown or restart, same as `SOURCE_SENDERS::cleanup_sender`; stale entries do not cause correctness issues (the channel will be closed and sends will fail) but should be cleaned up to avoid unbounded growth
- **Channel closed** — if the consumer shuts down before the sink calls commit, the send will fail; the callback returns `-1` so the sink can detect and handle the failure
GitHub link: https://github.com/apache/iggy/discussions/3203
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
GitHub user kriti-sc added a comment to the discussion: Manual Offset Commit
that makes sense.
more importantly, that requires making a decision on the i32 returned at the FFI boundary. currently, the i32 returned by the connector is dropped. if the runtime makes a decision based on the result returned by the connector, it does not make sense for the connector to make those decisions (for ex. offset commit, retry, routing to dlq). If the decision responsibility lies with the connector, the i32 returned to the runtime has to remain only for observability purposes.
More details here: https://github.com/apache/iggy/issues/2927#issuecomment-4699245941
@hubcio @spetz
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17299422
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
GitHub user hubcio added a comment to the discussion: Manual Offset Commit
@spetz @kriti-sc the callback-vs-consumer thing and the i32 thing are the same question, so let me dump both here.
no callback / registry. the `CommitCallback` + `SINK_COMMIT_SENDERS` idea copies `SOURCE_SENDERS`, but sources need that because the plugin pushes data back over the FFI. commit goes the other way. the `IggyConsumer` is already runtime-side, `consume_messages` already holds it (`mut consumer`, sink.rs:305), and `store_offset` is public + async. so spetz is right, after a good batch we just `consumer.store_offset(...).await`. no ABI change, nonew symbol. forget option A/B.
and no `AutoCommitWhen` variant fits either, none of them mean "after the sink wrote ok", the consumer commits on its own poll lifecycle and never sees the FFI result. only `AutoCommit::Disabled` + an explicit commit after the call returns ok gets us at-least-once.
now the i32. as @kriti-sc agreed, retry / DLQ / semantics belong to the sink, it's the only one that knows transient vs permanent and its own idempotency. but it can't be fully cosmetic. spetz's "commit after success" already needs the runtime to read the return to know it was a success. so: i32 decides commit (0 commit, non-zero don't) and nothing else. runtime doesn't retry, classify or route on it. that's not the runtime owning failures, it owns the offset, which it has to. i thinkit makes sense.
one thing toflag before anyone thinks this is trivial: "commit after success" is trickier than it sounds. the consumer keeps an in-memory cursor (`last_consumed_offsets`) that advances on every `next()` and dedups client-side (`allow_replay` is false), so just withholding the commit does NOT replay the failed batch to a running consumer, it only replays after a restart from the stored offset. and we can't blindly `store_offset` after each good batch either, a success straight after a failure would commit past the gap and drop it. so AtLeastOnce really means commit only up to the last contiguous success and hold the rest back, same trick as kafka connect's preCommit. transient blips stay the sink's job via in-call retry (HttpRetryMiddleware already does this, which is exactly @kriti-sc's sink-owns-retry point). on a hard failure we halt on a consecutive-failure circuit breaker instead of stepping over it, and that halt is where a DLQ hangs off later. (also worth saying out loud: at-le
ast-once leans on idempotent sinks, and not all of ours are, ES with auto-id isn't.)
and whatever we do, do not repeat #3061 / #3180, which `return Err` and kill the whole task on one bad batch.
so, no ABI change: one`SinkConfig` field, two modes. name the guarantee not the mechanism imo, `AtMostOnce` (default, today's `When(PollingMessages)`, fine for idempotent sinks) and `AtLeastOnce` (`Disabled` + commit up to the last contiguous success). plus read the i32, gate the commit, error metric, circuit breaker.
DLQ+ a richer return code (transient vs permanent) is its own thing, bigger, no design yet, and it shouldn't hold up at-least-once. when we get to it i'd steal the kafka connect split: framework DLQ for the pipeline drops the runtime already sees (decode/transform/serialize), sink write failures stay the sink's problem. FFI stays put.
i think this covers both #2927 and #2928.
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17310008
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "mlevkov (via GitHub)" <gi...@apache.org>.
GitHub user mlevkov edited a comment on the discussion: Manual Offset Commit
@hubcio @kriti-sc @spetz — +1 to the converged shape in hubcio's last comment; it matches what I'd independently landed digging through the consumer internals for #2927/#2928.
Since the commit story (here) and the DLQ (#3100) are really one design, I've written them up together as a single discussion so the whole picture lives in one place (per @hubcio's earlier "let's open a discussion and dump ideas"): #3518. This thread's piece — at-least-once — is Layer 1 there.
Confirming the Layer-1 shape so we're aligned:
- **Runtime-side commit, no callback.** `consume_messages` already holds `mut consumer` (`sink.rs:305`); `store_offset` is public + async and internally monotonic (`consumer.rs:438`), so a re-commit after restart-replay is a no-op. No `CommitCallback`/`SINK_COMMIT_SENDERS` registry needed.
- **Binary `i32` gates commit, nothing else.** `0` commits, non-zero doesn't; classification/retry/DLQ stay the sink's.
- **`AtLeastOnce` = `Disabled` + commit to last contiguous success.** The de-dup cursor (`messages.retain(|m| m.header.offset > consumed)` when `!allow_replay`, `consumer.rs:712-719`) means withholding a commit doesn't replay to a *running* consumer — so the loop re-runs the **in-hand** batch (not re-poll), and never steps over a gap. Worth a dedicated test.
- **Halt, never `return Err`.** A persistent failure trips a consecutive-failure `sdk::retry::CircuitBreaker` that pauses the partition — not the #3061/#3180 task-kill. That halt is also where the #3100 DLQ later hangs "dead-letter-and-advance."
One precision: with a binary per-batch `i32`, "last contiguous success" is necessarily **batch-granular** (the runtime can't know which records within a batch succeeded). Message-granular partial commit belongs to the DLQ work, not here.
Caveat (agreed): at-least-once leans on idempotent sinks (ES auto-id, Iceberg catalog-commit); `AtMostOnce` stays the default.
I filed #2927/#2928 and have this scoped (`DeliveryGuarantee` config → `AutoCommit`; bind+gate the `i32` at `sink.rs:740`; contiguous `store_offset`; the breaker; an `iggy_connector_*` error metric; a WireMock/Docker IT with a crash-redelivery leg). Happy to take it as **one PR closing both #2927 and #2928** — or split the commit-mode and the `i32`-gate if you'd prefer. OK to proceed?
See the discussion note that I filed in #3518.
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17378269
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "mlevkov (via GitHub)" <gi...@apache.org>.
GitHub user mlevkov added a comment to the discussion: Manual Offset Commit
@hubcio @kriti-sc @spetz — +1 to the converged shape in hubcio's last comment; it matches what I'd independently landed digging through the consumer internals for #2927/#2928.
Since the commit story (here) and the DLQ (#3100) are really one design, I've written them up together as a single discussion so the whole picture lives in one place (per @hubcio's earlier "let's open a discussion and dump ideas"): **<link to the consolidating discussion>**. This thread's piece — at-least-once — is Layer 1 there.
Confirming the Layer-1 shape so we're aligned:
- **Runtime-side commit, no callback.** `consume_messages` already holds `mut consumer` (`sink.rs:305`); `store_offset` is public + async and internally monotonic (`consumer.rs:438`), so a re-commit after restart-replay is a no-op. No `CommitCallback`/`SINK_COMMIT_SENDERS` registry needed.
- **Binary `i32` gates commit, nothing else.** `0` commits, non-zero doesn't; classification/retry/DLQ stay the sink's.
- **`AtLeastOnce` = `Disabled` + commit to last contiguous success.** The de-dup cursor (`messages.retain(|m| m.header.offset > consumed)` when `!allow_replay`, `consumer.rs:712-719`) means withholding a commit doesn't replay to a *running* consumer — so the loop re-runs the **in-hand** batch (not re-poll), and never steps over a gap. Worth a dedicated test.
- **Halt, never `return Err`.** A persistent failure trips a consecutive-failure `sdk::retry::CircuitBreaker` that pauses the partition — not the #3061/#3180 task-kill. That halt is also where the #3100 DLQ later hangs "dead-letter-and-advance."
One precision: with a binary per-batch `i32`, "last contiguous success" is necessarily **batch-granular** (the runtime can't know which records within a batch succeeded). Message-granular partial commit belongs to the DLQ work, not here.
Caveat (agreed): at-least-once leans on idempotent sinks (ES auto-id, Iceberg catalog-commit); `AtMostOnce` stays the default.
I filed #2927/#2928 and have this scoped (`DeliveryGuarantee` config → `AutoCommit`; bind+gate the `i32` at `sink.rs:740`; contiguous `store_offset`; the breaker; an `iggy_connector_*` error metric; a WireMock/Docker IT with a crash-redelivery leg). Happy to take it as **one PR closing both #2927 and #2928** — or split the commit-mode and the `i32`-gate if you'd prefer. OK to proceed?
See the discussion note that I filed in #3518.
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17378269
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: Manual Offset Commit
I like the idea, and for starters, 2 modes could be enough, e.g., `AfterPolling` (current one) and `AfterConsuming` - not sure about these names, but the more explicit these are, I think the better for the users. However, I'm not sure if we need an additional callback mechanism.
Actually, why not just make use of the existing `mut consumer: IggyConsumer` within `consume_messages()` and, depending on the mode, either do nothing or invoke `store_offset()` after `process_messages` has completed successfully?
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-16779424
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
GitHub user kriti-sc added a comment to the discussion: Manual Offset Commit
@hubcio -- agree with both points.
@mlevkov -- Thanks for the details. It aligns with the approach I was working through in the discussions. I am planning to implement this myself.
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17456753
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
GitHub user kriti-sc added a comment to the discussion: Manual Offset Commit
thought about this a bit more:
i am not sure if the offset advancing decision should hinge on i32 ( returned by sink @[sink.rs:740](https://github.com/apache/iggy/blob/7b2db334f244df06805f2c76c0490a5c1a6ec7e5/core/connectors/runtime/src/sink.rs#L740) )
Because a sink returning non-zero on a batch means a non-transient error, and the only option at this point is routing the batch to DLQ. Otherwise, the process will remain stuck on the same offset. So the offset should advance once sink returns regardless of success of the batch.
Re the i32 -- until the DLQ, I am not seeing any utility so [sink.rs:740](https://github.com/apache/iggy/blob/7b2db334f244df06805f2c76c0490a5c1a6ec7e5/core/connectors/runtime/src/sink.rs#L740) can remain as it is.
let me know thoughts @hubcio
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-17532884
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "kriti-sc (via GitHub) via dev" <de...@iggy.apache.org>.
GitHub user kriti-sc closed a discussion: Manual Offset Commit
The sink runtime currently configures consumers with `AutoCommit::When(AutoCommitWhen::PollingMessages)` ([runtime/src/sink.rs:421](runtime/src/sink.rs#L421)), meaning offsets are committed when messages are polled — before they are processed. A failed batch is silently lost.
Sink must have an option to decide exactly when it is safe to advance the offset, as only the sink knows when a write is truly durable from the target's perspective.
## Proposed Solution
Offset information is already available to the sink. `MessagesMetadata` (passed to `Sink::consume`) contains `partition_id` and `current_offset`. Each `ConsumedMessage` carries its own `offset`. What is missing is a mechanism to actually call commit.
The runtime passes a `CommitCallback` function pointer into the sink at `open` time, analogous to how `LogCallback` is already passed. The sink stores it and calls `commit_callback(partition_id, offset)` when its write to the target is durable. The runtime switches to `AutoCommit::Disabled` and trusts the sink to drive commits.
Note: The above mentioned `CommitCallback` mechanism is a breaking change but a non-breaking alternative is proposed below.
## Implementation
Three pieces of work are required.
### 1. Configurable Commit Mode (`OffsetCommitMode`)
Add a new field to `SinkConfig` ([runtime/src/configs/connectors.rs:106](runtime/src/configs/connectors.rs#L106)):
```rust
#[serde(default)]
pub offset_commit: OffsetCommitMode,
```
With the following variants:
- `Auto` — default; current behaviour, `AutoCommit::When(AutoCommitWhen::PollingMessages)`; existing sinks work unchanged
- `SinkControlled` — `AutoCommit::Disabled`; runtime passes a `CommitCallback` to the sink at `open` time, sink drives all offset commits
`setup_sink_consumers` ([runtime/src/sink.rs:379](runtime/src/sink.rs#L379)) reads this field and sets the consumer's `AutoCommit` accordingly:
```rust
let auto_commit = match config.offset_commit {
OffsetCommitMode::Auto => AutoCommit::When(AutoCommitWhen::PollingMessages),
OffsetCommitMode::SinkControlled => AutoCommit::Disabled,
};
```
### 2. `CommitCallback` in the ABI
Add to the SDK:
```rust
pub type CommitCallback = extern "C" fn(plugin_id: u32, stream_id: u32, topic_id: u32, partition_id: u32, offset: u64) -> i32;
```
The SDK's `SinkContainer` stores the callback and exposes it to the sink instance via a `CommitHandle` wrapper. Two options for how the callback is delivered:
**Option A: Extend `iggy_sink_open` (breaking)**
`iggy_sink_open` gains a `commit_callback: CommitCallback` parameter. All existing plugins break at the ABI level and must be recompiled. The runtime always passes either the real callback (when `SinkControlled`) or a no-op stub (when `Auto`).
**Option B: Separate `iggy_sink_set_commit_callback` symbol (non-breaking)**
A new optional symbol is added:
```rust
extern "C" fn iggy_sink_set_commit_callback(plugin_id: u32, callback: CommitCallback) -> i32;
```
The runtime calls it after `iggy_sink_open`, only when `SinkControlled`. Existing plugins that do not export this symbol continue to load and run unchanged — the runtime detects the symbol's absence via `dlopen2` and treats it as `Auto` mode. Only plugins that opt into `SinkControlled` in config need to export the new symbol.
Option B is preferred as it preserves backwards compatibility with all existing plugins.
### 3. Global Offset Sender Registry
The `CommitCallback` is `extern "C"` — a bare function pointer that cannot capture state. It needs to route a commit call from the plugin back to the correct `IggyConsumer` instance in the runtime. The solution follows the same pattern as `SOURCE_SENDERS` ([runtime/src/source.rs:52](runtime/src/source.rs#L52)), which solves the identical problem for sources:
```rust
pub static SINK_COMMIT_SENDERS: Lazy<DashMap<(u32, u32, u32, u32), flume::Sender<u64>>> =
Lazy::new(DashMap::new);
```
The registry is keyed by `(plugin_id, stream_id, topic_id, partition_id)`. Each part is necessary: a plugin can have multiple consumers across streams and topics, and partition IDs are only unique within a topic. The sink has all four values at hand — `stream_id` and `topic_id` from `TopicMetadata`, `partition_id` from `MessagesMetadata`, and `plugin_id` baked into the `CommitHandle`.
The runtime, for each consumer created under `SinkControlled`:
1. Creates the consumer
2. Inserts its `store_offset_sender` into `SINK_COMMIT_SENDERS` keyed by `(plugin_id, stream_id, topic_id, partition_id)`
3. Passes the static `commit_callback` into `iggy_sink_open`
**Considerations:**
- **Cleanup** — entries must be removed on sink shutdown or restart, same as `SOURCE_SENDERS::cleanup_sender`; stale entries do not cause correctness issues (the channel will be closed and sends will fail) but should be cleaned up to avoid unbounded growth
- **Channel closed** — if the consumer shuts down before the sink calls commit, the send will fail; the callback returns `-1` so the sink can detect and handle the failure
GitHub link: https://github.com/apache/iggy/discussions/3203
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Manual Offset Commit
Posted by "hubcio (via GitHub) via dev" <de...@iggy.apache.org>.
GitHub user hubcio added a comment to the discussion: Manual Offset Commit
closing this one: #4734 merged at-least-once delivery for sinks, and apache/iggy-connectors#53 tracks the rest. the connectors live in [apache/iggy-connectors](https://github.com/apache/iggy-connectors) now, so follow-ups go to the discussions there.
GitHub link: https://github.com/apache/iggy/discussions/3203#discussioncomment-19524552
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org