You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "mlevkov (via GitHub)" <gi...@apache.org> on 2026/03/12 20:54:07 UTC
[I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
mlevkov opened a new issue, #2928:
URL: https://github.com/apache/iggy/issues/2928
## Description
The `AutoCommitWhen::PollingMessages` strategy commits consumer group offsets **before** `consume()` is called on the sink plugin. This means if `consume()` fails (and the failure is already silently discarded per #2927), the messages are permanently lost — the consumer group has already advanced past them.
## Sequence
```
1. consumer.next() → messages received
2. Offsets committed → consumer group advances ← PROBLEM
3. process_messages() → calls sink consume() via FFI
4. consume() fails → messages already committed, lost forever
```
## Location
`core/connectors/runtime/src/sink.rs`:
- **Line 421**: Consumer configured with `AutoCommitWhen::PollingMessages`
```rust
.auto_commit(AutoCommit::When(AutoCommitWhen::PollingMessages))
```
- **Lines 266-344** (`consume_messages` loop): `consumer.next()` at line 272 triggers the auto-commit, but `process_messages()` doesn't execute until line 311.
## Impact
Combined with #2927 (consume return value discarded), **at-least-once delivery is not achievable** with the current runtime for any sink connector. If `consume()` fails and the process restarts, those messages will never be retried because the offsets have already been committed.
## Suggested Fix
Change auto-commit strategy to `AutoCommitWhen::ConsumerStopped` or `AutoCommit::Disabled`, and commit offsets **after** successful `consume()`:
```rust
// Option A: Commit after processing
.auto_commit(AutoCommit::Disabled)
// ... in the consume loop, after successful consume():
consumer.store_offset(offset).await?;
// Option B: Use AfterPollingMessages if available
.auto_commit(AutoCommit::When(AutoCommitWhen::AfterProcessing))
```
The exact API depends on the Iggy SDK's consumer group offset management capabilities.
## Related
- #2927 (consume return value discarded)
- Discussed in #2901 (HTTP sink connector proposal)
- HTTP sink PR: #2925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio closed issue #2928: bug(connectors): PollingMessages auto-commit commits offsets before sink processing
URL: https://github.com/apache/iggy/issues/2928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on issue #2928:
URL: https://github.com/apache/iggy/issues/2928#issuecomment-4698037541
Could I please be assigned this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "tmchow (via GitHub)" <gi...@apache.org>.
tmchow commented on issue #2928:
URL: https://github.com/apache/iggy/issues/2928#issuecomment-4152986200
Related to #2927. The auto-commit happens on the next `poll()` before `consume()` runs (`sink.rs:421` uses `AutoCommitWhen::PollingMessages`). I'd switch to `AutoCommit::Disabled` and commit manually after successful processing. Want to confirm the approach before coding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on issue #2928:
URL: https://github.com/apache/iggy/issues/2928#issuecomment-4702473739
Related discussion: https://github.com/apache/iggy/discussions/3203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on issue #2928:
URL: https://github.com/apache/iggy/issues/2928#issuecomment-6655722240
linking this to #4238. #4734 fixed it as an opt-in: with `[delivery] guarantee = "at_least_once"` the offset is stored only after the plugin's `flush` proves delivery (only the HTTP sink supports it so far). the default `at_most_once` still stores the offset on read, so this stays open until #4238 makes at-least-once the default for new configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(connectors): PollingMessages auto-commit commits offsets before sink processing [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on issue #2928:
URL: https://github.com/apache/iggy/issues/2928#issuecomment-6773721572
the connectors moved to [apache/iggy-connectors](https://github.com/apache/iggy-connectors), and this issue moved with them: apache/iggy-connectors#9. please continue there.
@mlevkov: github can't carry your assignment over, because it only lets us assign people who commented on an issue. please leave a short comment on the new issue, so a maintainer can assign you there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org