You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "flysen (via GitHub)" <gi...@apache.org> on 2026/09/09 15:51:59 UTC
[PR] Kafka Connect: Use manual partition assignment for Worker's control-topic consumer [iceberg]
flysen opened a new pull request, #18030:
URL: https://github.com/apache/iceberg/pull/18030
The Worker's control-topic reader used a single-member, never-reused, randomly-named consumer group via `consumer.subscribe()`. That group gets no benefit from consumer-group management (it never has more than one member) but is still exposed to the full JoinGroup/SyncGroup protocol, including broker-side member-fencing failures that can permanently starve a task's commits with no error or warning logged anywhere.
Switch to `consumer.assign()` for this reader, which never registers a consumer group with the broker at all, making this failure mode structurally impossible rather than just less likely. `Coordinator` is unaffected: its control-topic consumer uses a stable, reused group id and relies on real group membership for rebalance-based leader detection, so it keeps using `subscribe()`.
Also hardens the new `assign()`-based path against the control topic not existing yet at Worker startup (`subscribe()` tolerates this; a one-shot `assign()` does not), using `Tasks` for the bounded retry to match this package's existing retry idiom (see `Coordinator.java`).
Related: #15293, #16016 (open issues describing related control-topic/coordinator instability under rebalance; this fixes a distinct root cause found while investigating the same symptom class).
Adds a Testcontainers-based regression test asserting via `AdminClient.listConsumerGroups()` that the Worker never registers an ephemeral control-topic consumer group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: Use manual partition assignment for Worker's control-topic consumer [iceberg]
Posted by "ujasur (via GitHub)" <gi...@apache.org>.
ujasur commented on PR #18030:
URL: https://github.com/apache/iceberg/pull/18030#issuecomment-6477823526
Hey folks, is there ETA when this fix will land, we have been experiencing the bugs which this address where we are ending up with 2 coordinators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org