You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "adikou (via GitHub)" <gi...@apache.org> on 2026/05/08 14:13:37 UTC
[PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
adikou opened a new pull request, #22242:
URL: https://github.com/apache/kafka/pull/22242
Introduce RebalanceConsumer, a restricted view of the Consumer passed to
rebalance callbacks that exposes only operations safe during a rebalance
(seek, commit, position, pause/resume, etc.) and is invalidated after the callback returns.
Add two-argument variants to each ConsumerRebalanceListener callback that receive
a RebalanceConsumer, with defaults delegating to the existing one-argument methods
for backward compatibility.
Add setConsumerRebalanceListener() to Consumer as the new way to register a listener,
decoupling listener registration from subscription. Internally, ConsumerAwareRebalanceListener
wraps the user listener and manages DelegatingRebalanceConsumer view lifecycle,
with the Consumer reference provided at registration time.
Deprecate the Optional<ConsumerRebalanceListener> subscribe() variants on SubscriptionState in favor of listenerless overloads plus setConsumerRebalanceListener(listener, consumer).
The subscribe(topics, listener) overloads on Consumer are not yet annotated @deprecated to avoid
-Werror across the repo to keep the diff minimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4408925121
Thanks for the PR @adikou! I will also take a look next week, but wonder if it would be helpful to split this PR into smaller one to make progress better?
E.g, we could try smaller PRs for the new components/func introduced with the KIP at the lower level (without wiring them into the consumers): RebalanceConsumer/ConsumerRebalanceListener/RebalanceDelegate, their tests. Then maybe worth splitting the wiring into the Consumers (Classic and Async). Also can consider handling other components separately (e.g, MockConsumer). Up to you that have the code in hand, just suggesting for consideration in case it helps. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4547015608
@adikou I would close this one now it's been superseded. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4409242099
Nice! In this case you can keep the same jira if you prefer I would say, and just prefix the PRs with "KAFKA-20385 [1/N]: ...". I don't expect we need too many, but it will help review/delivery. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "adikou (via GitHub)" <gi...@apache.org>.
adikou commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4435502731
Hi @lianetm @AndrewJSchofield I've opened #22270 , #22271, and #22272 by splitting up this PR.
Please take a look when you can. #22271 and #22272 both point to trunk but are both dependent on #22270 so their compilation will break. But I wanted to leave the diffs around for review first. When I merge the changes to trunk I'll rebase and push new commits upstream.
If you'd like, I can close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #22242:
URL: https://github.com/apache/kafka/pull/22242#discussion_r3209934883
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCallbackTest.java:
##########
@@ -218,7 +219,8 @@ private void testOnPartitionsAssignedCalledWithExpectedPartitions(
private void subscribeAndExpectOnPartitionsAssigned(Consumer<byte[], byte[]> consumer, List<String> topics, Collection<TopicPartition> expectedPartitionsInCallback) throws InterruptedException {
var partitionsAssigned = new AtomicBoolean(false);
AtomicReference<Collection<TopicPartition>> partitionsFromCallback = new AtomicReference<>();
- consumer.subscribe(topics, new ConsumerRebalanceListener() {
+ consumer.subscribe(topics);
+ consumer.setConsumerRebalanceListener(new ConsumerRebalanceListener() {
Review Comment:
I'd set the rebalance listener before subscribing. Otherwise, you have a tiny race condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "adikou (via GitHub)" <gi...@apache.org>.
adikou closed pull request #22242: KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener
URL: https://github.com/apache/kafka/pull/22242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "adikou (via GitHub)" <gi...@apache.org>.
adikou commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4409089552
> Thanks for the PR @adikou! I will also take a look next week, but wonder if it would be helpful to split this PR into smaller ones to make progress better?
>
> E.g, we could try smaller PRs for the new components/func introduced with the KIP at the lower level (without wiring them into the consumers): RebalanceConsumer/ConsumerRebalanceListener/RebalanceDelegate, their tests. Then maybe worth splitting the wiring into the Consumers (Classic and Async). Also can consider handling other components separately (e.g, MockConsumer). Up to you that have the code in hand, just suggesting for consideration in case it helps. Thanks!
Thanks Lianet. I was leaning towards smaller PRs and generally prefer it that way. I wasn't sure was the preferred method for the community: can I open each PR against the same JIRA? Do you prefer squash and merge as opposed to smaller thread of PRs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "adikou (via GitHub)" <gi...@apache.org>.
adikou commented on PR #22242:
URL: https://github.com/apache/kafka/pull/22242#issuecomment-4407111122
Hi @AndrewJSchofield @lianetm would you be able to review this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20385: Add consumer-aware methods to ConsumerRebalanceListener [kafka]
Posted by "adikou (via GitHub)" <gi...@apache.org>.
adikou commented on code in PR #22242:
URL: https://github.com/apache/kafka/pull/22242#discussion_r3210455758
##########
clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerCallbackTest.java:
##########
@@ -218,7 +219,8 @@ private void testOnPartitionsAssignedCalledWithExpectedPartitions(
private void subscribeAndExpectOnPartitionsAssigned(Consumer<byte[], byte[]> consumer, List<String> topics, Collection<TopicPartition> expectedPartitionsInCallback) throws InterruptedException {
var partitionsAssigned = new AtomicBoolean(false);
AtomicReference<Collection<TopicPartition>> partitionsFromCallback = new AtomicReference<>();
- consumer.subscribe(topics, new ConsumerRebalanceListener() {
+ consumer.subscribe(topics);
+ consumer.setConsumerRebalanceListener(new ConsumerRebalanceListener() {
Review Comment:
Thanks, I'll go over the diff to swap the calls.
But, is there a concern since the KafkaConsumer is explicitly called out to be from single threads and we `acquireAndEnsureOpen` to check same threaded access in each call post subscription?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org