You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "danoSF (via GitHub)" <gi...@apache.org> on 2026/06/16 21:28:31 UTC
[PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
danoSF opened a new pull request, #22593:
URL: https://github.com/apache/kafka/pull/22593
Under the classic rebalance protocol with static membership, a static member that rejoins during the Stable state with the same selected protocol but a different set of subscribed topics did not trigger a rebalance. The newly subscribed topics (for example those discovered by a topics.regex member) were therefore never assigned to the group until the whole group was bounced for longer than session.timeout.ms.
This is a consistency gap rather than a new capability. Every other classic-group join path that can change a member's subscription already rebalances: dynamic and known-member rejoins go through classicGroupJoinExistingMember, which rebalances whenever !member.matches(request.protocols()) (a byte equality of the embedded protocol metadata, so any subscription change trips it), and the classic-to-consumer upgrade bridge already bumps the group epoch via hasMemberSubscriptionChanged. The static-member replace path (updateStaticMemberThenRebalanceOrCompleteJoin) was the sole exception: to avoid needless rebalances when a static instance simply reconnects with a new member id, it short-circuited on the selected protocol name alone and never performed the metadata comparison the other paths rely on.
This patch closes the gap by also comparing the rejoining member's own previous and new subscribed topics. The subscription is parsed with the version-safe V0 prefix already used by ClassicGroup#computeSubscribedTopics (the coordinator has parsed the embedded classic subscription this way for years, e.g. for offset expiration), so no new parsing capability is introduced. Only the subscribed topics are compared; owned partitions and user data do not require a new assignment and must not trigger a rebalance. When the consumer protocol cannot be parsed the coordinator falls back to its protocol-based behaviour, mirroring computeSubscribedTopics. The previous subscription is captured before the member's protocols are replaced.
Tests: GroupMetadataManagerTest covers a subscription that grows and one that shrinks (rebalance), an owned-partitions-only change and an unchanged subscription across protocol versions (no rebalance), and a multi-member static group asserting the decision is based on the rejoining member's own subscription rather than the group-wide union. JoinGroupRequestTest adds an end-to-end test reproducing the reported scenario against a broker: a static member that rejoins with an additional topic triggers a rebalance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "chia7712 (via GitHub) via jira" <ji...@kafka.apache.org>.
chia7712 commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6799630985
@danoSF please fix the conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "danoSF (via GitHub)" <gi...@apache.org>.
danoSF commented on code in PR #22593:
URL: https://github.com/apache/kafka/pull/22593#discussion_r4158888385
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -7930,6 +7934,58 @@ private CoordinatorResult<Void, CoordinatorRecord> updateStaticMemberThenRebalan
return maybeCompleteJoinPhase(group);
}
+ /**
+ * Parses the set of topics a classic member is subscribed to under the given protocol. This is used to
+ * detect a subscription change when a static member rejoins during the Stable stage with the same
+ * selected protocol but a different subscription (for example a topics.regex member that discovered new
+ * topics). Without it the coordinator would return the cached assignment and the newly subscribed topics
+ * would never be assigned to the group.
+ *
+ * <p>Only the subscribed topics are extracted. Other parts of the embedded subscription, such as the
+ * owned partitions or the user data, do not require a new assignment and must not trigger a rebalance.
+ *
+ * <p>Empty is returned when the group is not using the consumer protocol, when no protocol is selected,
+ * or when the embedded consumer protocol cannot be parsed. In those cases the coordinator falls back to
+ * its protocol-based behaviour: two members whose subscriptions are both empty compare equal and do not
+ * trigger a rebalance. This mirrors {@link ClassicGroup#computeSubscribedTopics()}, which also leaves the
+ * coordinator unaware of the subscribed topics when the consumer protocol cannot be parsed.
+ *
+ * @param group The group.
+ * @param protocols The member's supported protocols.
+ * @param protocolName The protocol whose subscription is extracted, or null if none is selected.
+ * @return The set of subscribed topics, or empty if it cannot be determined.
+ */
+ private Optional<Set<String>> parseSubscribedTopics(
Review Comment:
Yes, the subscription deserialization was based on the logic in ClassicGroup#computeSubscribedTopics. I extracted the shared parsing into ClassicGroup.deserializeSubscribedTopics(byte[]) and updated both callers to use it. The existing method still aggregates topics across the group, while this path compares the old and new subscriptions for a single member. Thanks @chia7712 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-4785879573
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6110921066
> If we start deserializing embedded protocols, where do we stop?
The method computeSubscribedTopics is already doing that, right? This PR leverages the same logic to get the topics, so I don't see there is a new “where do we stop” issue here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6304753697
@lianetm thanks for your explanation. I think the “stop line” is clear that we SHOULD NOT parse more than what we already do. If KAFKA-12169 and KAFKA-10078 plan to parse more, we should recommend those users try the new protocol instead.
Also, KIP-1274 states “The Classic protocol remains supported on the broker side”, so my sense is that we still need to fix the bugs as much as possible :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "danoSF (via GitHub)" <gi...@apache.org>.
danoSF commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-5719358350
@lianetm & @dajac if you folks have capacity, would you mind taking a look at this one? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22593:
URL: https://github.com/apache/kafka/pull/22593#discussion_r4152701247
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -7930,6 +7934,58 @@ private CoordinatorResult<Void, CoordinatorRecord> updateStaticMemberThenRebalan
return maybeCompleteJoinPhase(group);
}
+ /**
+ * Parses the set of topics a classic member is subscribed to under the given protocol. This is used to
+ * detect a subscription change when a static member rejoins during the Stable stage with the same
+ * selected protocol but a different subscription (for example a topics.regex member that discovered new
+ * topics). Without it the coordinator would return the cached assignment and the newly subscribed topics
+ * would never be assigned to the group.
+ *
+ * <p>Only the subscribed topics are extracted. Other parts of the embedded subscription, such as the
+ * owned partitions or the user data, do not require a new assignment and must not trigger a rebalance.
+ *
+ * <p>Empty is returned when the group is not using the consumer protocol, when no protocol is selected,
+ * or when the embedded consumer protocol cannot be parsed. In those cases the coordinator falls back to
+ * its protocol-based behaviour: two members whose subscriptions are both empty compare equal and do not
+ * trigger a rebalance. This mirrors {@link ClassicGroup#computeSubscribedTopics()}, which also leaves the
+ * coordinator unaware of the subscribed topics when the consumer protocol cannot be parsed.
+ *
+ * @param group The group.
+ * @param protocols The member's supported protocols.
+ * @param protocolName The protocol whose subscription is extracted, or null if none is selected.
+ * @return The set of subscribed topics, or empty if it cannot be determined.
+ */
+ private Optional<Set<String>> parseSubscribedTopics(
Review Comment:
Is this logic from ClassicGroup#computeSubscribedTopics? If so, would you mind adding a helper method to reduce duplicate code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-5770755157
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "dajac (via GitHub) via jira" <ji...@kafka.apache.org>.
dajac commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6833863313
I still believe that we must not proceed with this.There are other case where the classic protocol is deficient because we don't parse the subscriptions. If we start triggering rebalance in this case based on the parsed subscription, we would need to re-consider all the triggers to be consistent and it is not a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-5953403868
@danoSF Thanks for the patch!
I am not really comfortable with this change because it breaks the encapsulation of the classic protocol. If we start deserializing embedded protocols, where do we stop? It can get messy pretty quickly if we start on this path. Another thing to be careful with is that a client may use `consumer` name for the embedded protocol without using the same as ours. We have seen cases like this.
I would rather prefer to get users on KIP-848 as the new consumer protocol address all these flaws in the classic protocol. At this point, if a user is impacted by this, upgrading seems to be a reasonable path. What do you guys think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "dajac (via GitHub) via jira" <ji...@kafka.apache.org>.
dajac commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6833883457
@danoSF Have you considered upgrading to the new protocol?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6224804386
Hey all! Chia agree that the embedded subscription is already being parsed, but I do see the "where do we stop" challenge too (actually in 2 ways, where to stop trying to interpret the embedded protocol to take decisions, and where to stop fixing related issues in the same area of the classic, that are greatly improved in the new protocol)
This PR fixes one trigger of the "static member stale assignment" issue (and needs to parse the embedded subscription to fix). But then we would still have the same wrong behaviour but under other triggers (metadata change, assignment dropped), and to fix we would need to parse some other part of the embedded protocol (the assignment). E.g. this 2 issues [KAFKA-12169](https://issues.apache.org/jira/browse/KAFKA-12169) and [KAFKA-10078](https://issues.apache.org/jira/browse/KAFKA-10078). Would we want to look at those too, same symptom, different fix/parsing (actually trickier fixes needing protocol parsing vs metadata)
Also to consider KIP-1274, that has the classic protocol on a deprecation timeline already (we can/will still fix, but cnsidering this). Just my 2 cents, open to hearing back :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-12759: Rebalance when a static member's subscription changes (classic protocol) [kafka]
Posted by "chia7712 (via GitHub) via jira" <ji...@kafka.apache.org>.
chia7712 commented on PR #22593:
URL: https://github.com/apache/kafka/pull/22593#issuecomment-6897194091
> we would need to re-consider all the triggers to be consistent and it is not a good idea.
@dajac Pardon me, I didn't catch the point of "consistent" here. Parsing the bytes already happens in the broker, so I guess that is not your point. I guess the phrase is about similar issues like [KAFKA-12169](https://issues.apache.org/jira/browse/KAFKA-12169) and [KAFKA-10078](https://issues.apache.org/jira/browse/KAFKA-10078), and we don't want to open the door to making the broker parse more and more bytes it should not know. Is that correct?
If so, I agree we should not go further. KAFKA-12169 and KAFKA-10078 need to parse more than what we do today, so they are out of scope and those users should move to the new protocol. This PR only uses the topics which `computeSubscribedTopics` already reads, so it seems to me this one is inside the line. Do you see a case where fixing this one forces us to fix the others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org