You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "flashmouse (via GitHub)" <gi...@apache.org> on 2023/07/06 11:01:10 UTC
[GitHub] [kafka] flashmouse opened a new pull request, #13965: MINOR: CooperativeStickyAssignor may fail adjust assignment
flashmouse opened a new pull request, #13965:
URL: https://github.com/apache/kafka/pull/13965
ConstrainedAssignmentBuilder in AbstractStickyAssignor cannot build partitionsTransferringOwnership correctly, that would result in execute adjustAssignment fail in CooperativeStickyAssignor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-2109209243
@flashmouse can you rebase this PR please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599572094
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -58,7 +58,7 @@ public abstract class AbstractStickyAssignor extends AbstractPartitionAssignor {
// Keep track of the partitions being migrated from one consumer to another during assignment
// so the cooperative assignor can adjust the assignment
- protected Map<TopicPartition, String> partitionsTransferringOwnership = new HashMap<>();
+ public Map<TopicPartition, String> partitionsTransferringOwnership = new HashMap<>();
Review Comment:
that make sense, but set to private would result in many modifications in ``org.apache.kafka.clients.consumer.internals.AbstractStickyAssignorTest``, I think you mean set protected is enough? changed to protected now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman merged PR #13965:
URL: https://github.com/apache/kafka/pull/13965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] flashmouse commented on pull request #13965: KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-1633768249
@rreddy-22 I see you are familiar with this implementation, plz help me check this, thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-2111341726
Test failures are unrelated, merging to trunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] flashmouse commented on pull request #13965: KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-1633686392
please note another bugs in this implementation.
the function assignRackAwareRoundRobin of ConstrainedAssignmentBuilder have 2 logical error:
1. (about line 726)when assignmentCount >= minQuota, the consumer should be added to
unfilledMembersWithExactlyMinQuotaPartitions only if
currentNumMembersWithOverMinQuotaPartitions < expectedNumMembersWithOverMinQuotaPartitions
2. (about line 739)unfilledMembersWithExactlyMinQuotaPartitions should execute clear when
assignment.get(consumer).size() + 1 == maxQuota.
this error will lead to fail execute verifyUnfilledMembers, throw exception, raise new rebalance and return the same assignment result so group may stuck forever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599510047
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -739,8 +750,13 @@ private void assignRackAwareRoundRobin(List<TopicPartition> unassignedPartitions
int firstIndex = rackInfo.nextRackConsumer(unassignedPartition, unfilledMembersWithExactlyMinQuotaPartitions, 0);
if (firstIndex >= 0) {
consumer = unfilledMembersWithExactlyMinQuotaPartitions.get(firstIndex);
- if (assignment.get(consumer).size() + 1 == maxQuota)
+ if (assignment.get(consumer).size() + 1 == maxQuota) {
unfilledMembersWithExactlyMinQuotaPartitions.remove(firstIndex);
+ currentNumMembersWithOverMinQuotaPartitions++;
+ if (currentNumMembersWithOverMinQuotaPartitions == expectedNumMembersWithOverMinQuotaPartitions) {
Review Comment:
comment could be useful, added it. plz check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] flashmouse commented on pull request #13965: KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-1633765383
> please note another bugs in this implementation.
>
> the function assignRackAwareRoundRobin of ConstrainedAssignmentBuilder have 2 logical error:
>
> 1. (about line 726)when assignmentCount >= minQuota, the consumer should be added to
>
> unfilledMembersWithExactlyMinQuotaPartitions only if
>
> currentNumMembersWithOverMinQuotaPartitions < expectedNumMembersWithOverMinQuotaPartitions
>
> 2. (about line 739)unfilledMembersWithExactlyMinQuotaPartitions should execute clear when
>
> assignment.get(consumer).size() + 1 == maxQuota.
>
> this error will lead to fail execute verifyUnfilledMembers, throw exception, raise new rebalance and return the same assignment result so group may stuck forever.
I suggest ``verifyUnfilledMembers`` shouldn't throw Exception because the error it checks only caused to assignment not full balance, but is still a valid result, we cannot allow rebalance stuck forever in such situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599581134
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -211,6 +211,10 @@ public boolean isSticky() {
return partitionMovements.isSticky();
}
+ public Map<TopicPartition, String> getPartitionsTransferringOwnership() {
Review Comment:
I know, plz check again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599471029
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -739,8 +750,13 @@ private void assignRackAwareRoundRobin(List<TopicPartition> unassignedPartitions
int firstIndex = rackInfo.nextRackConsumer(unassignedPartition, unfilledMembersWithExactlyMinQuotaPartitions, 0);
if (firstIndex >= 0) {
consumer = unfilledMembersWithExactlyMinQuotaPartitions.get(firstIndex);
- if (assignment.get(consumer).size() + 1 == maxQuota)
+ if (assignment.get(consumer).size() + 1 == maxQuota) {
unfilledMembersWithExactlyMinQuotaPartitions.remove(firstIndex);
+ currentNumMembersWithOverMinQuotaPartitions++;
+ if (currentNumMembersWithOverMinQuotaPartitions == expectedNumMembersWithOverMinQuotaPartitions) {
Review Comment:
Let's add a comment above this line, too. Something like
```
// Clear this once the current num consumers over minQuota reaches the expected number since this
// means all consumers at minQuota are now considered filled
```
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -729,8 +739,9 @@ private void assignRackAwareRoundRobin(List<TopicPartition> unassignedPartitions
int assignmentCount = assignment.get(consumer).size() + 1;
if (assignmentCount >= minQuota) {
unfilledMembersWithUnderMinQuotaPartitions.remove(consumer);
- if (assignmentCount < maxQuota)
+ if (assignmentCount < maxQuota && (currentNumMembersWithOverMinQuotaPartitions < expectedNumMembersWithOverMinQuotaPartitions)) {
Review Comment:
This fix makes sense, good find. But let's add a comment because obviously this is difficult to understand if the bug slipped in.
Something like:
```
// Only add this consumer if the current num members at maxQuota is less than the expected number
// since a consumer at minQuota can only be considered unfilled if it's possible to add another partition,
// which would bump it to maxQuota and exceed the expectedNumMembersWithOverMinQuotaPartitions
```
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -579,7 +579,7 @@ protected List<TopicPartition> getAllTopicPartitions(List<String> sortedAllTopic
private class ConstrainedAssignmentBuilder extends AbstractAssignmentBuilder {
private final Set<TopicPartition> partitionsWithMultiplePreviousOwners;
- private final Set<TopicPartition> allRevokedPartitions;
+ private final Map<TopicPartition, String> mayRevokedPartitions;
Review Comment:
nit: this name is a bit confusing, how about `maybeRevokedPartitions`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599556043
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -58,7 +58,7 @@ public abstract class AbstractStickyAssignor extends AbstractPartitionAssignor {
// Keep track of the partitions being migrated from one consumer to another during assignment
// so the cooperative assignor can adjust the assignment
- protected Map<TopicPartition, String> partitionsTransferringOwnership = new HashMap<>();
+ public Map<TopicPartition, String> partitionsTransferringOwnership = new HashMap<>();
Review Comment:
Instead of making this public can you actually make it private and just add a getter method that returns it? We don't want any class fields being modifiable from the outside
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-2109370426
updated to latest commit now @ableegoldman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "rishiraj88 (via GitHub)" <gi...@apache.org>.
rishiraj88 commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599543619
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -579,7 +579,7 @@ protected List<TopicPartition> getAllTopicPartitions(List<String> sortedAllTopic
private class ConstrainedAssignmentBuilder extends AbstractAssignmentBuilder {
private final Set<TopicPartition> partitionsWithMultiplePreviousOwners;
- private final Set<TopicPartition> allRevokedPartitions;
+ private final Map<TopicPartition, String> mayRevokedPartitions;
Review Comment:
maybeRevokedPartitions sounds very comprehensive and short. It is indeed a nice choice of words.
More options for who do not prefer 'maybe' may be: possiblyRevokedPartitions, perhapsRevokedPartitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599577026
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -211,6 +211,10 @@ public boolean isSticky() {
return partitionMovements.isSticky();
}
+ public Map<TopicPartition, String> getPartitionsTransferringOwnership() {
Review Comment:
Forgot to mention, it's not a big deal but typically we don't include the "get" in getter names in the Kafka clients. Just a naming convention. ie this would be just `partitionsTransferringOwnership()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-1794032521
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "flashmouse (via GitHub)" <gi...@apache.org>.
flashmouse commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599515470
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -579,7 +579,7 @@ protected List<TopicPartition> getAllTopicPartitions(List<String> sortedAllTopic
private class ConstrainedAssignmentBuilder extends AbstractAssignmentBuilder {
private final Set<TopicPartition> partitionsWithMultiplePreviousOwners;
- private final Set<TopicPartition> allRevokedPartitions;
+ private final Map<TopicPartition, String> mayRevokedPartitions;
Review Comment:
modified, plz check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on code in PR #13965:
URL: https://github.com/apache/kafka/pull/13965#discussion_r1599572714
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java:
##########
@@ -579,7 +579,7 @@ protected List<TopicPartition> getAllTopicPartitions(List<String> sortedAllTopic
private class ConstrainedAssignmentBuilder extends AbstractAssignmentBuilder {
private final Set<TopicPartition> partitionsWithMultiplePreviousOwners;
- private final Set<TopicPartition> allRevokedPartitions;
+ private final Map<TopicPartition, String> mayRevokedPartitions;
Review Comment:
> More options for who do not prefer 'maybe' may be: possiblyRevokedPartitions, perhapsRevokedPartitions.
I actually considered both of those as well -- ultimately went with "maybe" since the variables in this class are all way too long already, but I'm happy with any of these choices 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15170: CooperativeStickyAssignor may fail adjust assignment [kafka]
Posted by "ableegoldman (via GitHub)" <gi...@apache.org>.
ableegoldman commented on PR #13965:
URL: https://github.com/apache/kafka/pull/13965#issuecomment-2111359637
Merged to trunk and cherrypicked back to 3.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org