You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mjsax (via GitHub)" <gi...@apache.org> on 2026/07/29 18:05:43 UTC
[PR] MINOR: improve "streams" default rebalance listener [kafka]
mjsax opened a new pull request, #22991:
URL: https://github.com/apache/kafka/pull/22991
If the broker side assignor makes an error on task assignment, it could crash the client with a "duplicate taskId" error.
This PR adds a proper task-id deduplication guard plus new WARN logs to let the user know that it received some bad assignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #22991:
URL: https://github.com/apache/kafka/pull/22991#discussion_r3738278436
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListener.java:
##########
@@ -88,10 +91,14 @@ public void onTasksRevoked(final Set<StreamsRebalanceData.TaskId> tasks) {
@Override
public void onTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
final long start = time.milliseconds();
+ final StreamsRebalanceData.Assignment runningAssignment = deduplicateTasks(assignment);
final Map<TaskId, Set<TopicPartition>> activeTasksWithPartitions =
- pairWithTopicPartitions(assignment.activeTasks().stream());
+ pairWithTopicPartitions(runningAssignment.activeTasks().stream());
final Map<TaskId, Set<TopicPartition>> standbyTasksWithPartitions =
- pairWithTopicPartitions(Stream.concat(assignment.standbyTasks().stream(), assignment.warmupTasks().stream()));
+ pairWithTopicPartitions(Stream.concat(
+ runningAssignment.standbyTasks().stream(),
+ runningAssignment.warmupTasks().stream()
+ ));
log.info("Processing new assignment {} from Streams Rebalance Protocol", assignment);
Review Comment:
Should this log line use `runningAssignment` and get moved up to occur before the deduplication runs and potentially logs a `WARN`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22991:
URL: https://github.com/apache/kafka/pull/22991#discussion_r3753966771
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListener.java:
##########
@@ -88,10 +91,14 @@ public void onTasksRevoked(final Set<StreamsRebalanceData.TaskId> tasks) {
@Override
public void onTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
final long start = time.milliseconds();
+ final StreamsRebalanceData.Assignment runningAssignment = deduplicateTasks(assignment);
final Map<TaskId, Set<TopicPartition>> activeTasksWithPartitions =
- pairWithTopicPartitions(assignment.activeTasks().stream());
+ pairWithTopicPartitions(runningAssignment.activeTasks().stream());
final Map<TaskId, Set<TopicPartition>> standbyTasksWithPartitions =
- pairWithTopicPartitions(Stream.concat(assignment.standbyTasks().stream(), assignment.warmupTasks().stream()));
+ pairWithTopicPartitions(Stream.concat(
+ runningAssignment.standbyTasks().stream(),
+ runningAssignment.warmupTasks().stream()
+ ));
log.info("Processing new assignment {} from Streams Rebalance Protocol", assignment);
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #22991:
URL: https://github.com/apache/kafka/pull/22991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #22991:
URL: https://github.com/apache/kafka/pull/22991#issuecomment-5249875602
Merged to `trunk` and cherry-picked to `4.4`, `4.3`, and `4.2` branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22991:
URL: https://github.com/apache/kafka/pull/22991#discussion_r3753966771
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStreamsRebalanceListener.java:
##########
@@ -88,10 +91,14 @@ public void onTasksRevoked(final Set<StreamsRebalanceData.TaskId> tasks) {
@Override
public void onTasksAssigned(final StreamsRebalanceData.Assignment assignment) {
final long start = time.milliseconds();
+ final StreamsRebalanceData.Assignment runningAssignment = deduplicateTasks(assignment);
final Map<TaskId, Set<TopicPartition>> activeTasksWithPartitions =
- pairWithTopicPartitions(assignment.activeTasks().stream());
+ pairWithTopicPartitions(runningAssignment.activeTasks().stream());
final Map<TaskId, Set<TopicPartition>> standbyTasksWithPartitions =
- pairWithTopicPartitions(Stream.concat(assignment.standbyTasks().stream(), assignment.warmupTasks().stream()));
+ pairWithTopicPartitions(Stream.concat(
+ runningAssignment.standbyTasks().stream(),
+ runningAssignment.warmupTasks().stream()
+ ));
log.info("Processing new assignment {} from Streams Rebalance Protocol", assignment);
Review Comment:
Good catch about `runningAssignment`.
But not sure about WARN? Why would we want to change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: improve "streams" default rebalance listener [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #22991:
URL: https://github.com/apache/kafka/pull/22991#issuecomment-5121717125
I think we should cherry-pick this one back to 4.3 and 4.2 branches (older clients could still connect to newer brokers which will support custom broker side task assignors)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org