You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/01/20 16:51:26 UTC
[jira] [Commented] (KAFKA-4677) Avoid unnecessary task movement
across threads of the same process during rebalance
[ https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832090#comment-15832090 ]
ASF GitHub Bot commented on KAFKA-4677:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/2411
[WIP] KAFKA-4677: Avoid unnecessary task movement across threads of the same process during rebalance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka kstreams-446
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2411.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2411
----
commit a899864e484e859e44a71ebe8c9890dedfc403cc
Author: Damian Guy <da...@gmail.com>
Date: 2017-01-20T15:19:28Z
Avoid unnecessary task movement across threads of the same process during rebalances
----
> Avoid unnecessary task movement across threads of the same process during rebalance
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-4677
> URL: https://issues.apache.org/jira/browse/KAFKA-4677
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.2.0
> Reporter: Damian Guy
> Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid expensive task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus we assume that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment flips the assignment for task B and C (task A is newly assigned to either thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for B and C does change to the other thread resulting in unnecessary task migration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)