You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Damian Guy (JIRA)" <ji...@apache.org> on 2017/01/20 13:02:26 UTC
[jira] [Created] (KAFKA-4677) Make StreamPartitionAssigner more
sticky when assigning to threads
Damian Guy created KAFKA-4677:
---------------------------------
Summary: Make StreamPartitionAssigner more sticky when assigning to threads
Key: KAFKA-4677
URL: https://issues.apache.org/jira/browse/KAFKA-4677
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 0.10.2.0
Reporter: Damian Guy
Assignee: Damian Guy
Fix For: 0.10.3.0
StreamPartitionAssigner tries to follow a sticky assignment policy to avoid expensive task migration. Currently, it does this in a best-effort approach.
We could observe a case, for which tasks did migrate for no good reason, thus we assume that the current implementation could be improved to be more sticky.
The concrete scenario is as follows:
assume we have topology with 3 tasks, A, B, C
assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
for some reason, thread 1 goes down and a rebalance gets triggered
thread 2 and 3 get their partitions revoked
sometimes (not sure what the exact condition for this is), the new assignment flips the assignment for task B and C (task A is newly assigned to either thread 2 or 3)
> possible new assignment 2(A,C) and 3-B
There is no obvious reason (like load-balancing) why the task assignment for B and C does change to the other thread resulting in unnecessary task migration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)