You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2015/01/09 21:02:34 UTC
[jira] [Commented] (KAFKA-1792) change behavior of --generate to
produce assignment config with fair replica distribution and minimal number
of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271791#comment-14271791 ]
Neha Narkhede commented on KAFKA-1792:
--------------------------------------
[~Dmitry Pekar] Not sure if you are waiting on me to proceed here. I'm waiting to see a proposal for how the user will interact with partition reassignment given the new way to generate assignment.
> change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
> Issue Type: Sub-task
> Components: tools
> Reporter: Dmitry Pekar
> Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt
>
>
> Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3,
> generate will create an assignment config which will redistribute replicas fairly across brokers [0..3]
> in the same way as those partitions were created from scratch. It will not take into consideration current replica
> assignment and accordingly will not try to minimize number of replica moves between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas assigned;
> - minimum of reassignments - number of replica moves between brokers will be minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3):
> - broker - 0, 1, 2, 3
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)