[jira] [Comment Edited] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

["Dmitry Pekar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14237796#comment-14237796 ] 

Dmitry Pekar edited comment on KAFKA-1792 at 12/8/14 11:48 AM:
---------------------------------------------------------------

[~nehanarkhede] Thank you for your comments. I've updated rb and added a patch for fixed items.
I can't agree with you about unit test for the algorithm. If it contains a bug or could be improved in future,
than we would not be able to guarantee it correctness after the fix/improvement if not having the unit-test. 

The above unit test, IMHO, already contains those scenarios, but may be I've missed some important scenario.
Could you please review it also?


was (Author: dmitry pekar):
[~nehanarkhede] Thank you for your comments. I've updated and added a patch for fixed items.
I can't agree with you about unit test for the algorithm. If it contains a bug or could be improved in future,
than we would not be able to guarantee it correctness after the fix/improvement if not having the unit-test. 

The above unit test, IMHO, already contains those scenarios, but may be I've missed some important scenario.
Could you please review it also?

> change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-1792
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1792
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: tools
>            Reporter: Dmitry Pekar
>            Assignee: Dmitry Pekar
>             Fix For: 0.8.3
>
>         Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch
>
>
> Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, 
> generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas assigned;
> - minimum of reassignments - number of replica moves between brokers will be minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)