[jira] [Commented] (KAFKA-1313) Support adding replicas to existing topic partitions

["Geoffrey Anderson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347795#comment-14347795 ] 

Geoffrey Anderson commented on KAFKA-1313:
------------------------------------------

Quoting the Kafka docs:
The first step is to hand craft the custom reassignment plan in a json file-

> cat increase-replication-factor.json
{"version":1,
 "partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}

Then, use the json file with the --execute option to start the reassignment process-
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
-------

As I understand it, the pain point is "hand crafting" the reassignment plan. From there on out, updating replication factor is functionally identical to any other reassignment. So my proposal is to add another file type (--replication-factor-json-file) to kafka-reassign-partitions.sh --generate (and/or --rebalance per KIP 6) which allows users to specify desired replication_factor per topic/partition.

For example:
>cat update-replication-factor.json
{"version":1,
 "partitions":[{"topic":"foo","partition":0,"replication_factor":3}]}

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --replication-factor-json-file update-replication-factor.json --generate

The user would then feed generated output into kafka-reassign-partitions.sh --execute as before.

[~nehanarkhede], let me know if this seems reasonable/is in the ballpark of what you had in mind.

> Support adding replicas to existing topic partitions
> ----------------------------------------------------
>
>                 Key: KAFKA-1313
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1313
>             Project: Kafka
>          Issue Type: New Feature
>          Components: tools
>    Affects Versions: 0.8.0
>            Reporter: Marc Labbe
>            Assignee: Geoffrey Anderson
>            Priority: Critical
>              Labels: newbie++
>             Fix For: 0.9.0
>
>
> There is currently no easy way to add replicas to an existing topic partitions.
> For example, topic create-test has been created with ReplicationFactor=1: 
> Topic:create-test  PartitionCount:3    ReplicationFactor:1 Configs:
>     Topic: create-test Partition: 0    Leader: 1   Replicas: 1 Isr: 1
>     Topic: create-test Partition: 1    Leader: 2   Replicas: 2 Isr: 2
>     Topic: create-test Partition: 2    Leader: 3   Replicas: 3 Isr: 3
> I would like to increase the ReplicationFactor=2 (or more) so it shows up like this instead.
> Topic:create-test  PartitionCount:3    ReplicationFactor:2 Configs:
>     Topic: create-test Partition: 0    Leader: 1   Replicas: 1,2 Isr: 1,2
>     Topic: create-test Partition: 1    Leader: 2   Replicas: 2,3 Isr: 2,3
>     Topic: create-test Partition: 2    Leader: 3   Replicas: 3,1 Isr: 3,1
> Use cases for this:
> - adding brokers and thus increase fault tolerance
> - fixing human errors for topics created with wrong values



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)