Re: Issue with kafka-topics.sh tool for adding new partitions with replica assignment

[Jason Rosenberg <jb...@squareup.com>]

Probably makes sense to file a Jira for this issue.

On Mon, May 11, 2015 at 8:28 AM, Stefan Schadwinkel <
stefan.schadwinkel@smaato.com> wrote:

> Hi,
>
> with Kafka 0.8 it is possible to add new partitions on newly added brokers
> and supply a partition assignment to put the new partitions mainly on the
> new brokers (4 and 5 are the new brokers):
>
> bin/kafka-add-partitions.sh --topic scale-test-001 \
> --partition 14 \
> --replica-assignment-list
> 4:5,4:1,4:2,4:3,4:5,4:1,4:2,5:3,5:4,5:1,5:2,5:3,5:4,5:1 \
> --zookeeper 127.0.0.1:2181
>
> For 0.8.1+ the kafka-add-partitions.sh tool was merged into
> kafka-topics.sh, but when you try to execute something similar you receive
> the following error (in Kafka 0.8.2.1):
>
> kafka_2.10-0.8.2.1$ bin/kafka-topics.sh --alter --topic scale-test-002 \
> > --zookeeper 127.0.0.1:2181 \
> > --partitions 35 \
> > --replica-assignment
>
>  2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,3:2,2:3,4:5,4:2,4:2,4:3,4:5,4:3,4:2,5:3,5:4,5:4,5:2,5:3,5:4,5:3
> Option "[replica-assignment]" can't be used with option"[partitions]"
>
>
> However, upon investigation of alterTopics in TopicCommand.scala the code
> it wants to execute is:
>
> val nPartitions = opts.options.valueOf(opts.partitionsOpt).intValue
> val replicaAssignmentStr = opts.options.valueOf(opts.replicaAssignmentOpt)
> AdminUtils.addPartitions(zkClient, topic, nPartitions,
> replicaAssignmentStr, config = configs)
>
> So assigning both the --partitions and the --replica-assignment parameters
> should be totally fine.
>
> The issue is with the following line in checkArgs:
>
> CommandLineUtils.checkInvalidArgs(parser, options, replicaAssignmentOpt,
>       allTopicLevelOpts -- Set(alterOpt, createOpt) + partitionsOpt +
> replicationFactorOpt)
>
> If it is removed, then the above command executes just fine. The created
> partitions are as well filled quite happily.
>
> I'm not fully sure what the correct configuration of the
> replicaAssignmentOpt should be, so I don't provide a patch, but it would be
> great if that could be fixed.
>
> Best regards,
> Stefan
>
>
> --
>
> *Dr. Stefan Schadwinkel*
> Senior Big Data Developer
> stefan.schadwinkel@smaato.com
>
>
>
>
> Smaato Inc.
> San Francisco – New York - Hamburg - Singapore
> www.smaato.com
>
>
>
>
>
> Germany:
> Valentinskamp 70, Emporio, 19th Floor
>
> 20355 Hamburg
>
>
> T          +49 (40) 3480 949 0
> F          +49 (40) 492 19 055
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above. If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited. If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>