You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Allen Wang <aw...@netflix.com.INVALID> on 2014/11/08 01:17:35 UTC
Add partitions with replica assignment in same command
I am trying to figure out how to add partitions and assign replicas using
one admin command. I tried kafka.admin.TopicCommand to increase the
partition number from 9 to 12 with the following options:
/apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper
${ZOOKEEPER} --alter --topic test_topic_4 --partitions 12
--replica-assignment 2:1,0:2,1:0,1:2,2:0,0:1,1:0,2:1,0:2,2:1,0:2,1:0
This gives me an error
Option "[replica-assignment]" can't be used with option"[partitions]"
Looking into the TopicCommand, alterTopic function seems to be able to
handle that but the command exits with the above error before this function
is invoked.
Is there any workaround or other recommended way to achieve this?
Thanks,
Allen
Re: Add partitions with replica assignment in same command
Posted by Neha Narkhede <ne...@gmail.com>.
When we first add partitions, will it change the
assignment of replicas for existing partitions?
Nope. It should not touch the existing partitions.
Also, will there be any issues executing the second reassignment
command which will change the assignment again for the new partitions added?
No. 2nd reassignment should work as expected.
On Wed, Nov 12, 2014 at 2:24 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:
> I found this JIRA
>
> https://issues.apache.org/jira/browse/KAFKA-1656
>
> Now, we have to use two commands to accomplish the goal - first add
> partitions using TopicCommand and then reassign replicas using
> ReassignPartitionsCommand. When we first add partitions, will it change the
> assignment of replicas for existing partitions? This is what we would like
> to avoid. Also, will there be any issues executing the second reassignment
> command which will change the assignment again for the new partitions
> added?
>
>
>
>
> On Sun, Nov 9, 2014 at 9:01 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Yes, it seems that we need to fix the tool to support that. It's probably
> > more intuitive to have TopicCommand just take the replica-assignment (for
> > the new partitions) when altering a topic. Could you file a jira?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang <aw...@netflix.com.invalid>
> > wrote:
> >
> > > I am trying to figure out how to add partitions and assign replicas
> using
> > > one admin command. I tried kafka.admin.TopicCommand to increase the
> > > partition number from 9 to 12 with the following options:
> > >
> > > /apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand
> --zookeeper
> > > ${ZOOKEEPER} --alter --topic test_topic_4 --partitions 12
> > > --replica-assignment 2:1,0:2,1:0,1:2,2:0,0:1,1:0,2:1,0:2,2:1,0:2,1:0
> > >
> > > This gives me an error
> > >
> > > Option "[replica-assignment]" can't be used with option"[partitions]"
> > >
> > > Looking into the TopicCommand, alterTopic function seems to be able to
> > > handle that but the command exits with the above error before this
> > function
> > > is invoked.
> > >
> > > Is there any workaround or other recommended way to achieve this?
> > >
> > > Thanks,
> > > Allen
> > >
> >
>
Re: Add partitions with replica assignment in same command
Posted by Jun Rao <ju...@gmail.com>.
If you add partitions, existing partitions' assignment won't change.
Changing the replica assignment subsequently should be fine.
Thanks,
Jun
On Wed, Nov 12, 2014 at 2:24 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:
> I found this JIRA
>
> https://issues.apache.org/jira/browse/KAFKA-1656
>
> Now, we have to use two commands to accomplish the goal - first add
> partitions using TopicCommand and then reassign replicas using
> ReassignPartitionsCommand. When we first add partitions, will it change the
> assignment of replicas for existing partitions? This is what we would like
> to avoid. Also, will there be any issues executing the second reassignment
> command which will change the assignment again for the new partitions
> added?
>
>
>
>
> On Sun, Nov 9, 2014 at 9:01 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Yes, it seems that we need to fix the tool to support that. It's probably
> > more intuitive to have TopicCommand just take the replica-assignment (for
> > the new partitions) when altering a topic. Could you file a jira?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang <aw...@netflix.com.invalid>
> > wrote:
> >
> > > I am trying to figure out how to add partitions and assign replicas
> using
> > > one admin command. I tried kafka.admin.TopicCommand to increase the
> > > partition number from 9 to 12 with the following options:
> > >
> > > /apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand
> --zookeeper
> > > ${ZOOKEEPER} --alter --topic test_topic_4 --partitions 12
> > > --replica-assignment 2:1,0:2,1:0,1:2,2:0,0:1,1:0,2:1,0:2,2:1,0:2,1:0
> > >
> > > This gives me an error
> > >
> > > Option "[replica-assignment]" can't be used with option"[partitions]"
> > >
> > > Looking into the TopicCommand, alterTopic function seems to be able to
> > > handle that but the command exits with the above error before this
> > function
> > > is invoked.
> > >
> > > Is there any workaround or other recommended way to achieve this?
> > >
> > > Thanks,
> > > Allen
> > >
> >
>
Re: Add partitions with replica assignment in same command
Posted by Allen Wang <aw...@netflix.com.INVALID>.
I found this JIRA
https://issues.apache.org/jira/browse/KAFKA-1656
Now, we have to use two commands to accomplish the goal - first add
partitions using TopicCommand and then reassign replicas using
ReassignPartitionsCommand. When we first add partitions, will it change the
assignment of replicas for existing partitions? This is what we would like
to avoid. Also, will there be any issues executing the second reassignment
command which will change the assignment again for the new partitions added?
On Sun, Nov 9, 2014 at 9:01 PM, Jun Rao <ju...@gmail.com> wrote:
> Yes, it seems that we need to fix the tool to support that. It's probably
> more intuitive to have TopicCommand just take the replica-assignment (for
> the new partitions) when altering a topic. Could you file a jira?
>
> Thanks,
>
> Jun
>
> On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang <aw...@netflix.com.invalid>
> wrote:
>
> > I am trying to figure out how to add partitions and assign replicas using
> > one admin command. I tried kafka.admin.TopicCommand to increase the
> > partition number from 9 to 12 with the following options:
> >
> > /apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper
> > ${ZOOKEEPER} --alter --topic test_topic_4 --partitions 12
> > --replica-assignment 2:1,0:2,1:0,1:2,2:0,0:1,1:0,2:1,0:2,2:1,0:2,1:0
> >
> > This gives me an error
> >
> > Option "[replica-assignment]" can't be used with option"[partitions]"
> >
> > Looking into the TopicCommand, alterTopic function seems to be able to
> > handle that but the command exits with the above error before this
> function
> > is invoked.
> >
> > Is there any workaround or other recommended way to achieve this?
> >
> > Thanks,
> > Allen
> >
>
Re: Add partitions with replica assignment in same command
Posted by Jun Rao <ju...@gmail.com>.
Yes, it seems that we need to fix the tool to support that. It's probably
more intuitive to have TopicCommand just take the replica-assignment (for
the new partitions) when altering a topic. Could you file a jira?
Thanks,
Jun
On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:
> I am trying to figure out how to add partitions and assign replicas using
> one admin command. I tried kafka.admin.TopicCommand to increase the
> partition number from 9 to 12 with the following options:
>
> /apps/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper
> ${ZOOKEEPER} --alter --topic test_topic_4 --partitions 12
> --replica-assignment 2:1,0:2,1:0,1:2,2:0,0:1,1:0,2:1,0:2,2:1,0:2,1:0
>
> This gives me an error
>
> Option "[replica-assignment]" can't be used with option"[partitions]"
>
> Looking into the TopicCommand, alterTopic function seems to be able to
> handle that but the command exits with the above error before this function
> is invoked.
>
> Is there any workaround or other recommended way to achieve this?
>
> Thanks,
> Allen
>