You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Debraj Manna <su...@gmail.com> on 2018/06/29 17:37:44 UTC
Partitions reassignment is failing in Kafka 1.1.0
Hi
I altered a topic like below in kafka 1.1.0
/home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181
--alter --topic Topic3 --config min.insync.replicas=2
But whenever I am trying to verify the reassignment it is showing the below
exception
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --reassignment-json-file
/home/ubuntu/deploy/kafka/reassignment.json --verify
Partitions reassignment failed due to Size of replicas list Vector(3,
0, 2) is different from size of log dirs list Vector(any) for
partition Topic3-7
kafka.common.AdminCommandFailedException: Size of replicas list
Vector(3, 0, 2) is different from size of log dirs list Vector(any)
for partition Topic3-7
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262)
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251)
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249)
at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84)
at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58)
at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
My reassignment.json & server.properties is attached. Same thing used to
work fine in kafka 0.10. Can someone let me what is going wrong? Is
anything changed related to this in kafka 1.1.0 ?
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
Thanks
On Sat, Jun 30, 2018 at 12:36 PM, Manikumar <ma...@gmail.com>
wrote:
> It will be taken as "any" directory for each replica, which means replica
> will placed on any one of the
> configured directory on that broker.
>
> Since it is "log_dirs" optional, you can remove from the json.
>
> On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <su...@gmail.com>
> wrote:
>
> > It is problem on my side. The code was changing the replicas count but
> not
> > the log_dirs. Since I am migrating from 0.10 this part of the code was
> not
> > changed.
> >
> > I have a follow up question what is the default value of log_dirs if I
> > don't specify it in reassignment.json ?
> >
> > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <subharaj.manna@gmail.com
> >
> > wrote:
> >
> > > I am generating the reassignent.json like below
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> -1 >
> > /home/ubuntu/deploy/kafka/reassignment.json"
> > >
> > > Then I am doing the reassignment using the generated file
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --execute --reassignment-json-file
> > /home/ubuntu/deploy/kafka/reassignment.json
> > >
> > > kafka-reassign-partitions.sh helps states
> > >
> > > The JSON file with the partition reassignment configurationThe format
> to
> > >> use is -
> > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> "log_dirs"
> > is
> > >> optional. When it is specified, its length must equal the length of
> the
> > >> replicas list. The value in this list can be either "any" or the
> > absolution
> > >> path of the log directory on the broker. If absolute log directory
> path
> > is
> > >> specified, it is currently required that the replica has not already
> > been
> > >> created on that broker. The replica will then be created in the
> > specified
> > >> log directory on the broker later.
> > >
> > >
> > > So it appears reassignment json that is generated by
> > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> from
> > my
> > > side. ?
> > >
> > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> subharaj.manna@gmail.com
> > >
> > > wrote:
> > >
> > >> Please find the server.properties from one of the broker.
> > >>
> > >> broker.id=0
> > >> port=9092
> > >> num.network.threads=3
> > >> num.io.threads=8
> > >> socket.send.buffer.bytes=102400
> > >> socket.receive.buffer.bytes=102400
> > >> socket.request.max.bytes=104857600
> > >> log.dirs=/var/lib/kafka/kafka-logs
> > >> num.recovery.threads.per.data.dir=1
> > >> log.retention.hours=36
> > >> log.retention.bytes=1073741824
> > >> log.segment.bytes=536870912
> > >> log.retention.check.interval.ms=300000
> > >> log.cleaner.enable=false
> > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > >> message.max.bytes=15000000
> > >> replica.fetch.max.bytes=15000000
> > >> auto.create.topics.enable=true
> > >> zookeeper.connection.timeout.ms=6000
> > >> unclean.leader.election.enable=false
> > >> delete.topic.enable=false
> > >> offsets.topic.replication.factor=1
> > >> transaction.state.log.replication.factor=1
> > >> transaction.state.log.min.isr=1
> > >>
> > >> I have placed server.log from a broker at https://gist.github.com/deb
> > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > >>
> > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >>> Seems to be related to KIP-113.
> > >>>
> > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> content
> > ?
> > >>>
> > >>> If you can pastebin logs from broker, that should help.
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > subharaj.manna@gmail.com
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Hi
> > >>> >
> > >>> > I altered a topic like below in kafka 1.1.0
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > >>> 127.0.0.1:2181
> > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > >>> >
> > >>> > But whenever I am trying to verify the reassignment it is showing
> the
> > >>> > below exception
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > >>> >
> > >>> > Partitions reassignment failed due to Size of replicas list
> Vector(3,
> > >>> 0, 2) is different from size of log dirs list Vector(any) for
> partition
> > >>> Topic3-7
> > >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> > >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any)
> for
> > >>> partition Topic3-7
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > >>> ReassignPartitionsCommand.scala:262)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > >>> ReassignPartitionsCommand.scala:251)
> > >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:
> 891)
> > >>> > at scala.collection.AbstractIterator.foreach(
> Iterator.scala:133
> > >>> 4)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > >>> sCommand.scala:251)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > >>> sCommand.scala:250)
> > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> parsePartitionReassig
> > >>> nmentData(ReassignPartitionsCommand.scala:249)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> verifyAssignment(Reas
> > >>> signPartitionsCommand.scala:90)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> verifyAssignment(Reas
> > >>> signPartitionsCommand.scala:84)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> main(ReassignPartitio
> > >>> nsCommand.scala:58)
> > >>> > at kafka.admin.ReassignPartitionsCommand.
> main(ReassignPartition
> > >>> sCommand.scala)
> > >>> >
> > >>> >
> > >>> > My reassignment.json & server.properties is attached. Same thing
> used
> > >>> to
> > >>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
> > >>> > anything changed related to this in kafka 1.1.0 ?
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Ted Yu <yu...@gmail.com>.
See KAFKA-7124
Cheers
On Sun, Jul 1, 2018 at 3:26 AM, Debraj Manna <su...@gmail.com>
wrote:
> Is there any bug filed for this ?
>
> On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Code snippet from ReassignPartitionsCommand.scala :
> >
> > "log_dirs" -> replicas.map(r =>
> > replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
> > tp.partition, r), AnyLogDir)).asJava
> >
> > We know that the appearance of "any" was due to the OrElse clause.
> > Arguably there is a bug in the above code that the number of AnyLogDir
> > should match the length of the replicas list, or "log_dirs" should be
> > omitted in such case.
> >
> > On Sat, Jun 30, 2018 at 12:06 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > It will be taken as "any" directory for each replica, which means
> replica
> > > will placed on any one of the
> > > configured directory on that broker.
> > >
> > > Since it is "log_dirs" optional, you can remove from the json.
> > >
> > > On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <
> subharaj.manna@gmail.com>
> > > wrote:
> > >
> > > > It is problem on my side. The code was changing the replicas count
> but
> > > not
> > > > the log_dirs. Since I am migrating from 0.10 this part of the code
> was
> > > not
> > > > changed.
> > > >
> > > > I have a follow up question what is the default value of log_dirs if
> I
> > > > don't specify it in reassignment.json ?
> > > >
> > > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <
> > subharaj.manna@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I am generating the reassignent.json like below
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > > > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s'
> |tail
> > > -1 >
> > > > /home/ubuntu/deploy/kafka/reassignment.json"
> > > > >
> > > > > Then I am doing the reassignment using the generated file
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --execute --reassignment-json-file
> > > > /home/ubuntu/deploy/kafka/reassignment.json
> > > > >
> > > > > kafka-reassign-partitions.sh helps states
> > > > >
> > > > > The JSON file with the partition reassignment configurationThe
> format
> > > to
> > > > >> use is -
> > > > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas":
> [1,2,3],
> > > > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> > > "log_dirs"
> > > > is
> > > > >> optional. When it is specified, its length must equal the length
> of
> > > the
> > > > >> replicas list. The value in this list can be either "any" or the
> > > > absolution
> > > > >> path of the log directory on the broker. If absolute log directory
> > > path
> > > > is
> > > > >> specified, it is currently required that the replica has not
> already
> > > > been
> > > > >> created on that broker. The replica will then be created in the
> > > > specified
> > > > >> log directory on the broker later.
> > > > >
> > > > >
> > > > > So it appears reassignment json that is generated by
> > > > > kafka-reassign-partions.sh is creating an issue with logdirs. Is
> this
> > > > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> > > from
> > > > my
> > > > > side. ?
> > > > >
> > > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> > > subharaj.manna@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Please find the server.properties from one of the broker.
> > > > >>
> > > > >> broker.id=0
> > > > >> port=9092
> > > > >> num.network.threads=3
> > > > >> num.io.threads=8
> > > > >> socket.send.buffer.bytes=102400
> > > > >> socket.receive.buffer.bytes=102400
> > > > >> socket.request.max.bytes=104857600
> > > > >> log.dirs=/var/lib/kafka/kafka-logs
> > > > >> num.recovery.threads.per.data.dir=1
> > > > >> log.retention.hours=36
> > > > >> log.retention.bytes=1073741824
> > > > >> log.segment.bytes=536870912
> > > > >> log.retention.check.interval.ms=300000
> > > > >> log.cleaner.enable=false
> > > > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > > > >> message.max.bytes=15000000
> > > > >> replica.fetch.max.bytes=15000000
> > > > >> auto.create.topics.enable=true
> > > > >> zookeeper.connection.timeout.ms=6000
> > > > >> unclean.leader.election.enable=false
> > > > >> delete.topic.enable=false
> > > > >> offsets.topic.replication.factor=1
> > > > >> transaction.state.log.replication.factor=1
> > > > >> transaction.state.log.min.isr=1
> > > > >>
> > > > >> I have placed server.log from a broker at
> > https://gist.github.com/deb
> > > > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > > > >>
> > > > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >>
> > > > >>> Seems to be related to KIP-113.
> > > > >>>
> > > > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> > > content
> > > > ?
> > > > >>>
> > > > >>> If you can pastebin logs from broker, that should help.
> > > > >>>
> > > > >>> Thanks
> > > > >>>
> > > > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > > > subharaj.manna@gmail.com
> > > > >>> >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi
> > > > >>> >
> > > > >>> > I altered a topic like below in kafka 1.1.0
> > > > >>> >
> > > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > > > >>> 127.0.0.1:2181
> > > > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > > > >>> >
> > > > >>> > But whenever I am trying to verify the reassignment it is
> showing
> > > the
> > > > >>> > below exception
> > > > >>> >
> > > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > > > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > > > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > > > >>> >
> > > > >>> > Partitions reassignment failed due to Size of replicas list
> > > Vector(3,
> > > > >>> 0, 2) is different from size of log dirs list Vector(any) for
> > > partition
> > > > >>> Topic3-7
> > > > >>> > kafka.common.AdminCommandFailedException: Size of replicas
> list
> > > > >>> Vector(3, 0, 2) is different from size of log dirs list
> Vector(any)
> > > for
> > > > >>> partition Topic3-7
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > > >>> ReassignPartitionsCommand.scala:262)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > > >>> ReassignPartitionsCommand.scala:251)
> > > > >>> > at scala.collection.Iterator$
> class.foreach(Iterator.scala:
> > > 891)
> > > > >>> > at scala.collection.AbstractIterator.foreach(
> > > Iterator.scala:133
> > > > >>> 4)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > > >>> sCommand.scala:251)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > > >>> sCommand.scala:250)
> > > > >>> > at scala.collection.immutable.
> List.foreach(List.scala:392)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > > anonfun$parsePartitio
> > > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> > > > >>> > at scala.collection.immutable.
> List.foreach(List.scala:392)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > parsePartitionReassig
> > > > >>> nmentData(ReassignPartitionsCommand.scala:249)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > verifyAssignment(Reas
> > > > >>> signPartitionsCommand.scala:90)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > verifyAssignment(Reas
> > > > >>> signPartitionsCommand.scala:84)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > > main(ReassignPartitio
> > > > >>> nsCommand.scala:58)
> > > > >>> > at kafka.admin.ReassignPartitionsCommand.
> > > main(ReassignPartition
> > > > >>> sCommand.scala)
> > > > >>> >
> > > > >>> >
> > > > >>> > My reassignment.json & server.properties is attached. Same
> thing
> > > used
> > > > >>> to
> > > > >>> > work fine in kafka 0.10. Can someone let me what is going
> wrong?
> > Is
> > > > >>> > anything changed related to this in kafka 1.1.0 ?
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
Is there any bug filed for this ?
On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu <yu...@gmail.com> wrote:
> Code snippet from ReassignPartitionsCommand.scala :
>
> "log_dirs" -> replicas.map(r =>
> replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
> tp.partition, r), AnyLogDir)).asJava
>
> We know that the appearance of "any" was due to the OrElse clause.
> Arguably there is a bug in the above code that the number of AnyLogDir
> should match the length of the replicas list, or "log_dirs" should be
> omitted in such case.
>
> On Sat, Jun 30, 2018 at 12:06 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > It will be taken as "any" directory for each replica, which means replica
> > will placed on any one of the
> > configured directory on that broker.
> >
> > Since it is "log_dirs" optional, you can remove from the json.
> >
> > On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <su...@gmail.com>
> > wrote:
> >
> > > It is problem on my side. The code was changing the replicas count but
> > not
> > > the log_dirs. Since I am migrating from 0.10 this part of the code was
> > not
> > > changed.
> > >
> > > I have a follow up question what is the default value of log_dirs if I
> > > don't specify it in reassignment.json ?
> > >
> > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <
> subharaj.manna@gmail.com
> > >
> > > wrote:
> > >
> > > > I am generating the reassignent.json like below
> > > >
> > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> --zookeeper
> > > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> > -1 >
> > > /home/ubuntu/deploy/kafka/reassignment.json"
> > > >
> > > > Then I am doing the reassignment using the generated file
> > > >
> > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> --zookeeper
> > > 127.0.0.1:2181 --execute --reassignment-json-file
> > > /home/ubuntu/deploy/kafka/reassignment.json
> > > >
> > > > kafka-reassign-partitions.sh helps states
> > > >
> > > > The JSON file with the partition reassignment configurationThe format
> > to
> > > >> use is -
> > > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> > "log_dirs"
> > > is
> > > >> optional. When it is specified, its length must equal the length of
> > the
> > > >> replicas list. The value in this list can be either "any" or the
> > > absolution
> > > >> path of the log directory on the broker. If absolute log directory
> > path
> > > is
> > > >> specified, it is currently required that the replica has not already
> > > been
> > > >> created on that broker. The replica will then be created in the
> > > specified
> > > >> log directory on the broker later.
> > > >
> > > >
> > > > So it appears reassignment json that is generated by
> > > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> > from
> > > my
> > > > side. ?
> > > >
> > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> > subharaj.manna@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Please find the server.properties from one of the broker.
> > > >>
> > > >> broker.id=0
> > > >> port=9092
> > > >> num.network.threads=3
> > > >> num.io.threads=8
> > > >> socket.send.buffer.bytes=102400
> > > >> socket.receive.buffer.bytes=102400
> > > >> socket.request.max.bytes=104857600
> > > >> log.dirs=/var/lib/kafka/kafka-logs
> > > >> num.recovery.threads.per.data.dir=1
> > > >> log.retention.hours=36
> > > >> log.retention.bytes=1073741824
> > > >> log.segment.bytes=536870912
> > > >> log.retention.check.interval.ms=300000
> > > >> log.cleaner.enable=false
> > > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > > >> message.max.bytes=15000000
> > > >> replica.fetch.max.bytes=15000000
> > > >> auto.create.topics.enable=true
> > > >> zookeeper.connection.timeout.ms=6000
> > > >> unclean.leader.election.enable=false
> > > >> delete.topic.enable=false
> > > >> offsets.topic.replication.factor=1
> > > >> transaction.state.log.replication.factor=1
> > > >> transaction.state.log.min.isr=1
> > > >>
> > > >> I have placed server.log from a broker at
> https://gist.github.com/deb
> > > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > > >>
> > > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >>
> > > >>> Seems to be related to KIP-113.
> > > >>>
> > > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> > content
> > > ?
> > > >>>
> > > >>> If you can pastebin logs from broker, that should help.
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > > subharaj.manna@gmail.com
> > > >>> >
> > > >>> wrote:
> > > >>>
> > > >>> > Hi
> > > >>> >
> > > >>> > I altered a topic like below in kafka 1.1.0
> > > >>> >
> > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > > >>> 127.0.0.1:2181
> > > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > > >>> >
> > > >>> > But whenever I am trying to verify the reassignment it is showing
> > the
> > > >>> > below exception
> > > >>> >
> > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > > >>> >
> > > >>> > Partitions reassignment failed due to Size of replicas list
> > Vector(3,
> > > >>> 0, 2) is different from size of log dirs list Vector(any) for
> > partition
> > > >>> Topic3-7
> > > >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> > > >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any)
> > for
> > > >>> partition Topic3-7
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > >>> ReassignPartitionsCommand.scala:262)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > > >>> ReassignPartitionsCommand.scala:251)
> > > >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:
> > 891)
> > > >>> > at scala.collection.AbstractIterator.foreach(
> > Iterator.scala:133
> > > >>> 4)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > >>> sCommand.scala:251)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > > >>> sCommand.scala:250)
> > > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$$
> > anonfun$parsePartitio
> > > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> > > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > parsePartitionReassig
> > > >>> nmentData(ReassignPartitionsCommand.scala:249)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > verifyAssignment(Reas
> > > >>> signPartitionsCommand.scala:90)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > verifyAssignment(Reas
> > > >>> signPartitionsCommand.scala:84)
> > > >>> > at kafka.admin.ReassignPartitionsCommand$.
> > main(ReassignPartitio
> > > >>> nsCommand.scala:58)
> > > >>> > at kafka.admin.ReassignPartitionsCommand.
> > main(ReassignPartition
> > > >>> sCommand.scala)
> > > >>> >
> > > >>> >
> > > >>> > My reassignment.json & server.properties is attached. Same thing
> > used
> > > >>> to
> > > >>> > work fine in kafka 0.10. Can someone let me what is going wrong?
> Is
> > > >>> > anything changed related to this in kafka 1.1.0 ?
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Ted Yu <yu...@gmail.com>.
Code snippet from ReassignPartitionsCommand.scala :
"log_dirs" -> replicas.map(r =>
replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
tp.partition, r), AnyLogDir)).asJava
We know that the appearance of "any" was due to the OrElse clause.
Arguably there is a bug in the above code that the number of AnyLogDir
should match the length of the replicas list, or "log_dirs" should be
omitted in such case.
On Sat, Jun 30, 2018 at 12:06 AM, Manikumar <ma...@gmail.com>
wrote:
> It will be taken as "any" directory for each replica, which means replica
> will placed on any one of the
> configured directory on that broker.
>
> Since it is "log_dirs" optional, you can remove from the json.
>
> On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <su...@gmail.com>
> wrote:
>
> > It is problem on my side. The code was changing the replicas count but
> not
> > the log_dirs. Since I am migrating from 0.10 this part of the code was
> not
> > changed.
> >
> > I have a follow up question what is the default value of log_dirs if I
> > don't specify it in reassignment.json ?
> >
> > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <subharaj.manna@gmail.com
> >
> > wrote:
> >
> > > I am generating the reassignent.json like below
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> -1 >
> > /home/ubuntu/deploy/kafka/reassignment.json"
> > >
> > > Then I am doing the reassignment using the generated file
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --execute --reassignment-json-file
> > /home/ubuntu/deploy/kafka/reassignment.json
> > >
> > > kafka-reassign-partitions.sh helps states
> > >
> > > The JSON file with the partition reassignment configurationThe format
> to
> > >> use is -
> > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> "log_dirs"
> > is
> > >> optional. When it is specified, its length must equal the length of
> the
> > >> replicas list. The value in this list can be either "any" or the
> > absolution
> > >> path of the log directory on the broker. If absolute log directory
> path
> > is
> > >> specified, it is currently required that the replica has not already
> > been
> > >> created on that broker. The replica will then be created in the
> > specified
> > >> log directory on the broker later.
> > >
> > >
> > > So it appears reassignment json that is generated by
> > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> from
> > my
> > > side. ?
> > >
> > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> subharaj.manna@gmail.com
> > >
> > > wrote:
> > >
> > >> Please find the server.properties from one of the broker.
> > >>
> > >> broker.id=0
> > >> port=9092
> > >> num.network.threads=3
> > >> num.io.threads=8
> > >> socket.send.buffer.bytes=102400
> > >> socket.receive.buffer.bytes=102400
> > >> socket.request.max.bytes=104857600
> > >> log.dirs=/var/lib/kafka/kafka-logs
> > >> num.recovery.threads.per.data.dir=1
> > >> log.retention.hours=36
> > >> log.retention.bytes=1073741824
> > >> log.segment.bytes=536870912
> > >> log.retention.check.interval.ms=300000
> > >> log.cleaner.enable=false
> > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > >> message.max.bytes=15000000
> > >> replica.fetch.max.bytes=15000000
> > >> auto.create.topics.enable=true
> > >> zookeeper.connection.timeout.ms=6000
> > >> unclean.leader.election.enable=false
> > >> delete.topic.enable=false
> > >> offsets.topic.replication.factor=1
> > >> transaction.state.log.replication.factor=1
> > >> transaction.state.log.min.isr=1
> > >>
> > >> I have placed server.log from a broker at https://gist.github.com/deb
> > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > >>
> > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >>> Seems to be related to KIP-113.
> > >>>
> > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> content
> > ?
> > >>>
> > >>> If you can pastebin logs from broker, that should help.
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > subharaj.manna@gmail.com
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Hi
> > >>> >
> > >>> > I altered a topic like below in kafka 1.1.0
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > >>> 127.0.0.1:2181
> > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > >>> >
> > >>> > But whenever I am trying to verify the reassignment it is showing
> the
> > >>> > below exception
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > >>> >
> > >>> > Partitions reassignment failed due to Size of replicas list
> Vector(3,
> > >>> 0, 2) is different from size of log dirs list Vector(any) for
> partition
> > >>> Topic3-7
> > >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> > >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any)
> for
> > >>> partition Topic3-7
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > >>> ReassignPartitionsCommand.scala:262)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> > >>> ReassignPartitionsCommand.scala:251)
> > >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:
> 891)
> > >>> > at scala.collection.AbstractIterator.foreach(
> Iterator.scala:133
> > >>> 4)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > >>> sCommand.scala:251)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> > >>> sCommand.scala:250)
> > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
> > >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> > >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> parsePartitionReassig
> > >>> nmentData(ReassignPartitionsCommand.scala:249)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> verifyAssignment(Reas
> > >>> signPartitionsCommand.scala:90)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> verifyAssignment(Reas
> > >>> signPartitionsCommand.scala:84)
> > >>> > at kafka.admin.ReassignPartitionsCommand$.
> main(ReassignPartitio
> > >>> nsCommand.scala:58)
> > >>> > at kafka.admin.ReassignPartitionsCommand.
> main(ReassignPartition
> > >>> sCommand.scala)
> > >>> >
> > >>> >
> > >>> > My reassignment.json & server.properties is attached. Same thing
> used
> > >>> to
> > >>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
> > >>> > anything changed related to this in kafka 1.1.0 ?
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Manikumar <ma...@gmail.com>.
It will be taken as "any" directory for each replica, which means replica
will placed on any one of the
configured directory on that broker.
Since it is "log_dirs" optional, you can remove from the json.
On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <su...@gmail.com>
wrote:
> It is problem on my side. The code was changing the replicas count but not
> the log_dirs. Since I am migrating from 0.10 this part of the code was not
> changed.
>
> I have a follow up question what is the default value of log_dirs if I
> don't specify it in reassignment.json ?
>
> On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <su...@gmail.com>
> wrote:
>
> > I am generating the reassignent.json like below
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --generate --topics-to-move-json-file
> /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 >
> /home/ubuntu/deploy/kafka/reassignment.json"
> >
> > Then I am doing the reassignment using the generated file
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --execute --reassignment-json-file
> /home/ubuntu/deploy/kafka/reassignment.json
> >
> > kafka-reassign-partitions.sh helps states
> >
> > The JSON file with the partition reassignment configurationThe format to
> >> use is -
> >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs"
> is
> >> optional. When it is specified, its length must equal the length of the
> >> replicas list. The value in this list can be either "any" or the
> absolution
> >> path of the log directory on the broker. If absolute log directory path
> is
> >> specified, it is currently required that the replica has not already
> been
> >> created on that broker. The replica will then be created in the
> specified
> >> log directory on the broker later.
> >
> >
> > So it appears reassignment json that is generated by
> > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > some issue in kafka-reassign-partitions.sh or some misconfiguration from
> my
> > side. ?
> >
> > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <subharaj.manna@gmail.com
> >
> > wrote:
> >
> >> Please find the server.properties from one of the broker.
> >>
> >> broker.id=0
> >> port=9092
> >> num.network.threads=3
> >> num.io.threads=8
> >> socket.send.buffer.bytes=102400
> >> socket.receive.buffer.bytes=102400
> >> socket.request.max.bytes=104857600
> >> log.dirs=/var/lib/kafka/kafka-logs
> >> num.recovery.threads.per.data.dir=1
> >> log.retention.hours=36
> >> log.retention.bytes=1073741824
> >> log.segment.bytes=536870912
> >> log.retention.check.interval.ms=300000
> >> log.cleaner.enable=false
> >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> >> message.max.bytes=15000000
> >> replica.fetch.max.bytes=15000000
> >> auto.create.topics.enable=true
> >> zookeeper.connection.timeout.ms=6000
> >> unclean.leader.election.enable=false
> >> delete.topic.enable=false
> >> offsets.topic.replication.factor=1
> >> transaction.state.log.replication.factor=1
> >> transaction.state.log.min.isr=1
> >>
> >> I have placed server.log from a broker at https://gist.github.com/deb
> >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> >>
> >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> Seems to be related to KIP-113.
> >>>
> >>> server.properties didn't go thru. Do you mind pastebin'ing its content
> ?
> >>>
> >>> If you can pastebin logs from broker, that should help.
> >>>
> >>> Thanks
> >>>
> >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> subharaj.manna@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi
> >>> >
> >>> > I altered a topic like below in kafka 1.1.0
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> >>> 127.0.0.1:2181
> >>> > --alter --topic Topic3 --config min.insync.replicas=2
> >>> >
> >>> > But whenever I am trying to verify the reassignment it is showing the
> >>> > below exception
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> >>> >
> >>> > Partitions reassignment failed due to Size of replicas list Vector(3,
> >>> 0, 2) is different from size of log dirs list Vector(any) for partition
> >>> Topic3-7
> >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
> >>> partition Topic3-7
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:262)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:251)
> >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> >>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:133
> >>> 4)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> >>> sCommand.scala:251)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
> >>> sCommand.scala:250)
> >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> >>> > at scala.collection.immutable.List.foreach(List.scala:392)
> >>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig
> >>> nmentData(ReassignPartitionsCommand.scala:249)
> >>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
> >>> signPartitionsCommand.scala:90)
> >>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
> >>> signPartitionsCommand.scala:84)
> >>> > at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitio
> >>> nsCommand.scala:58)
> >>> > at kafka.admin.ReassignPartitionsCommand.main(ReassignPartition
> >>> sCommand.scala)
> >>> >
> >>> >
> >>> > My reassignment.json & server.properties is attached. Same thing used
> >>> to
> >>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
> >>> > anything changed related to this in kafka 1.1.0 ?
> >>> >
> >>>
> >>
> >>
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
It is problem on my side. The code was changing the replicas count but not
the log_dirs. Since I am migrating from 0.10 this part of the code was not
changed.
I have a follow up question what is the default value of log_dirs if I
don't specify it in reassignment.json ?
On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <su...@gmail.com>
wrote:
> I am generating the reassignent.json like below
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --generate --topics-to-move-json-file /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 > /home/ubuntu/deploy/kafka/reassignment.json"
>
> Then I am doing the reassignment using the generated file
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --execute --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json
>
> kafka-reassign-partitions.sh helps states
>
> The JSON file with the partition reassignment configurationThe format to
>> use is -
>> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
>> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" is
>> optional. When it is specified, its length must equal the length of the
>> replicas list. The value in this list can be either "any" or the absolution
>> path of the log directory on the broker. If absolute log directory path is
>> specified, it is currently required that the replica has not already been
>> created on that broker. The replica will then be created in the specified
>> log directory on the broker later.
>
>
> So it appears reassignment json that is generated by
> kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> some issue in kafka-reassign-partitions.sh or some misconfiguration from my
> side. ?
>
> On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <su...@gmail.com>
> wrote:
>
>> Please find the server.properties from one of the broker.
>>
>> broker.id=0
>> port=9092
>> num.network.threads=3
>> num.io.threads=8
>> socket.send.buffer.bytes=102400
>> socket.receive.buffer.bytes=102400
>> socket.request.max.bytes=104857600
>> log.dirs=/var/lib/kafka/kafka-logs
>> num.recovery.threads.per.data.dir=1
>> log.retention.hours=36
>> log.retention.bytes=1073741824
>> log.segment.bytes=536870912
>> log.retention.check.interval.ms=300000
>> log.cleaner.enable=false
>> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
>> message.max.bytes=15000000
>> replica.fetch.max.bytes=15000000
>> auto.create.topics.enable=true
>> zookeeper.connection.timeout.ms=6000
>> unclean.leader.election.enable=false
>> delete.topic.enable=false
>> offsets.topic.replication.factor=1
>> transaction.state.log.replication.factor=1
>> transaction.state.log.min.isr=1
>>
>> I have placed server.log from a broker at https://gist.github.com/deb
>> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
>>
>> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Seems to be related to KIP-113.
>>>
>>> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>>>
>>> If you can pastebin logs from broker, that should help.
>>>
>>> Thanks
>>>
>>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <subharaj.manna@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi
>>> >
>>> > I altered a topic like below in kafka 1.1.0
>>> >
>>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
>>> 127.0.0.1:2181
>>> > --alter --topic Topic3 --config min.insync.replicas=2
>>> >
>>> > But whenever I am trying to verify the reassignment it is showing the
>>> > below exception
>>> >
>>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
>>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
>>> /home/ubuntu/deploy/kafka/reassignment.json --verify
>>> >
>>> > Partitions reassignment failed due to Size of replicas list Vector(3,
>>> 0, 2) is different from size of log dirs list Vector(any) for partition
>>> Topic3-7
>>> > kafka.common.AdminCommandFailedException: Size of replicas list
>>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
>>> partition Topic3-7
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
>>> ReassignPartitionsCommand.scala:262)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
>>> ReassignPartitionsCommand.scala:251)
>>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:133
>>> 4)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
>>> sCommand.scala:251)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
>>> sCommand.scala:250)
>>> > at scala.collection.immutable.List.foreach(List.scala:392)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
>>> > at scala.collection.immutable.List.foreach(List.scala:392)
>>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig
>>> nmentData(ReassignPartitionsCommand.scala:249)
>>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>>> signPartitionsCommand.scala:90)
>>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>>> signPartitionsCommand.scala:84)
>>> > at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitio
>>> nsCommand.scala:58)
>>> > at kafka.admin.ReassignPartitionsCommand.main(ReassignPartition
>>> sCommand.scala)
>>> >
>>> >
>>> > My reassignment.json & server.properties is attached. Same thing used
>>> to
>>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
>>> > anything changed related to this in kafka 1.1.0 ?
>>> >
>>>
>>
>>
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
I am generating the reassignent.json like below
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --generate --topics-to-move-json-file
/home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
-1 > /home/ubuntu/deploy/kafka/reassignment.json"
Then I am doing the reassignment using the generated file
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --execute --reassignment-json-file
/home/ubuntu/deploy/kafka/reassignment.json
kafka-reassign-partitions.sh helps states
The JSON file with the partition reassignment configurationThe format to
> use is -
> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" is
> optional. When it is specified, its length must equal the length of the
> replicas list. The value in this list can be either "any" or the absolution
> path of the log directory on the broker. If absolute log directory path is
> specified, it is currently required that the replica has not already been
> created on that broker. The replica will then be created in the specified
> log directory on the broker later.
So it appears reassignment json that is generated by
kafka-reassign-partions.sh is creating an issue with logdirs. Is this some
issue in kafka-reassign-partitions.sh or some misconfiguration from my
side. ?
On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <su...@gmail.com>
wrote:
> Please find the server.properties from one of the broker.
>
> broker.id=0
> port=9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/var/lib/kafka/kafka-logs
> num.recovery.threads.per.data.dir=1
> log.retention.hours=36
> log.retention.bytes=1073741824
> log.segment.bytes=536870912
> log.retention.check.interval.ms=300000
> log.cleaner.enable=false
> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> message.max.bytes=15000000
> replica.fetch.max.bytes=15000000
> auto.create.topics.enable=true
> zookeeper.connection.timeout.ms=6000
> unclean.leader.election.enable=false
> delete.topic.enable=false
> offsets.topic.replication.factor=1
> transaction.state.log.replication.factor=1
> transaction.state.log.min.isr=1
>
> I have placed server.log from a broker at https://gist.github.com/
> debraj-manna/4b4bdae8a1c15c36b313a04f37e8776d
>
> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Seems to be related to KIP-113.
>>
>> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>>
>> If you can pastebin logs from broker, that should help.
>>
>> Thanks
>>
>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <su...@gmail.com>
>> wrote:
>>
>> > Hi
>> >
>> > I altered a topic like below in kafka 1.1.0
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
>> 127.0.0.1:2181
>> > --alter --topic Topic3 --config min.insync.replicas=2
>> >
>> > But whenever I am trying to verify the reassignment it is showing the
>> > below exception
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
>> 127.0.0.1:2181 --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json
>> --verify
>> >
>> > Partitions reassignment failed due to Size of replicas list Vector(3,
>> 0, 2) is different from size of log dirs list Vector(any) for partition
>> Topic3-7
>> > kafka.common.AdminCommandFailedException: Size of replicas list
>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
>> partition Topic3-7
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:262)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:251)
>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:251)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:250)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig
>> nmentData(ReassignPartitionsCommand.scala:249)
>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>> signPartitionsCommand.scala:90)
>> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(Reas
>> signPartitionsCommand.scala:84)
>> > at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitio
>> nsCommand.scala:58)
>> > at kafka.admin.ReassignPartitionsCommand.main(ReassignPartition
>> sCommand.scala)
>> >
>> >
>> > My reassignment.json & server.properties is attached. Same thing used to
>> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
>> > anything changed related to this in kafka 1.1.0 ?
>> >
>>
>
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
Please find the server.properties from one of the broker.
broker.id=0
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/lib/kafka/kafka-logs
num.recovery.threads.per.data.dir=1
log.retention.hours=36
log.retention.bytes=1073741824
log.segment.bytes=536870912
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
message.max.bytes=15000000
replica.fetch.max.bytes=15000000
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=6000
unclean.leader.election.enable=false
delete.topic.enable=false
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
I have placed server.log from a broker at
https://gist.github.com/debraj-manna/4b4bdae8a1c15c36b313a04f37e8776d
On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu <yu...@gmail.com> wrote:
> Seems to be related to KIP-113.
>
> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>
> If you can pastebin logs from broker, that should help.
>
> Thanks
>
> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <su...@gmail.com>
> wrote:
>
> > Hi
> >
> > I altered a topic like below in kafka 1.1.0
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181
> > --alter --topic Topic3 --config min.insync.replicas=2
> >
> > But whenever I am trying to verify the reassignment it is showing the
> > below exception
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json
> --verify
> >
> > Partitions reassignment failed due to Size of replicas list Vector(3, 0,
> 2) is different from size of log dirs list Vector(any) for partition
> Topic3-7
> > kafka.common.AdminCommandFailedException: Size of replicas list
> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
> partition Topic3-7
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> ReassignPartitionsCommand.scala:262)
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> ReassignPartitionsCommand.scala:251)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1$$anonfun$apply$4.apply(
> ReassignPartitionsCommand.scala:251)
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1$$anonfun$apply$4.apply(
> ReassignPartitionsCommand.scala:250)
> > at scala.collection.immutable.List.foreach(List.scala:392)
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.
> scala:250)
> > at kafka.admin.ReassignPartitionsCommand$$anonfun$
> parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.
> scala:249)
> > at scala.collection.immutable.List.foreach(List.scala:392)
> > at kafka.admin.ReassignPartitionsCommand$.
> parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249)
> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(
> ReassignPartitionsCommand.scala:90)
> > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(
> ReassignPartitionsCommand.scala:84)
> > at kafka.admin.ReassignPartitionsCommand$.main(
> ReassignPartitionsCommand.scala:58)
> > at kafka.admin.ReassignPartitionsCommand.main(
> ReassignPartitionsCommand.scala)
> >
> >
> > My reassignment.json & server.properties is attached. Same thing used to
> > work fine in kafka 0.10. Can someone let me what is going wrong? Is
> > anything changed related to this in kafka 1.1.0 ?
> >
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Ted Yu <yu...@gmail.com>.
Seems to be related to KIP-113.
server.properties didn't go thru. Do you mind pastebin'ing its content ?
If you can pastebin logs from broker, that should help.
Thanks
On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <su...@gmail.com>
wrote:
> Hi
>
> I altered a topic like below in kafka 1.1.0
>
> /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181
> --alter --topic Topic3 --config min.insync.replicas=2
>
> But whenever I am trying to verify the reassignment it is showing the
> below exception
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json --verify
>
> Partitions reassignment failed due to Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7
> kafka.common.AdminCommandFailedException: Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249)
> at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
> at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84)
> at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58)
> at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>
>
> My reassignment.json & server.properties is attached. Same thing used to
> work fine in kafka 0.10. Can someone let me what is going wrong? Is
> anything changed related to this in kafka 1.1.0 ?
>
Re: Partitions reassignment is failing in Kafka 1.1.0
Posted by Debraj Manna <su...@gmail.com>.
Hi
Anyone any thoughts?
On Fri 29 Jun, 2018, 11:07 PM Debraj Manna, <su...@gmail.com>
wrote:
> Hi
>
> I altered a topic like below in kafka 1.1.0
>
> /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181
> --alter --topic Topic3 --config min.insync.replicas=2
>
> But whenever I am trying to verify the reassignment it is showing the
> below exception
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json --verify
>
> Partitions reassignment failed due to Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7
> kafka.common.AdminCommandFailedException: Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
> at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249)
> at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
> at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84)
> at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58)
> at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>
>
> My reassignment.json & server.properties is attached. Same thing used to
> work fine in kafka 0.10. Can someone let me what is going wrong? Is
> anything changed related to this in kafka 1.1.0 ?
>