You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rajiv Kurian <ra...@signalfuse.com> on 2014/11/22 07:14:52 UTC
Create topic creates extra partitions
I have used the kafka-topics.sh script to create a topic with a number of
partitions.
bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
--partitions 1024 --replication-factor 3
I was expecting this topic to be created with 1024 total partitions across
3 of my kafka brokers. Instead it seems like it ended up creating 1024 * 3
= 3072 partitions. Is this a bug?
I got the number of partitions running the code from the SimpleConsumer
example at
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
specifically the findLeader(...) function. I just count the number of
partitionsMetadata objects received. That seems to give me the number
3072. The relevant snippet looks like:
List<TopicMetadata> metaData = resp.topicsMetadata();
for (TopicMetadata item : metaData) {
for (PartitionMetadata part : item.partitionsMetadata()) {
numPartitions++;
}
}
Given I am running the above snippet for every broker, should I not expect
to see the numPartitions as 1024? I am assuming that
item.partitionsMetadata() only returns PartitionMetadata for the partitions
this broker is responsible for? If not then how do I exactly figure out
what partitions this broker is the leader for?
I am using the SimpleConsumer and my logic is getting completely muddled
because of this problem.
Also is there a way through the kafka console scripts to find out the
number of partitions for a given topic?
Thanks,
Rajiv
Re: Create topic creates extra partitions
Posted by Rajiv Kurian <ra...@signalfuse.com>.
Got it. This was indeed my problem. Thanks so much!
On Fri, Nov 21, 2014 at 10:39 PM, Gwen Shapira <gs...@cloudera.com>
wrote:
> I think the issue is that you are:
> " running the above snippet for every broker ... I am assuming that
> item.partitionsMetadata() only returns PartitionMetadata for the partitions
> this broker is responsible for "
>
> This is inaccurate. Each broker will check ZooKeeper for PartitionMetadata
> and return information about all partitions.
> You only need to check one broker. You can try other brokers if the first
> broker returns an error.
>
> Gwen
>
> On Fri, Nov 21, 2014 at 10:14 PM, Rajiv Kurian <ra...@signalfuse.com>
> wrote:
>
> > I have used the kafka-topics.sh script to create a topic with a number of
> > partitions.
> >
> > bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
> > --partitions 1024 --replication-factor 3
> >
> > I was expecting this topic to be created with 1024 total partitions
> across
> > 3 of my kafka brokers. Instead it seems like it ended up creating 1024 *
> 3
> > = 3072 partitions. Is this a bug?
> >
> > I got the number of partitions running the code from the SimpleConsumer
> > example at
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > specifically the findLeader(...) function. I just count the number of
> > partitionsMetadata objects received. That seems to give me the number
> > 3072. The relevant snippet looks like:
> >
> > List<TopicMetadata> metaData = resp.topicsMetadata();
> > for (TopicMetadata item : metaData) {
> >
> > for (PartitionMetadata part : item.partitionsMetadata()) {
> >
> > numPartitions++;
> >
> > }
> >
> > }
> >
> >
> > Given I am running the above snippet for every broker, should I not
> expect
> > to see the numPartitions as 1024? I am assuming that
> > item.partitionsMetadata() only returns PartitionMetadata for the
> partitions
> > this broker is responsible for? If not then how do I exactly figure out
> > what partitions this broker is the leader for?
> >
> > I am using the SimpleConsumer and my logic is getting completely muddled
> > because of this problem.
> >
> > Also is there a way through the kafka console scripts to find out the
> > number of partitions for a given topic?
> >
> >
> > Thanks,
> > Rajiv
> >
>
Re: Create topic creates extra partitions
Posted by Gwen Shapira <gs...@cloudera.com>.
I think the issue is that you are:
" running the above snippet for every broker ... I am assuming that
item.partitionsMetadata() only returns PartitionMetadata for the partitions
this broker is responsible for "
This is inaccurate. Each broker will check ZooKeeper for PartitionMetadata
and return information about all partitions.
You only need to check one broker. You can try other brokers if the first
broker returns an error.
Gwen
On Fri, Nov 21, 2014 at 10:14 PM, Rajiv Kurian <ra...@signalfuse.com> wrote:
> I have used the kafka-topics.sh script to create a topic with a number of
> partitions.
>
> bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
> --partitions 1024 --replication-factor 3
>
> I was expecting this topic to be created with 1024 total partitions across
> 3 of my kafka brokers. Instead it seems like it ended up creating 1024 * 3
> = 3072 partitions. Is this a bug?
>
> I got the number of partitions running the code from the SimpleConsumer
> example at
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> specifically the findLeader(...) function. I just count the number of
> partitionsMetadata objects received. That seems to give me the number
> 3072. The relevant snippet looks like:
>
> List<TopicMetadata> metaData = resp.topicsMetadata();
> for (TopicMetadata item : metaData) {
>
> for (PartitionMetadata part : item.partitionsMetadata()) {
>
> numPartitions++;
>
> }
>
> }
>
>
> Given I am running the above snippet for every broker, should I not expect
> to see the numPartitions as 1024? I am assuming that
> item.partitionsMetadata() only returns PartitionMetadata for the partitions
> this broker is responsible for? If not then how do I exactly figure out
> what partitions this broker is the leader for?
>
> I am using the SimpleConsumer and my logic is getting completely muddled
> because of this problem.
>
> Also is there a way through the kafka console scripts to find out the
> number of partitions for a given topic?
>
>
> Thanks,
> Rajiv
>