You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Bateman, Matt" <ma...@ebay.com> on 2012/02/16 23:48:19 UTC
Only one broker has partitions
Hi All,
I feel like this was discussed at some point but couldn't find it in the docs or the mailing list archives, hopefully this isn't redundant.
We are running 3 version 0.7 brokers using an ensemble of 3 ZooKeeper nodes. We use a ZooKeeper connect string in the producers and consumers. ZooKeeper is enabled on each broker. We use the default partitioner in the producer supplying it an integer. Each broker has a unique Id.
However, only one broker has any log files. The log from that machine shows it registering the topics in ZooKeeper. The other brokers don't have any log files.
Have a I missed a basic configuration setting or misunderstood something operational? The only slight mismatch I see is we run ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
Thanks,
Matt
RE: Only one broker has partitions
Posted by "Bateman, Matt" <ma...@ebay.com>.
For reference, and to close this out:
It turns out there is another way for this to happen. Given a stable Kafka cluster, if the producer that first sends messages for a new topic does not send "enough" messages, or it's session does not last "long enough", the bootstrapping process will not distribute the topic and partitions to all brokers.
I can't nail down the definition of "enough messages" or a session that is "long enough", but the point is that this can happen even with a stable set of brokers.
I've updated the JIRA issue.
Thanks,
Matt
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, February 17, 2012 3:03 PM
To: kafka-users@incubator.apache.org
Subject: Re: Only one broker has partitions
Thanks, Bateman.
Jun
On Fri, Feb 17, 2012 at 2:58 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi Jun,
>
> https://issues.apache.org/jira/browse/KAFKA-278
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 2:23 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> I see the problem. This is a bug in Kafka. Basically, for a new topic,
> we bootstrap using all existing brokers. However, if a topic already
> exists on some brokers, we never bootstrap again, which means new
> brokers will be ignored. Could you file a jira for that? For now, you
> have to manually create the topic on the new brokers (e.g., by sending
> some data to the new broker directly).
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <ma...@ebay.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the details. What sort of time frame is eventually?
> >
> > In the case of a topic that exists before nodes are added, will the
> > other nodes ever get that topic? We've been running all three for
> > about a week now it's still only on our 0 broker.
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Friday, February 17, 2012 11:19 AM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Matt,
> >
> > The behavior is a bit weird for a new topic. Initially, non of the
> > brokers has the topic. So, the producer simply assumes there is 1
> > partition per broker. Once a topic is created in a broker, the
> > broker registers #partitions for that topic in ZK. Then the producer
> > will pick up the new #partitions for that broker and refresh its
> > list of
> available partitions.
> > Eventually, the topic will be created on every broker and the
> > producer should see a total of 60 partitions.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com>
> > wrote:
> >
> > > We're using a single topic ("test-topic") in our QA setup.
> > > Originally we were using only one broker (id 0) in or testing
> environment.
> > > Recently we added two more (id 2 and 3) for a total of 3 in our
> > > cluster. ZooKeeper shows that the topic and partitions are
> > > registered by
> > broker 0 as expected.
> > > They do not show for brokers 2 and 3. I'm guessing that if a
> > > broker is not registered with ZooKeeper when a producer sends the
> > > first message for a topic, it will never process messages for that topic?
> > >
> > > As a test, I sent messages into the cluster for a new topic. The
> > > producer logs show
> > >
> > > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are
> > > registered under
> > > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test
> > > with available brokers in the cluster with default number of
> > > partitions = 1
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Adding following broker id,
> > > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> > >
> > > This seems logical, all brokers are mentioned. Then I see the
> > > following (each broker is configured to use 20 partitions):
> > >
> > > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > > (kafka.producer.ProducerPool)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of
> > > topics changed at /brokers/topics Updated topics -> [topic1,
> > > topic2, cluster-test, topic3, test-topic, topic4]
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> > topics:
> > > Set(topic1, topic3, topic2, test-topic, topic4)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated
> > > list of
> > > topics: Set(topic4, test-topic, topic2, topic3, cluster-test,
> > > topic1)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of
> > > newly registered topics: Set(cluster-test)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > > registered list of brokers for topic: cluster-test are Buffer(2)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener]
> > > Unregistered list of brokers for topic: cluster-test are
> > > TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of
> > > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0,
> > > 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12,
> > > 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7,
> > > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18,
> > > 2-19, 3-0)
> > > (kafka.producer.Producer)
> > >
> > > This seems strange, shouldn't each broker host each partition for
> > > the new "cluster-topic"?
> > >
> > > I then see the producer debug messages showing messages sent to
> > > broker
> 2.
> > >
> > > What am I missing here?
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, February 16, 2012 6:08 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Only one broker has partitions
> > >
> > > Try enabling debug level logging in Producer class. It will show
> > > which broker the partitioner selects.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt
> > > <ma...@ebay.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I feel like this was discussed at some point but couldn't find
> > > > it in the docs or the mailing list archives, hopefully this
> > > > isn't
> redundant.
> > > >
> > > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > > ZooKeeper nodes. We use a ZooKeeper connect string in the
> > > > producers
> > and consumers.
> > > > ZooKeeper is enabled on each broker. We use the default
> > > > partitioner in the producer supplying it an integer. Each broker
> > > > has
> a unique Id.
> > > >
> > > > However, only one broker has any log files. The log from that
> > > > machine shows it registering the topics in ZooKeeper. The other
> > > > brokers don't have any log files.
> > > >
> > > > Have a I missed a basic configuration setting or misunderstood
> > > > something operational? The only slight mismatch I see is we run
> > > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > > >
> > >
> >
>
Re: Only one broker has partitions
Posted by Jun Rao <ju...@gmail.com>.
Thanks, Bateman.
Jun
On Fri, Feb 17, 2012 at 2:58 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi Jun,
>
> https://issues.apache.org/jira/browse/KAFKA-278
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 2:23 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> I see the problem. This is a bug in Kafka. Basically, for a new topic, we
> bootstrap using all existing brokers. However, if a topic already exists on
> some brokers, we never bootstrap again, which means new brokers will be
> ignored. Could you file a jira for that? For now, you have to manually
> create the topic on the new brokers (e.g., by sending some data to the new
> broker directly).
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <ma...@ebay.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the details. What sort of time frame is eventually?
> >
> > In the case of a topic that exists before nodes are added, will the
> > other nodes ever get that topic? We've been running all three for
> > about a week now it's still only on our 0 broker.
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Friday, February 17, 2012 11:19 AM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Matt,
> >
> > The behavior is a bit weird for a new topic. Initially, non of the
> > brokers has the topic. So, the producer simply assumes there is 1
> > partition per broker. Once a topic is created in a broker, the broker
> > registers #partitions for that topic in ZK. Then the producer will
> > pick up the new #partitions for that broker and refresh its list of
> available partitions.
> > Eventually, the topic will be created on every broker and the producer
> > should see a total of 60 partitions.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com>
> > wrote:
> >
> > > We're using a single topic ("test-topic") in our QA setup.
> > > Originally we were using only one broker (id 0) in or testing
> environment.
> > > Recently we added two more (id 2 and 3) for a total of 3 in our
> > > cluster. ZooKeeper shows that the topic and partitions are
> > > registered by
> > broker 0 as expected.
> > > They do not show for brokers 2 and 3. I'm guessing that if a broker
> > > is not registered with ZooKeeper when a producer sends the first
> > > message for a topic, it will never process messages for that topic?
> > >
> > > As a test, I sent messages into the cluster for a new topic. The
> > > producer logs show
> > >
> > > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> > > under
> > > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test
> > > with available brokers in the cluster with default number of
> > > partitions = 1
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Adding following broker id,
> > > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo)
> > > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> > >
> > > This seems logical, all brokers are mentioned. Then I see the
> > > following (each broker is configured to use 20 partitions):
> > >
> > > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > > (kafka.producer.ProducerPool)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of
> > > topics changed at /brokers/topics Updated topics -> [topic1, topic2,
> > > cluster-test, topic3, test-topic, topic4]
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> > topics:
> > > Set(topic1, topic3, topic2, test-topic, topic4)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list
> > > of
> > > topics: Set(topic4, test-topic, topic2, topic3, cluster-test,
> > > topic1)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> > > registered topics: Set(cluster-test)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > > registered list of brokers for topic: cluster-test are Buffer(2)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> > > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of
> > > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1,
> > > 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13,
> > > 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > > partitions registered for topic: cluster-test
> > > (kafka.producer.Producer)
> > > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7,
> > > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18,
> > > 2-19, 3-0)
> > > (kafka.producer.Producer)
> > >
> > > This seems strange, shouldn't each broker host each partition for
> > > the new "cluster-topic"?
> > >
> > > I then see the producer debug messages showing messages sent to broker
> 2.
> > >
> > > What am I missing here?
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, February 16, 2012 6:08 PM
> > > To: kafka-users@incubator.apache.org
> > > Subject: Re: Only one broker has partitions
> > >
> > > Try enabling debug level logging in Producer class. It will show
> > > which broker the partitioner selects.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I feel like this was discussed at some point but couldn't find it
> > > > in the docs or the mailing list archives, hopefully this isn't
> redundant.
> > > >
> > > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > > ZooKeeper nodes. We use a ZooKeeper connect string in the
> > > > producers
> > and consumers.
> > > > ZooKeeper is enabled on each broker. We use the default
> > > > partitioner in the producer supplying it an integer. Each broker has
> a unique Id.
> > > >
> > > > However, only one broker has any log files. The log from that
> > > > machine shows it registering the topics in ZooKeeper. The other
> > > > brokers don't have any log files.
> > > >
> > > > Have a I missed a basic configuration setting or misunderstood
> > > > something operational? The only slight mismatch I see is we run
> > > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > > >
> > >
> >
>
RE: Only one broker has partitions
Posted by "Bateman, Matt" <ma...@ebay.com>.
Hi Jun,
https://issues.apache.org/jira/browse/KAFKA-278
Thanks,
Matt
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, February 17, 2012 2:23 PM
To: kafka-users@incubator.apache.org
Subject: Re: Only one broker has partitions
I see the problem. This is a bug in Kafka. Basically, for a new topic, we bootstrap using all existing brokers. However, if a topic already exists on some brokers, we never bootstrap again, which means new brokers will be ignored. Could you file a jira for that? For now, you have to manually create the topic on the new brokers (e.g., by sending some data to the new broker directly).
Thanks,
Jun
On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi Jun,
>
> Thanks for the details. What sort of time frame is eventually?
>
> In the case of a topic that exists before nodes are added, will the
> other nodes ever get that topic? We've been running all three for
> about a week now it's still only on our 0 broker.
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 11:19 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Matt,
>
> The behavior is a bit weird for a new topic. Initially, non of the
> brokers has the topic. So, the producer simply assumes there is 1
> partition per broker. Once a topic is created in a broker, the broker
> registers #partitions for that topic in ZK. Then the producer will
> pick up the new #partitions for that broker and refresh its list of available partitions.
> Eventually, the topic will be created on every broker and the producer
> should see a total of 60 partitions.
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com>
> wrote:
>
> > We're using a single topic ("test-topic") in our QA setup.
> > Originally we were using only one broker (id 0) in or testing environment.
> > Recently we added two more (id 2 and 3) for a total of 3 in our
> > cluster. ZooKeeper shows that the topic and partitions are
> > registered by
> broker 0 as expected.
> > They do not show for brokers 2 and 3. I'm guessing that if a broker
> > is not registered with ZooKeeper when a producer sends the first
> > message for a topic, it will never process messages for that topic?
> >
> > As a test, I sent messages into the cluster for a new topic. The
> > producer logs show
> >
> > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> > under
> > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test
> > with available brokers in the cluster with default number of
> > partitions = 1
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Adding following broker id,
> > partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> >
> > This seems logical, all brokers are mentioned. Then I see the
> > following (each broker is configured to use 20 partitions):
> >
> > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > (kafka.producer.ProducerPool)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of
> > topics changed at /brokers/topics Updated topics -> [topic1, topic2,
> > cluster-test, topic3, test-topic, topic4]
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> topics:
> > Set(topic1, topic3, topic2, test-topic, topic4)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list
> > of
> > topics: Set(topic4, test-topic, topic2, topic3, cluster-test,
> > topic1)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> > registered topics: Set(cluster-test)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > registered list of brokers for topic: cluster-test are Buffer(2)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of
> > broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1,
> > 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13,
> > 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7,
> > 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18,
> > 2-19, 3-0)
> > (kafka.producer.Producer)
> >
> > This seems strange, shouldn't each broker host each partition for
> > the new "cluster-topic"?
> >
> > I then see the producer debug messages showing messages sent to broker 2.
> >
> > What am I missing here?
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 6:08 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Try enabling debug level logging in Producer class. It will show
> > which broker the partitioner selects.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I feel like this was discussed at some point but couldn't find it
> > > in the docs or the mailing list archives, hopefully this isn't redundant.
> > >
> > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > ZooKeeper nodes. We use a ZooKeeper connect string in the
> > > producers
> and consumers.
> > > ZooKeeper is enabled on each broker. We use the default
> > > partitioner in the producer supplying it an integer. Each broker has a unique Id.
> > >
> > > However, only one broker has any log files. The log from that
> > > machine shows it registering the topics in ZooKeeper. The other
> > > brokers don't have any log files.
> > >
> > > Have a I missed a basic configuration setting or misunderstood
> > > something operational? The only slight mismatch I see is we run
> > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> >
>
Re: Only one broker has partitions
Posted by Jun Rao <ju...@gmail.com>.
I see the problem. This is a bug in Kafka. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. Could you file a jira for that? For now, you have to manually
create the topic on the new brokers (e.g., by sending some data to the new
broker directly).
Thanks,
Jun
On Fri, Feb 17, 2012 at 1:40 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi Jun,
>
> Thanks for the details. What sort of time frame is eventually?
>
> In the case of a topic that exists before nodes are added, will the other
> nodes ever get that topic? We've been running all three for about a week
> now it's still only on our 0 broker.
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 17, 2012 11:19 AM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Matt,
>
> The behavior is a bit weird for a new topic. Initially, non of the brokers
> has the topic. So, the producer simply assumes there is 1 partition per
> broker. Once a topic is created in a broker, the broker registers
> #partitions for that topic in ZK. Then the producer will pick up the new
> #partitions for that broker and refresh its list of available partitions.
> Eventually, the topic will be created on every broker and the producer
> should see a total of 60 partitions.
>
> Thanks,
>
> Jun
>
>
> On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com>
> wrote:
>
> > We're using a single topic ("test-topic") in our QA setup. Originally
> > we were using only one broker (id 0) in or testing environment.
> > Recently we added two more (id 2 and 3) for a total of 3 in our
> > cluster. ZooKeeper shows that the topic and partitions are registered by
> broker 0 as expected.
> > They do not show for brokers 2 and 3. I'm guessing that if a broker is
> > not registered with ZooKeeper when a producer sends the first message
> > for a topic, it will never process messages for that topic?
> >
> > As a test, I sent messages into the cluster for a new topic. The
> > producer logs show
> >
> > [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> > under
> > topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with
> > available brokers in the cluster with default number of partitions = 1
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition
> > id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo)
> > [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
> >
> > This seems logical, all brokers are mentioned. Then I see the
> > following (each broker is configured to use 20 partitions):
> >
> > [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> > (kafka.producer.ProducerPool)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics
> > changed at /brokers/topics Updated topics -> [topic1, topic2,
> > cluster-test, topic3, test-topic, topic4]
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of
> topics:
> > Set(topic1, topic3, topic2, test-topic, topic4)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of
> > topics: Set(topic4, test-topic, topic2, topic3, cluster-test, topic1)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> > registered topics: Set(cluster-test)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> > registered list of brokers for topic: cluster-test are Buffer(2)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> > list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker
> > partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2,
> > 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15,
> > 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> > [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> > partitions registered for topic: cluster-test
> > (kafka.producer.Producer)
> > [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> > cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8,
> > 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> > (kafka.producer.Producer)
> >
> > This seems strange, shouldn't each broker host each partition for the
> > new "cluster-topic"?
> >
> > I then see the producer debug messages showing messages sent to broker 2.
> >
> > What am I missing here?
> >
> > Thanks,
> >
> > Matt
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, February 16, 2012 6:08 PM
> > To: kafka-users@incubator.apache.org
> > Subject: Re: Only one broker has partitions
> >
> > Try enabling debug level logging in Producer class. It will show which
> > broker the partitioner selects.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I feel like this was discussed at some point but couldn't find it in
> > > the docs or the mailing list archives, hopefully this isn't redundant.
> > >
> > > We are running 3 version 0.7 brokers using an ensemble of 3
> > > ZooKeeper nodes. We use a ZooKeeper connect string in the producers
> and consumers.
> > > ZooKeeper is enabled on each broker. We use the default partitioner
> > > in the producer supplying it an integer. Each broker has a unique Id.
> > >
> > > However, only one broker has any log files. The log from that
> > > machine shows it registering the topics in ZooKeeper. The other
> > > brokers don't have any log files.
> > >
> > > Have a I missed a basic configuration setting or misunderstood
> > > something operational? The only slight mismatch I see is we run
> > > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> >
>
RE: Only one broker has partitions
Posted by "Bateman, Matt" <ma...@ebay.com>.
Hi Jun,
Thanks for the details. What sort of time frame is eventually?
In the case of a topic that exists before nodes are added, will the other nodes ever get that topic? We've been running all three for about a week now it's still only on our 0 broker.
Thanks,
Matt
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, February 17, 2012 11:19 AM
To: kafka-users@incubator.apache.org
Subject: Re: Only one broker has partitions
Matt,
The behavior is a bit weird for a new topic. Initially, non of the brokers has the topic. So, the producer simply assumes there is 1 partition per broker. Once a topic is created in a broker, the broker registers #partitions for that topic in ZK. Then the producer will pick up the new #partitions for that broker and refresh its list of available partitions.
Eventually, the topic will be created on every broker and the producer should see a total of 60 partitions.
Thanks,
Jun
On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com> wrote:
> We're using a single topic ("test-topic") in our QA setup. Originally
> we were using only one broker (id 0) in or testing environment.
> Recently we added two more (id 2 and 3) for a total of 3 in our
> cluster. ZooKeeper shows that the topic and partitions are registered by broker 0 as expected.
> They do not show for brokers 2 and 3. I'm guessing that if a broker is
> not registered with ZooKeeper when a producer sends the first message
> for a topic, it will never process messages for that topic?
>
> As a test, I sent messages into the cluster for a new topic. The
> producer logs show
>
> [2012-02-17 10:31:12,698] DEBUG Getting the number of broker
> partitions registered for topic: cluster-test
> (kafka.producer.Producer)
> [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered
> under
> topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with
> available brokers in the cluster with default number of partitions = 1
> (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition
> id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
>
> This seems logical, all brokers are mentioned. Then I see the
> following (each broker is configured to use 20 partitions):
>
> [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> (kafka.producer.ProducerPool)
> [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics
> changed at /brokers/topics Updated topics -> [topic1, topic2,
> cluster-test, topic3, test-topic, topic4]
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of topics:
> Set(topic1, topic3, topic2, test-topic, topic4)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of
> topics: Set(topic4, test-topic, topic2, topic3, cluster-test, topic1)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> registered topics: Set(cluster-test)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> registered list of brokers for topic: cluster-test are Buffer(2)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered
> list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker
> partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2,
> 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15,
> 2-16, 2-17, 2-18, 2-19, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:13,841] DEBUG Getting the number of broker
> partitions registered for topic: cluster-test
> (kafka.producer.Producer)
> [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8,
> 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> (kafka.producer.Producer)
>
> This seems strange, shouldn't each broker host each partition for the
> new "cluster-topic"?
>
> I then see the producer debug messages showing messages sent to broker 2.
>
> What am I missing here?
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, February 16, 2012 6:08 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Try enabling debug level logging in Producer class. It will show which
> broker the partitioner selects.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com> wrote:
>
> > Hi All,
> >
> > I feel like this was discussed at some point but couldn't find it in
> > the docs or the mailing list archives, hopefully this isn't redundant.
> >
> > We are running 3 version 0.7 brokers using an ensemble of 3
> > ZooKeeper nodes. We use a ZooKeeper connect string in the producers and consumers.
> > ZooKeeper is enabled on each broker. We use the default partitioner
> > in the producer supplying it an integer. Each broker has a unique Id.
> >
> > However, only one broker has any log files. The log from that
> > machine shows it registering the topics in ZooKeeper. The other
> > brokers don't have any log files.
> >
> > Have a I missed a basic configuration setting or misunderstood
> > something operational? The only slight mismatch I see is we run
> > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> >
> > Thanks,
> >
> > Matt
> >
>
Re: Only one broker has partitions
Posted by Jun Rao <ju...@gmail.com>.
Matt,
The behavior is a bit weird for a new topic. Initially, non of the brokers
has the topic. So, the producer simply assumes there is 1 partition per
broker. Once a topic is created in a broker, the broker registers
#partitions for that topic in ZK. Then the producer will pick up the new
#partitions for that broker and refresh its list of available partitions.
Eventually, the topic will be created on every broker and the producer
should see a total of 60 partitions.
Thanks,
Jun
On Fri, Feb 17, 2012 at 11:05 AM, Bateman, Matt <ma...@ebay.com> wrote:
> We're using a single topic ("test-topic") in our QA setup. Originally we
> were using only one broker (id 0) in or testing environment. Recently we
> added two more (id 2 and 3) for a total of 3 in our cluster. ZooKeeper
> shows that the topic and partitions are registered by broker 0 as expected.
> They do not show for brokers 2 and 3. I'm guessing that if a broker is not
> registered with ZooKeeper when a producer sends the first message for a
> topic, it will never process messages for that topic?
>
> As a test, I sent messages into the cluster for a new topic. The producer
> logs show
>
> [2012-02-17 10:31:12,698] DEBUG Getting the number of broker partitions
> registered for topic: cluster-test (kafka.producer.Producer)
> [2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered under
> topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with
> available brokers in the cluster with default number of partitions = 1
> (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition id
> for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo)
> [2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic:
> cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
>
> This seems logical, all brokers are mentioned. Then I see the following
> (each broker is configured to use 20 partitions):
>
> [2012-02-17 10:31:12,840] DEBUG Sending message to broker 2
> (kafka.producer.ProducerPool)
> [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics
> changed at /brokers/topics Updated topics -> [topic1, topic2, cluster-test,
> topic3, test-topic, topic4]
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of topics:
> Set(topic1, topic3, topic2, test-topic, topic4)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of
> topics: Set(topic4, test-topic, topic2, topic3, cluster-test, topic1)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly
> registered topics: Set(cluster-test)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently
> registered list of brokers for topic: cluster-test are Buffer(2)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered list
> of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker
> partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2, 2-3,
> 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16,
> 2-17, 2-18, 2-19, 3-0)
> (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
> [2012-02-17 10:31:13,841] DEBUG Getting the number of broker partitions
> registered for topic: cluster-test (kafka.producer.Producer)
> [2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic:
> cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9,
> 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0)
> (kafka.producer.Producer)
>
> This seems strange, shouldn't each broker host each partition for the new
> "cluster-topic"?
>
> I then see the producer debug messages showing messages sent to broker 2.
>
> What am I missing here?
>
> Thanks,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, February 16, 2012 6:08 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Only one broker has partitions
>
> Try enabling debug level logging in Producer class. It will show which
> broker the partitioner selects.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com> wrote:
>
> > Hi All,
> >
> > I feel like this was discussed at some point but couldn't find it in
> > the docs or the mailing list archives, hopefully this isn't redundant.
> >
> > We are running 3 version 0.7 brokers using an ensemble of 3 ZooKeeper
> > nodes. We use a ZooKeeper connect string in the producers and consumers.
> > ZooKeeper is enabled on each broker. We use the default partitioner in
> > the producer supplying it an integer. Each broker has a unique Id.
> >
> > However, only one broker has any log files. The log from that machine
> > shows it registering the topics in ZooKeeper. The other brokers don't
> > have any log files.
> >
> > Have a I missed a basic configuration setting or misunderstood
> > something operational? The only slight mismatch I see is we run
> > ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
> >
> > Thanks,
> >
> > Matt
> >
>
RE: Only one broker has partitions
Posted by "Bateman, Matt" <ma...@ebay.com>.
We're using a single topic ("test-topic") in our QA setup. Originally we were using only one broker (id 0) in or testing environment. Recently we added two more (id 2 and 3) for a total of 3 in our cluster. ZooKeeper shows that the topic and partitions are registered by broker 0 as expected. They do not show for brokers 2 and 3. I'm guessing that if a broker is not registered with ZooKeeper when a producer sends the first message for a topic, it will never process messages for that topic?
As a test, I sent messages into the cluster for a new topic. The producer logs show
[2012-02-17 10:31:12,698] DEBUG Getting the number of broker partitions registered for topic: cluster-test (kafka.producer.Producer)
[2012-02-17 10:31:12,698] DEBUG Currently, no brokers are registered under topic: cluster-test (kafka.producer.ZKBrokerPartitionInfo)
[2012-02-17 10:31:12,698] DEBUG Bootstrapping topic: cluster-test with available brokers in the cluster with default number of partitions = 1 (kafka.producer.ZKBrokerPartitionInfo)
[2012-02-17 10:31:12,705] DEBUG Adding following broker id, partition id for NEW topic: cluster-test=TreeSet(0-0, 2-0, 3-0) (kafka.producer.ZKBrokerPartitionInfo)
[2012-02-17 10:31:12,705] DEBUG Broker partitions registered for topic: cluster-test = List(0-0, 2-0, 3-0) (kafka.producer.Producer)
This seems logical, all brokers are mentioned. Then I see the following (each broker is configured to use 20 partitions):
[2012-02-17 10:31:12,840] DEBUG Sending message to broker 2 (kafka.producer.ProducerPool)
[2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] List of topics changed at /brokers/topics Updated topics -> [topic1, topic2, cluster-test, topic3, test-topic, topic4] (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,953] DEBUG [BrokerTopicsListener] Old list of topics: Set(topic1, topic3, topic2, test-topic, topic4) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] Updated list of topics: Set(topic4, test-topic, topic2, topic3, cluster-test, topic1) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,963] DEBUG [BrokerTopicsListener] List of newly registered topics: Set(cluster-test) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Currently registered list of brokers for topic: cluster-test are Buffer(2) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,978] DEBUG [BrokerTopicsListener] Unregistered list of brokers for topic: cluster-test are TreeSet(0-0, 2-0, 3-0) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:12,980] DEBUG [BrokerTopicsListener] List of broker partitions for topic: cluster-test are TreeSet(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0) (kafka.producer.ZKBrokerPartitionInfo$BrokerTopicsListener)
[2012-02-17 10:31:13,841] DEBUG Getting the number of broker partitions registered for topic: cluster-test (kafka.producer.Producer)
[2012-02-17 10:31:13,841] DEBUG Broker partitions registered for topic: cluster-test = List(0-0, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9, 2-10, 2-11, 2-12, 2-13, 2-14, 2-15, 2-16, 2-17, 2-18, 2-19, 3-0) (kafka.producer.Producer)
This seems strange, shouldn't each broker host each partition for the new "cluster-topic"?
I then see the producer debug messages showing messages sent to broker 2.
What am I missing here?
Thanks,
Matt
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Thursday, February 16, 2012 6:08 PM
To: kafka-users@incubator.apache.org
Subject: Re: Only one broker has partitions
Try enabling debug level logging in Producer class. It will show which broker the partitioner selects.
Thanks,
Jun
On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi All,
>
> I feel like this was discussed at some point but couldn't find it in
> the docs or the mailing list archives, hopefully this isn't redundant.
>
> We are running 3 version 0.7 brokers using an ensemble of 3 ZooKeeper
> nodes. We use a ZooKeeper connect string in the producers and consumers.
> ZooKeeper is enabled on each broker. We use the default partitioner in
> the producer supplying it an integer. Each broker has a unique Id.
>
> However, only one broker has any log files. The log from that machine
> shows it registering the topics in ZooKeeper. The other brokers don't
> have any log files.
>
> Have a I missed a basic configuration setting or misunderstood
> something operational? The only slight mismatch I see is we run
> ZooKeeper 3.3.4 but it seems like 3.3.3 is being used by 0.7.
>
> Thanks,
>
> Matt
>
Re: Only one broker has partitions
Posted by Jun Rao <ju...@gmail.com>.
Try enabling debug level logging in Producer class. It will show which
broker the partitioner selects.
Thanks,
Jun
On Thu, Feb 16, 2012 at 2:48 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Hi All,
>
> I feel like this was discussed at some point but couldn't find it in the
> docs or the mailing list archives, hopefully this isn't redundant.
>
> We are running 3 version 0.7 brokers using an ensemble of 3 ZooKeeper
> nodes. We use a ZooKeeper connect string in the producers and consumers.
> ZooKeeper is enabled on each broker. We use the default partitioner in the
> producer supplying it an integer. Each broker has a unique Id.
>
> However, only one broker has any log files. The log from that machine
> shows it registering the topics in ZooKeeper. The other brokers don't have
> any log files.
>
> Have a I missed a basic configuration setting or misunderstood something
> operational? The only slight mismatch I see is we run ZooKeeper 3.3.4 but
> it seems like 3.3.3 is being used by 0.7.
>
> Thanks,
>
> Matt
>