You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tom Brown <to...@gmail.com> on 2013/04/12 21:16:33 UTC
Not balancing across multiple brokers
We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
topic has 2 partitions per server. We have a two processes that that write
to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
The problem is that the first process only writes to the first broker. The
second process (using the exact same code to perform the write)
successfully writes to both brokers.
How can I identify the cause of the imbalance in the first process? How
does the Producer decide which broker is the recipient of each message?
Thanks!
--Tom
Re: Not balancing across multiple brokers
Posted by Neha Narkhede <ne...@gmail.com>.
Tom,
I think you maybe hitting
https://issues.apache.org/jira/browse/KAFKA-278. A workaround is to
create the topic-partition directories on the newly added broker and
restart the broker. After that the zookeeper based producer will load
balance correctly.
Thanks,
Neha
On Fri, Apr 12, 2013 at 4:02 PM, Tom Brown <to...@gmail.com> wrote:
> In the producer config, we use the zk connect string:
> zk001,zk002,zk003/kafka.
>
> Both brokers have registered themselves with zookeeper. Because only the
> first broker has ever received any writes, only the first broker is
> registered for the topic in question.
>
> --Tom
>
>
> On Fri, Apr 12, 2013 at 3:32 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> Do you use a VIP or zookeeper for producer side load balancing ? In
>> other words, what are the values you override for "broker.list" and
>> "zk.connect" in the producer config ?
>>
>> Thanks,
>> Neha
>>
>> On Fri, Apr 12, 2013 at 12:16 PM, Tom Brown <to...@gmail.com> wrote:
>> > We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
>> > topic has 2 partitions per server. We have a two processes that that
>> write
>> > to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
>> >
>> > The problem is that the first process only writes to the first broker.
>> The
>> > second process (using the exact same code to perform the write)
>> > successfully writes to both brokers.
>> >
>> > How can I identify the cause of the imbalance in the first process? How
>> > does the Producer decide which broker is the recipient of each message?
>> >
>> > Thanks!
>> >
>> > --Tom
>>
Re: Not balancing across multiple brokers
Posted by Tom Brown <to...@gmail.com>.
In the producer config, we use the zk connect string:
zk001,zk002,zk003/kafka.
Both brokers have registered themselves with zookeeper. Because only the
first broker has ever received any writes, only the first broker is
registered for the topic in question.
--Tom
On Fri, Apr 12, 2013 at 3:32 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Do you use a VIP or zookeeper for producer side load balancing ? In
> other words, what are the values you override for "broker.list" and
> "zk.connect" in the producer config ?
>
> Thanks,
> Neha
>
> On Fri, Apr 12, 2013 at 12:16 PM, Tom Brown <to...@gmail.com> wrote:
> > We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
> > topic has 2 partitions per server. We have a two processes that that
> write
> > to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
> >
> > The problem is that the first process only writes to the first broker.
> The
> > second process (using the exact same code to perform the write)
> > successfully writes to both brokers.
> >
> > How can I identify the cause of the imbalance in the first process? How
> > does the Producer decide which broker is the recipient of each message?
> >
> > Thanks!
> >
> > --Tom
>
Re: Not balancing across multiple brokers
Posted by Neha Narkhede <ne...@gmail.com>.
Do you use a VIP or zookeeper for producer side load balancing ? In
other words, what are the values you override for "broker.list" and
"zk.connect" in the producer config ?
Thanks,
Neha
On Fri, Apr 12, 2013 at 12:16 PM, Tom Brown <to...@gmail.com> wrote:
> We have recently setup a new kafka (0.7.1) cluster with two brokers. Each
> topic has 2 partitions per server. We have a two processes that that write
> to the cluster using the class: kafka.javaapi.producer.Producer.Producer.
>
> The problem is that the first process only writes to the first broker. The
> second process (using the exact same code to perform the write)
> successfully writes to both brokers.
>
> How can I identify the cause of the imbalance in the first process? How
> does the Producer decide which broker is the recipient of each message?
>
> Thanks!
>
> --Tom