Kafka Cluster issue..

[Yogesh G <jo...@gmail.com>]

Hi

I am using Kafka 0.7.2...the setup have two Kafka brokers running on two
different machines both pointing to a single Zookeper server which is on
one of the machines hosting Kafka....Now if I run the Producer, either one
provided in getting started guide or custom one, although it states that
Creating async producer for broker 1 and broker 2....but the entire load is
being processed by single Kafka broker...the second instance have no impact
and nor does it creates any logs....I have num.partition=6 for both
brokers....

I would like to understand that wouldn't Zookeeper discover the two brokers
and distribute the load.....am I missing anything here....

Also for supporting a load of 2 million messages (say 200KB each) what is
kind of topology and configuration is recommended...

Thanks,

Re: Kafka Cluster issue..

[Jun Rao <ju...@gmail.com>]

There is a known bug in Kafka 0.7.x. Basically, for a new topic, we
bootstrap using all existing brokers. However, if a topic already exists on
some brokers, we never bootstrap again, which means new brokers will be
ignored. For now, you have to manually create the topic on the new brokers
(e.g., by sending some data to the new broker directly).

See the discussion in

http://apache.markmail.org/thread/c7tdalfketpusqkg<https://email.corp.linkedin.com/owa/redir.aspx?C=vZZC9djRrEur-lWAgOQM7PtQdxMJ3M8IQEdO3N7Snpy8CUDf2IYTRBuzdFKt-90dlUl7k3E8H8k.&URL=http%3a%2f%2fapache.markmail.org%2fthread%2fc7tdalfketpusqkg>

As for configs, you can take a look at
https://cwiki.apache.org/KAFKA/operations.html

Thanks,

Jun

On Fri, Feb 8, 2013 at 7:28 AM, Yogesh G <jo...@gmail.com> wrote:

> Hi
>
> I am using Kafka 0.7.2...the setup have two Kafka brokers running on two
> different machines both pointing to a single Zookeper server which is on
> one of the machines hosting Kafka....Now if I run the Producer, either one
> provided in getting started guide or custom one, although it states that
> Creating async producer for broker 1 and broker 2....but the entire load is
> being processed by single Kafka broker...the second instance have no impact
> and nor does it creates any logs....I have num.partition=6 for both
> brokers....
>
> I would like to understand that wouldn't Zookeeper discover the two brokers
> and distribute the load.....am I missing anything here....
>
> Also for supporting a load of 2 million messages (say 200KB each) what is
> kind of topology and configuration is recommended...
>
> Thanks,
>