You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Haithem Jarraya <a-...@expedia.com> on 2013/08/02 17:23:56 UTC

[0.8] Can Kafka producer discover nodes from Zookeeper?

Hi, 

I am new to Kafka, and I am a bit confused for this particular use case.
We are running a global Zookeeper cluster and I was wondering how I can have my Kafka producer discover Kafka hostnames?
I saw that in the server.properties I can specify host.name and at that point I will be able to see the list of hostnames in zookeeper /brokers/ids/ so I can have my producer connect to zookeeper and resolve Kafka hostnames.
Are there any other solutions?
Also, Can we have two separate Kafka Clusters using the same Zookeeper cluster? Just to make sure that we do not collide with other teams. I guess not given that the zookeeper path is hard coded. Am I wrong?

Many Thanks,

Haithem

Re: [0.8] Can Kafka producer discover nodes from Zookeeper?

Posted by Haithem Jarraya <a-...@expedia.com>.
Thanks a lot Chris, you saved us a lot of time with this.

-Haithem
On 2 Aug 2013, at 16:58, Chris Hogue <cs...@gmail.com> wrote:

> A couple of notes on name-spacing since there are a couple of gotchas when
> trying to configure it:
> 
> It's specified via the zookeeper.connect property described here:
> 
> http://kafka.apache.org/08/configuration.html
> 
> If you have multiple servers in the zk cluster rather than a standalone
> node the path is only specified once at the end, not on each node. So it
> would look something like:
> 
> zookeeper.connect=myzk1.example.org:2181,myzk2.example.org:2181,
> myzk3.example.org:2181/foo/bar
> 
> Also, the /foo/bar path must exist in the zk cluster before starting the
> brokers or they will fail on startup. There is an error message but it's
> not totally clear what the issue is. You can create the path via the
> zookeeper shell if nothing else.
> 
> Hope that helps.
> 
> -Chris
> 
> 
> 
> On Fri, Aug 2, 2013 at 9:44 AM, Jun Rao <ju...@gmail.com> wrote:
> 
>> Kafka producers identify brokers registered in ZK through a getMetadata
>> request to one of the brokers, not through ZK directly.
>> 
>> Two Kafka clusters can share the same ZK cluster, as long as they use
>> different ZK namespace.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> 
>> On Fri, Aug 2, 2013 at 8:23 AM, Haithem Jarraya <a-hjarraya@expedia.com
>>> wrote:
>> 
>>> Hi,
>>> 
>>> I am new to Kafka, and I am a bit confused for this particular use case.
>>> We are running a global Zookeeper cluster and I was wondering how I can
>>> have my Kafka producer discover Kafka hostnames?
>>> I saw that in the server.properties I can specify host.name and at that
>>> point I will be able to see the list of hostnames in zookeeper
>>> /brokers/ids/ so I can have my producer connect to zookeeper and resolve
>>> Kafka hostnames.
>>> Are there any other solutions?
>>> Also, Can we have two separate Kafka Clusters using the same Zookeeper
>>> cluster? Just to make sure that we do not collide with other teams. I
>> guess
>>> not given that the zookeeper path is hard coded. Am I wrong?
>>> 
>>> Many Thanks,
>>> 
>>> Haithem
>> 


Re: [0.8] Can Kafka producer discover nodes from Zookeeper?

Posted by Chris Hogue <cs...@gmail.com>.
A couple of notes on name-spacing since there are a couple of gotchas when
trying to configure it:

It's specified via the zookeeper.connect property described here:

http://kafka.apache.org/08/configuration.html

If you have multiple servers in the zk cluster rather than a standalone
node the path is only specified once at the end, not on each node. So it
would look something like:

zookeeper.connect=myzk1.example.org:2181,myzk2.example.org:2181,
myzk3.example.org:2181/foo/bar

Also, the /foo/bar path must exist in the zk cluster before starting the
brokers or they will fail on startup. There is an error message but it's
not totally clear what the issue is. You can create the path via the
zookeeper shell if nothing else.

Hope that helps.

-Chris



On Fri, Aug 2, 2013 at 9:44 AM, Jun Rao <ju...@gmail.com> wrote:

> Kafka producers identify brokers registered in ZK through a getMetadata
> request to one of the brokers, not through ZK directly.
>
> Two Kafka clusters can share the same ZK cluster, as long as they use
> different ZK namespace.
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 2, 2013 at 8:23 AM, Haithem Jarraya <a-hjarraya@expedia.com
> >wrote:
>
> > Hi,
> >
> > I am new to Kafka, and I am a bit confused for this particular use case.
> > We are running a global Zookeeper cluster and I was wondering how I can
> > have my Kafka producer discover Kafka hostnames?
> > I saw that in the server.properties I can specify host.name and at that
> > point I will be able to see the list of hostnames in zookeeper
> > /brokers/ids/ so I can have my producer connect to zookeeper and resolve
> > Kafka hostnames.
> > Are there any other solutions?
> > Also, Can we have two separate Kafka Clusters using the same Zookeeper
> > cluster? Just to make sure that we do not collide with other teams. I
> guess
> > not given that the zookeeper path is hard coded. Am I wrong?
> >
> > Many Thanks,
> >
> > Haithem
>

Re: [0.8] Can Kafka producer discover nodes from Zookeeper?

Posted by Jun Rao <ju...@gmail.com>.
Kafka producers identify brokers registered in ZK through a getMetadata
request to one of the brokers, not through ZK directly.

Two Kafka clusters can share the same ZK cluster, as long as they use
different ZK namespace.

Thanks,

Jun


On Fri, Aug 2, 2013 at 8:23 AM, Haithem Jarraya <a-...@expedia.com>wrote:

> Hi,
>
> I am new to Kafka, and I am a bit confused for this particular use case.
> We are running a global Zookeeper cluster and I was wondering how I can
> have my Kafka producer discover Kafka hostnames?
> I saw that in the server.properties I can specify host.name and at that
> point I will be able to see the list of hostnames in zookeeper
> /brokers/ids/ so I can have my producer connect to zookeeper and resolve
> Kafka hostnames.
> Are there any other solutions?
> Also, Can we have two separate Kafka Clusters using the same Zookeeper
> cluster? Just to make sure that we do not collide with other teams. I guess
> not given that the zookeeper path is hard coded. Am I wrong?
>
> Many Thanks,
>
> Haithem