You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by David Ormsbee <da...@datadoghq.com> on 2011/12/06 15:34:58 UTC
ZooKeeper-based Producer behavior before there are topics
Hi folks,
I'm implementing ZooKeeper support in our Python client, and I was
wondering what the expected behavior of the Producer is when the brokers
are completely empty. At this point, the brokers have registered themselves
in /brokers/ids/[0..N], but there are no topics registered under
/brokers/topics. If someone wants to create a Producer at this point and
start sending messages to a new topic, is that an error because no brokers
are registered for that topic? Or should I just make the initial assumption
that all brokers have one partition, send messages to those to create the
topic on the brokers, and then readjust when the the brokers publish their
partitions-per-topic information to ZK after the initial writes?
I'm sure I'm missing something obvious here, since we'd run into a similar
problem for new, empty brokers joining the pool.
As an aside, is there any way to query metadata about the broker directly?
So that a simple consumer might be able to ask a broker for what topics
they have, how many partitions, what version it's running, etc? Or is that
only ever published through ZooKeeper?
Thank you.
Dave
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Jun Rao <ju...@gmail.com>.
David,
There is some logic in ZKBrokerPartitionInfo.bootstrapWithExistingBrokers
that deals with the case that the topic is initially empty.
Currently, there is no API to get metadata. One has to query ZK directly.
We can discuss whether to add metadata apis. Do you want to create a jira
for that?
Thanks,
Jun
On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com> wrote:
> Hi folks,
>
> I'm implementing ZooKeeper support in our Python client, and I was
> wondering what the expected behavior of the Producer is when the brokers
> are completely empty. At this point, the brokers have registered themselves
> in /brokers/ids/[0..N], but there are no topics registered under
> /brokers/topics. If someone wants to create a Producer at this point and
> start sending messages to a new topic, is that an error because no brokers
> are registered for that topic? Or should I just make the initial assumption
> that all brokers have one partition, send messages to those to create the
> topic on the brokers, and then readjust when the the brokers publish their
> partitions-per-topic information to ZK after the initial writes?
>
> I'm sure I'm missing something obvious here, since we'd run into a similar
> problem for new, empty brokers joining the pool.
>
> As an aside, is there any way to query metadata about the broker directly?
> So that a simple consumer might be able to ask a broker for what topics
> they have, how many partitions, what version it's running, etc? Or is that
> only ever published through ZooKeeper?
>
> Thank you.
>
> Dave
>
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Jun Rao <ju...@gmail.com>.
Inder,
If you just want to do random distribution, ZK-based load balancing is not
too much different from VIP. We have experienced some subtle issues with
VIP. For example, it's not easy to figure out the real IP of a client
connected to a broker. Also, VIP timeout can be tricky (need to send RST to
the client if connection is killed by timeouts).
Further more, if you want to partition based on a key, then this is hard
for VIP to do.
Thanks,
Jun
On Sun, Dec 11, 2011 at 9:20 PM, Inder Pall <in...@gmail.com> wrote:
> Neha,
>
> how would it be better than putting the new broker behind a VIP.
> Is the ZK based producer equipped with more info to take a better call.
>
>
>
> On Sun, Dec 11, 2011 at 11:57 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Inder,
> >
> > Zookeeper is used on the producer side for load balancing. It is used for
> > automatic detection of new and failed brokers.
> >
> > Thanks,
> > Neha
> >
> > On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <in...@gmail.com>
> wrote:
> >
> > > why do u want to integrate zk at producer side. can you please
> elaborate
> > > your usecase.
> > >
> > > is it for load balancing.
> > > On Dec 7, 2011 12:35 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > > >
> > > > David,
> > > >
> > > > Could you also comment whether you want a programming api or a
> > monitoring
> > > > api?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <da...@datadoghq.com>
> > > wrote:
> > > >
> > > > > Jun, Neha:
> > > > >
> > > > > Thank you for pointing me the right way on the Producer issue. :-)
> > > > >
> > > > > As for the metadata requests, I've filed a ticket here:
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-223
> > > > >
> > > > > Take care.
> > > > >
> > > > > Dave
> > > > >
> > > > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> > > neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi Dave,
> > > > > >
> > > > > > >> Or should I just make the initial assumption
> > > > > > that all brokers have one partition, send messages to those to
> > create
> > > the
> > > > > > topic on the brokers, and then readjust when the the brokers
> > publish
> > > > > their
> > > > > > partitions-per-topic information to ZK after the initial writes?
> > > > > >
> > > > > > That's exactly what we do. You can take a look at
> > > > > > bootstrapWithExistingBrokers() in
> > > kafka.producer.ZKBrokerPartitionInfo.
> > > > > >
> > > > > > >> As an aside, is there any way to query metadata about the
> broker
> > > > > > directly?
> > > > > > So that a simple consumer might be able to ask a broker for what
> > > topics
> > > > > > they have, how many partitions, what version it's running, etc?
> > > > > >
> > > > > > Today, we don't have such APIs. But I was thinking we can have
> > > something
> > > > > > similar to the 4-letter commands in zookeeper.
> > > > > > For example,
> > > > > >
> > > > > > echo topics | nc broker-host broker-port (Returns all topics
> > > with
> > > > > > number of partitions on each broker)
> > > > > > echo srvr | nc broker-host broker-port (Returns basic
> > broker
> > > > > stats
> > > > > > like version, number of topics etc)
> > > > > >
> > > > > > If the above functionality is useful, lets file a JIRA for it.
> > > > > >
> > > > > > Thanks
> > > > > > Neha
> > > > > >
> > > > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <
> dave@datadoghq.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > I'm implementing ZooKeeper support in our Python client, and I
> > was
> > > > > > > wondering what the expected behavior of the Producer is when
> the
> > > > > brokers
> > > > > > > are completely empty. At this point, the brokers have
> registered
> > > > > > themselves
> > > > > > > in /brokers/ids/[0..N], but there are no topics registered
> under
> > > > > > > /brokers/topics. If someone wants to create a Producer at this
> > > point
> > > > > and
> > > > > > > start sending messages to a new topic, is that an error because
> > no
> > > > > > brokers
> > > > > > > are registered for that topic? Or should I just make the
> initial
> > > > > > assumption
> > > > > > > that all brokers have one partition, send messages to those to
> > > create
> > > > > the
> > > > > > > topic on the brokers, and then readjust when the the brokers
> > > publish
> > > > > > their
> > > > > > > partitions-per-topic information to ZK after the initial
> writes?
> > > > > > >
> > > > > > > I'm sure I'm missing something obvious here, since we'd run
> into
> > a
> > > > > > similar
> > > > > > > problem for new, empty brokers joining the pool.
> > > > > > >
> > > > > > > As an aside, is there any way to query metadata about the
> broker
> > > > > > directly?
> > > > > > > So that a simple consumer might be able to ask a broker for
> what
> > > topics
> > > > > > > they have, how many partitions, what version it's running, etc?
> > Or
> > > is
> > > > > > that
> > > > > > > only ever published through ZooKeeper?
> > > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > > > Dave
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
>
>
> --
> Thanks,
> - Inder
> Tech Platforms @Inmobi
> Linkedin - http://goo.gl/eR4Ub
>
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Inder Pall <in...@gmail.com>.
Neha,
how would it be better than putting the new broker behind a VIP.
Is the ZK based producer equipped with more info to take a better call.
On Sun, Dec 11, 2011 at 11:57 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Inder,
>
> Zookeeper is used on the producer side for load balancing. It is used for
> automatic detection of new and failed brokers.
>
> Thanks,
> Neha
>
> On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <in...@gmail.com> wrote:
>
> > why do u want to integrate zk at producer side. can you please elaborate
> > your usecase.
> >
> > is it for load balancing.
> > On Dec 7, 2011 12:35 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > >
> > > David,
> > >
> > > Could you also comment whether you want a programming api or a
> monitoring
> > > api?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <da...@datadoghq.com>
> > wrote:
> > >
> > > > Jun, Neha:
> > > >
> > > > Thank you for pointing me the right way on the Producer issue. :-)
> > > >
> > > > As for the metadata requests, I've filed a ticket here:
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-223
> > > >
> > > > Take care.
> > > >
> > > > Dave
> > > >
> > > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Dave,
> > > > >
> > > > > >> Or should I just make the initial assumption
> > > > > that all brokers have one partition, send messages to those to
> create
> > the
> > > > > topic on the brokers, and then readjust when the the brokers
> publish
> > > > their
> > > > > partitions-per-topic information to ZK after the initial writes?
> > > > >
> > > > > That's exactly what we do. You can take a look at
> > > > > bootstrapWithExistingBrokers() in
> > kafka.producer.ZKBrokerPartitionInfo.
> > > > >
> > > > > >> As an aside, is there any way to query metadata about the broker
> > > > > directly?
> > > > > So that a simple consumer might be able to ask a broker for what
> > topics
> > > > > they have, how many partitions, what version it's running, etc?
> > > > >
> > > > > Today, we don't have such APIs. But I was thinking we can have
> > something
> > > > > similar to the 4-letter commands in zookeeper.
> > > > > For example,
> > > > >
> > > > > echo topics | nc broker-host broker-port (Returns all topics
> > with
> > > > > number of partitions on each broker)
> > > > > echo srvr | nc broker-host broker-port (Returns basic
> broker
> > > > stats
> > > > > like version, number of topics etc)
> > > > >
> > > > > If the above functionality is useful, lets file a JIRA for it.
> > > > >
> > > > > Thanks
> > > > > Neha
> > > > >
> > > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com>
> > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > I'm implementing ZooKeeper support in our Python client, and I
> was
> > > > > > wondering what the expected behavior of the Producer is when the
> > > > brokers
> > > > > > are completely empty. At this point, the brokers have registered
> > > > > themselves
> > > > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > > > /brokers/topics. If someone wants to create a Producer at this
> > point
> > > > and
> > > > > > start sending messages to a new topic, is that an error because
> no
> > > > > brokers
> > > > > > are registered for that topic? Or should I just make the initial
> > > > > assumption
> > > > > > that all brokers have one partition, send messages to those to
> > create
> > > > the
> > > > > > topic on the brokers, and then readjust when the the brokers
> > publish
> > > > > their
> > > > > > partitions-per-topic information to ZK after the initial writes?
> > > > > >
> > > > > > I'm sure I'm missing something obvious here, since we'd run into
> a
> > > > > similar
> > > > > > problem for new, empty brokers joining the pool.
> > > > > >
> > > > > > As an aside, is there any way to query metadata about the broker
> > > > > directly?
> > > > > > So that a simple consumer might be able to ask a broker for what
> > topics
> > > > > > they have, how many partitions, what version it's running, etc?
> Or
> > is
> > > > > that
> > > > > > only ever published through ZooKeeper?
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > Dave
> > > > > >
> > > > >
> > > >
> >
>
--
Thanks,
- Inder
Tech Platforms @Inmobi
Linkedin - http://goo.gl/eR4Ub
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Neha Narkhede <ne...@gmail.com>.
Inder,
Zookeeper is used on the producer side for load balancing. It is used for
automatic detection of new and failed brokers.
Thanks,
Neha
On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <in...@gmail.com> wrote:
> why do u want to integrate zk at producer side. can you please elaborate
> your usecase.
>
> is it for load balancing.
> On Dec 7, 2011 12:35 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >
> > David,
> >
> > Could you also comment whether you want a programming api or a monitoring
> > api?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <da...@datadoghq.com>
> wrote:
> >
> > > Jun, Neha:
> > >
> > > Thank you for pointing me the right way on the Producer issue. :-)
> > >
> > > As for the metadata requests, I've filed a ticket here:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-223
> > >
> > > Take care.
> > >
> > > Dave
> > >
> > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Hi Dave,
> > > >
> > > > >> Or should I just make the initial assumption
> > > > that all brokers have one partition, send messages to those to create
> the
> > > > topic on the brokers, and then readjust when the the brokers publish
> > > their
> > > > partitions-per-topic information to ZK after the initial writes?
> > > >
> > > > That's exactly what we do. You can take a look at
> > > > bootstrapWithExistingBrokers() in
> kafka.producer.ZKBrokerPartitionInfo.
> > > >
> > > > >> As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > they have, how many partitions, what version it's running, etc?
> > > >
> > > > Today, we don't have such APIs. But I was thinking we can have
> something
> > > > similar to the 4-letter commands in zookeeper.
> > > > For example,
> > > >
> > > > echo topics | nc broker-host broker-port (Returns all topics
> with
> > > > number of partitions on each broker)
> > > > echo srvr | nc broker-host broker-port (Returns basic broker
> > > stats
> > > > like version, number of topics etc)
> > > >
> > > > If the above functionality is useful, lets file a JIRA for it.
> > > >
> > > > Thanks
> > > > Neha
> > > >
> > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com>
> > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm implementing ZooKeeper support in our Python client, and I was
> > > > > wondering what the expected behavior of the Producer is when the
> > > brokers
> > > > > are completely empty. At this point, the brokers have registered
> > > > themselves
> > > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > > /brokers/topics. If someone wants to create a Producer at this
> point
> > > and
> > > > > start sending messages to a new topic, is that an error because no
> > > > brokers
> > > > > are registered for that topic? Or should I just make the initial
> > > > assumption
> > > > > that all brokers have one partition, send messages to those to
> create
> > > the
> > > > > topic on the brokers, and then readjust when the the brokers
> publish
> > > > their
> > > > > partitions-per-topic information to ZK after the initial writes?
> > > > >
> > > > > I'm sure I'm missing something obvious here, since we'd run into a
> > > > similar
> > > > > problem for new, empty brokers joining the pool.
> > > > >
> > > > > As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > > they have, how many partitions, what version it's running, etc? Or
> is
> > > > that
> > > > > only ever published through ZooKeeper?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Dave
> > > > >
> > > >
> > >
>
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Inder Pall <in...@gmail.com>.
why do u want to integrate zk at producer side. can you please elaborate
your usecase.
is it for load balancing.
On Dec 7, 2011 12:35 AM, "Jun Rao" <ju...@gmail.com> wrote:
>
> David,
>
> Could you also comment whether you want a programming api or a monitoring
> api?
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <da...@datadoghq.com> wrote:
>
> > Jun, Neha:
> >
> > Thank you for pointing me the right way on the Producer issue. :-)
> >
> > As for the metadata requests, I've filed a ticket here:
> >
> > https://issues.apache.org/jira/browse/KAFKA-223
> >
> > Take care.
> >
> > Dave
> >
> > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Hi Dave,
> > >
> > > >> Or should I just make the initial assumption
> > > that all brokers have one partition, send messages to those to create
the
> > > topic on the brokers, and then readjust when the the brokers publish
> > their
> > > partitions-per-topic information to ZK after the initial writes?
> > >
> > > That's exactly what we do. You can take a look at
> > > bootstrapWithExistingBrokers() in
kafka.producer.ZKBrokerPartitionInfo.
> > >
> > > >> As an aside, is there any way to query metadata about the broker
> > > directly?
> > > So that a simple consumer might be able to ask a broker for what
topics
> > > they have, how many partitions, what version it's running, etc?
> > >
> > > Today, we don't have such APIs. But I was thinking we can have
something
> > > similar to the 4-letter commands in zookeeper.
> > > For example,
> > >
> > > echo topics | nc broker-host broker-port (Returns all topics with
> > > number of partitions on each broker)
> > > echo srvr | nc broker-host broker-port (Returns basic broker
> > stats
> > > like version, number of topics etc)
> > >
> > > If the above functionality is useful, lets file a JIRA for it.
> > >
> > > Thanks
> > > Neha
> > >
> > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com>
> > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I'm implementing ZooKeeper support in our Python client, and I was
> > > > wondering what the expected behavior of the Producer is when the
> > brokers
> > > > are completely empty. At this point, the brokers have registered
> > > themselves
> > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > /brokers/topics. If someone wants to create a Producer at this point
> > and
> > > > start sending messages to a new topic, is that an error because no
> > > brokers
> > > > are registered for that topic? Or should I just make the initial
> > > assumption
> > > > that all brokers have one partition, send messages to those to
create
> > the
> > > > topic on the brokers, and then readjust when the the brokers publish
> > > their
> > > > partitions-per-topic information to ZK after the initial writes?
> > > >
> > > > I'm sure I'm missing something obvious here, since we'd run into a
> > > similar
> > > > problem for new, empty brokers joining the pool.
> > > >
> > > > As an aside, is there any way to query metadata about the broker
> > > directly?
> > > > So that a simple consumer might be able to ask a broker for what
topics
> > > > they have, how many partitions, what version it's running, etc? Or
is
> > > that
> > > > only ever published through ZooKeeper?
> > > >
> > > > Thank you.
> > > >
> > > > Dave
> > > >
> > >
> >
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Jun Rao <ju...@gmail.com>.
David,
Could you also comment whether you want a programming api or a monitoring
api?
Thanks,
Jun
On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <da...@datadoghq.com> wrote:
> Jun, Neha:
>
> Thank you for pointing me the right way on the Producer issue. :-)
>
> As for the metadata requests, I've filed a ticket here:
>
> https://issues.apache.org/jira/browse/KAFKA-223
>
> Take care.
>
> Dave
>
> On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Hi Dave,
> >
> > >> Or should I just make the initial assumption
> > that all brokers have one partition, send messages to those to create the
> > topic on the brokers, and then readjust when the the brokers publish
> their
> > partitions-per-topic information to ZK after the initial writes?
> >
> > That's exactly what we do. You can take a look at
> > bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.
> >
> > >> As an aside, is there any way to query metadata about the broker
> > directly?
> > So that a simple consumer might be able to ask a broker for what topics
> > they have, how many partitions, what version it's running, etc?
> >
> > Today, we don't have such APIs. But I was thinking we can have something
> > similar to the 4-letter commands in zookeeper.
> > For example,
> >
> > echo topics | nc broker-host broker-port (Returns all topics with
> > number of partitions on each broker)
> > echo srvr | nc broker-host broker-port (Returns basic broker
> stats
> > like version, number of topics etc)
> >
> > If the above functionality is useful, lets file a JIRA for it.
> >
> > Thanks
> > Neha
> >
> > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com>
> wrote:
> >
> > > Hi folks,
> > >
> > > I'm implementing ZooKeeper support in our Python client, and I was
> > > wondering what the expected behavior of the Producer is when the
> brokers
> > > are completely empty. At this point, the brokers have registered
> > themselves
> > > in /brokers/ids/[0..N], but there are no topics registered under
> > > /brokers/topics. If someone wants to create a Producer at this point
> and
> > > start sending messages to a new topic, is that an error because no
> > brokers
> > > are registered for that topic? Or should I just make the initial
> > assumption
> > > that all brokers have one partition, send messages to those to create
> the
> > > topic on the brokers, and then readjust when the the brokers publish
> > their
> > > partitions-per-topic information to ZK after the initial writes?
> > >
> > > I'm sure I'm missing something obvious here, since we'd run into a
> > similar
> > > problem for new, empty brokers joining the pool.
> > >
> > > As an aside, is there any way to query metadata about the broker
> > directly?
> > > So that a simple consumer might be able to ask a broker for what topics
> > > they have, how many partitions, what version it's running, etc? Or is
> > that
> > > only ever published through ZooKeeper?
> > >
> > > Thank you.
> > >
> > > Dave
> > >
> >
>
Re: ZooKeeper-based Producer behavior before there are topics
Posted by David Ormsbee <da...@datadoghq.com>.
Jun, Neha:
Thank you for pointing me the right way on the Producer issue. :-)
As for the metadata requests, I've filed a ticket here:
https://issues.apache.org/jira/browse/KAFKA-223
Take care.
Dave
On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Hi Dave,
>
> >> Or should I just make the initial assumption
> that all brokers have one partition, send messages to those to create the
> topic on the brokers, and then readjust when the the brokers publish their
> partitions-per-topic information to ZK after the initial writes?
>
> That's exactly what we do. You can take a look at
> bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.
>
> >> As an aside, is there any way to query metadata about the broker
> directly?
> So that a simple consumer might be able to ask a broker for what topics
> they have, how many partitions, what version it's running, etc?
>
> Today, we don't have such APIs. But I was thinking we can have something
> similar to the 4-letter commands in zookeeper.
> For example,
>
> echo topics | nc broker-host broker-port (Returns all topics with
> number of partitions on each broker)
> echo srvr | nc broker-host broker-port (Returns basic broker stats
> like version, number of topics etc)
>
> If the above functionality is useful, lets file a JIRA for it.
>
> Thanks
> Neha
>
> On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com> wrote:
>
> > Hi folks,
> >
> > I'm implementing ZooKeeper support in our Python client, and I was
> > wondering what the expected behavior of the Producer is when the brokers
> > are completely empty. At this point, the brokers have registered
> themselves
> > in /brokers/ids/[0..N], but there are no topics registered under
> > /brokers/topics. If someone wants to create a Producer at this point and
> > start sending messages to a new topic, is that an error because no
> brokers
> > are registered for that topic? Or should I just make the initial
> assumption
> > that all brokers have one partition, send messages to those to create the
> > topic on the brokers, and then readjust when the the brokers publish
> their
> > partitions-per-topic information to ZK after the initial writes?
> >
> > I'm sure I'm missing something obvious here, since we'd run into a
> similar
> > problem for new, empty brokers joining the pool.
> >
> > As an aside, is there any way to query metadata about the broker
> directly?
> > So that a simple consumer might be able to ask a broker for what topics
> > they have, how many partitions, what version it's running, etc? Or is
> that
> > only ever published through ZooKeeper?
> >
> > Thank you.
> >
> > Dave
> >
>
Re: ZooKeeper-based Producer behavior before there are topics
Posted by Neha Narkhede <ne...@gmail.com>.
Hi Dave,
>> Or should I just make the initial assumption
that all brokers have one partition, send messages to those to create the
topic on the brokers, and then readjust when the the brokers publish their
partitions-per-topic information to ZK after the initial writes?
That's exactly what we do. You can take a look at
bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.
>> As an aside, is there any way to query metadata about the broker
directly?
So that a simple consumer might be able to ask a broker for what topics
they have, how many partitions, what version it's running, etc?
Today, we don't have such APIs. But I was thinking we can have something
similar to the 4-letter commands in zookeeper.
For example,
echo topics | nc broker-host broker-port (Returns all topics with
number of partitions on each broker)
echo srvr | nc broker-host broker-port (Returns basic broker stats
like version, number of topics etc)
If the above functionality is useful, lets file a JIRA for it.
Thanks
Neha
On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <da...@datadoghq.com> wrote:
> Hi folks,
>
> I'm implementing ZooKeeper support in our Python client, and I was
> wondering what the expected behavior of the Producer is when the brokers
> are completely empty. At this point, the brokers have registered themselves
> in /brokers/ids/[0..N], but there are no topics registered under
> /brokers/topics. If someone wants to create a Producer at this point and
> start sending messages to a new topic, is that an error because no brokers
> are registered for that topic? Or should I just make the initial assumption
> that all brokers have one partition, send messages to those to create the
> topic on the brokers, and then readjust when the the brokers publish their
> partitions-per-topic information to ZK after the initial writes?
>
> I'm sure I'm missing something obvious here, since we'd run into a similar
> problem for new, empty brokers joining the pool.
>
> As an aside, is there any way to query metadata about the broker directly?
> So that a simple consumer might be able to ask a broker for what topics
> they have, how many partitions, what version it's running, etc? Or is that
> only ever published through ZooKeeper?
>
> Thank you.
>
> Dave
>