You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Prabhjot Bharaj <pr...@gmail.com> on 2015/08/18 15:03:32 UTC
Zookeeper use cases with Kafka
Hello Folks,
I wish to contribute to Kafka internals. And, one of the things which can
help me do that is understanding how kafka uses zookeeper. I have some of
these basic doubts:-
1. Is zookeeper primarily used for locking ? If yes, in what cases and what
kind of nodes does it use - sequential/ephemeral?
2. Does kafka use zookeeper watches for any of functions ?
3. What kind of state is stored in Zookeeper ? (I believe it has to be the
leader information per partition, but is there anything apart from it?)
What is the scale of data that is stored in Zookeeper ?
Looking forward for your help.
Thanks,
prabcs
Re: Zookeeper use cases with Kafka
Posted by Grant Henke <gh...@cloudera.com>.
Hi Prabcs,
Much of that information can be found in the documentation and on the wiki.
The remaining can be found in the code. Any improvements to the
documentation is not only welcome but encouraged. Below are a few links to
get you started:
Documentation (See "Zookeeper Directories"):
http://kafka.apache.org/documentation.html#distributionimpl
Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
I also recommended that contribution oriented emails be sent to
the Kafka development email list: dev@kafka.apache.org
Thank you,
Grant
On Tue, Aug 18, 2015 at 8:03 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hello Folks,
>
> I wish to contribute to Kafka internals. And, one of the things which can
> help me do that is understanding how kafka uses zookeeper. I have some of
> these basic doubts:-
>
> 1. Is zookeeper primarily used for locking ? If yes, in what cases and what
> kind of nodes does it use - sequential/ephemeral?
>
> 2. Does kafka use zookeeper watches for any of functions ?
>
> 3. What kind of state is stored in Zookeeper ? (I believe it has to be the
> leader information per partition, but is there anything apart from it?)
> What is the scale of data that is stored in Zookeeper ?
>
> Looking forward for your help.
>
> Thanks,
> prabcs
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: Zookeeper use cases with Kafka
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
Hi,
1. ZK is used for co-ordination between brokers, controller election,
leader election, storing topic configuration etc.
I think we use both sequential and ephemeral nodes.
2. Yes, Kafka uses ZK watches for controller changes, new topic creation,
new partition creation, leader change on partition etc. some links here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala
3. Kafka data structures in ZK are documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
Kumar
On Tue, Aug 18, 2015 at 6:33 PM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hello Folks,
>
> I wish to contribute to Kafka internals. And, one of the things which can
> help me do that is understanding how kafka uses zookeeper. I have some of
> these basic doubts:-
>
> 1. Is zookeeper primarily used for locking ? If yes, in what cases and what
> kind of nodes does it use - sequential/ephemeral?
>
> 2. Does kafka use zookeeper watches for any of functions ?
>
> 3. What kind of state is stored in Zookeeper ? (I believe it has to be the
> leader information per partition, but is there anything apart from it?)
> What is the scale of data that is stored in Zookeeper ?
>
> Looking forward for your help.
>
> Thanks,
> prabcs
>
Re: Zookeeper use cases with Kafka
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
Hi,
1. ZK is used for co-ordination between brokers, controller election,
leader election, storing topic configuration etc.
I think we use both sequential and ephemeral nodes.
2. Yes, Kafka uses ZK watches for controller changes, new topic creation,
new partition creation, leader change on partition etc. some links here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala
3. Kafka data structures in ZK are documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
Kumar
On Tue, Aug 18, 2015 at 6:33 PM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hello Folks,
>
> I wish to contribute to Kafka internals. And, one of the things which can
> help me do that is understanding how kafka uses zookeeper. I have some of
> these basic doubts:-
>
> 1. Is zookeeper primarily used for locking ? If yes, in what cases and what
> kind of nodes does it use - sequential/ephemeral?
>
> 2. Does kafka use zookeeper watches for any of functions ?
>
> 3. What kind of state is stored in Zookeeper ? (I believe it has to be the
> leader information per partition, but is there anything apart from it?)
> What is the scale of data that is stored in Zookeeper ?
>
> Looking forward for your help.
>
> Thanks,
> prabcs
>
Re: Zookeeper use cases with Kafka
Posted by Grant Henke <gh...@cloudera.com>.
Hi Prabcs,
Much of that information can be found in the documentation and on the wiki.
The remaining can be found in the code. Any improvements to the
documentation is not only welcome but encouraged. Below are a few links to
get you started:
Documentation (See "Zookeeper Directories"):
http://kafka.apache.org/documentation.html#distributionimpl
Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
I also recommended that contribution oriented emails be sent to
the Kafka development email list: dev@kafka.apache.org
Thank you,
Grant
On Tue, Aug 18, 2015 at 8:03 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hello Folks,
>
> I wish to contribute to Kafka internals. And, one of the things which can
> help me do that is understanding how kafka uses zookeeper. I have some of
> these basic doubts:-
>
> 1. Is zookeeper primarily used for locking ? If yes, in what cases and what
> kind of nodes does it use - sequential/ephemeral?
>
> 2. Does kafka use zookeeper watches for any of functions ?
>
> 3. What kind of state is stored in Zookeeper ? (I believe it has to be the
> leader information per partition, but is there anything apart from it?)
> What is the scale of data that is stored in Zookeeper ?
>
> Looking forward for your help.
>
> Thanks,
> prabcs
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke