You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yuheng Du <yu...@gmail.com> on 2015/03/09 16:19:00 UTC
kafka topic information
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng
Re: kafka topic information
Posted by Yuheng Du <yu...@gmail.com>.
Thanks, got it!
best,
Yuheng
On Mon, Mar 9, 2015 at 11:52 AM, Harsha <ka...@harsha.io> wrote:
> In general users are expected to run zookeeper cluster of 3 or 5 nodes.
> Zookeeper requires quorum of servers running which means at least ceil(n/2)
> servers need to be up. For 3 zookeeper nodes there needs to be atleast 2 zk
> nodes up at any time , i.e your cluster can function fine incase of 1
> machine failure and incase of 5 there should be at least 3 nodes to be up
> and running. For more info on zookeeper you can look under here
> http://zookeeper.apache.org/doc/r3.4.6/
> http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html
>
>
> --
> Harsha
>
> On March 9, 2015 at 8:39:00 AM, Yuheng Du (yuheng.du.hust@gmail.com)
> wrote:
>
> Harsha,
>
> Thanks for reply. So what if the zookeeper cluster fails? Will the topics
> information be lost? What fault-tolerant mechanism does zookeeper offer?
>
> best,
>
> On Mon, Mar 9, 2015 at 11:36 AM, Harsha <ka...@harsha.io> wrote:
>
>> Yuheng,
>> kafka keeps cluster metadata in zookeeper along with topic
>> metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk
>> nodes, /brokers/topics will give you the list of topics .
>>
>> --
>> Harsha
>>
>>
>> On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.hust@gmail.com)
>> wrote:
>>
>> I am wondering where does kafka cluster keep the topic metadata (name,
>> partition, replication, etc)? How does a server recover the topic's
>> metadata and messages after restart and what data will be lost?
>>
>> Thanks for anyone to answer my questions.
>>
>> best,
>> Yuheng
>>
>>
>
Re: kafka topic information
Posted by Harsha <ka...@harsha.io>.
In general users are expected to run zookeeper cluster of 3 or 5 nodes. Zookeeper requires quorum of servers running which means at least ceil(n/2) servers need to be up. For 3 zookeeper nodes there needs to be atleast 2 zk nodes up at any time , i.e your cluster can function fine incase of 1 machine failure and incase of 5 there should be at least 3 nodes to be up and running. For more info on zookeeper you can look under here http://zookeeper.apache.org/doc/r3.4.6/
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html
--
Harsha
On March 9, 2015 at 8:39:00 AM, Yuheng Du (yuheng.du.hust@gmail.com) wrote:
Harsha,
Thanks for reply. So what if the zookeeper cluster fails? Will the topics information be lost? What fault-tolerant mechanism does zookeeper offer?
best,
On Mon, Mar 9, 2015 at 11:36 AM, Harsha <ka...@harsha.io> wrote:
Yuheng,
kafka keeps cluster metadata in zookeeper along with topic metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk nodes, /brokers/topics will give you the list of topics .
--
Harsha
On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.hust@gmail.com) wrote:
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng
Re: kafka topic information
Posted by Yuheng Du <yu...@gmail.com>.
Harsha,
Thanks for reply. So what if the zookeeper cluster fails? Will the topics
information be lost? What fault-tolerant mechanism does zookeeper offer?
best,
On Mon, Mar 9, 2015 at 11:36 AM, Harsha <ka...@harsha.io> wrote:
> Yuheng,
> kafka keeps cluster metadata in zookeeper along with topic
> metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk
> nodes, /brokers/topics will give you the list of topics .
>
> --
> Harsha
>
>
> On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.hust@gmail.com)
> wrote:
>
> I am wondering where does kafka cluster keep the topic metadata (name,
> partition, replication, etc)? How does a server recover the topic's
> metadata and messages after restart and what data will be lost?
>
> Thanks for anyone to answer my questions.
>
> best,
> Yuheng
>
>
Re: kafka topic information
Posted by Harsha <ka...@harsha.io>.
Yuheng,
kafka keeps cluster metadata in zookeeper along with topic metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk nodes, /brokers/topics will give you the list of topics .
--
Harsha
On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.hust@gmail.com) wrote:
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng