Kafka Zookeeper queries

[Achanta Vamsi Subhash <ac...@flipkart.com>]

Hi,

How often does Kafka query zookeeper while producing and consuming?

Ex:
If there is a single partition to which we produce and a HighLevel consumer
running on it, how many read/write queries to zookeeper happen.

Extending further, multiple topics with ~100 partitions each, how many
zookeeper calls will be made (read/write).

What is the max limit of no of partitions / kafka cluster that zookeeper
can handle?

-- 
Regards
Vamsi Subhash

Re: Kafka Zookeeper queries

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

I might not be the best one to answer this question. It will be better if
someone has more operation experience can help out here.

Jiangjie (Becket) Qin

On 4/21/15, 12:00 PM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
wrote:

>@Qin
>Thanks for the inputs. Could you point us what we should monitor in
>zookeeper if we have 20k partitions (this might go ~1 lakh in the coming 6
>months)? We are not running zookeeper on SSDs and we were facing
>ZkConnectionLoss in HighLevelConsumer, sometimes for all the topics. What
>are the critical things to monitor/optimise wrt. zookeeper.
>
>@Pushkar
>Yes. We are using kafka for storing offsets instead of zookeeper.
>
>On Tue, Apr 21, 2015 at 11:55 PM, pushkar priyadarshi <
>priyadarshi.pushkar@gmail.com> wrote:
>
>> In my knowledge if you are using 0.8.2.1 which is latest stable you can
>> sync up your consumer offsets in kafka itself instead of Zk which
>>further
>> brings down write load on ZKs.
>>
>> Regards,
>> Pushkar
>>
>> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin
>><jq...@linkedin.com.invalid>
>> wrote:
>>
>> > 20000 partitions should be OK.
>> >
>> > On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <
>> achanta.vamsi@flipkart.com>
>> > wrote:
>> >
>> > >We are planning to have ~20000 partitions. Will it be a bottleneck?
>> > >
>> > >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin
>> <jqin@linkedin.com.invalid
>> > >
>> > >wrote:
>> > >
>> > >> Producers usually do not query zookeeper at all.
>> > >> Consumers usually query zookeeper at beginning or rebalance. It is
>> > >> supposed to be in frequent if you don¹t have consumers come and go
>>all
>> > >>the
>> > >> time. One exception is that if you are using zookeeper based
>>consumer
>> > >> offset commit, it will commit offset to zookeeper frequently.
>> > >> In Kafka, the most heavily used mechanism for zookeeper is
>>zookeeper
>> > >> listener and they are not fired in a regular frequency.
>> > >>
>> > >> The limitation of Zookeeper usage for Kafka I am aware of is
>>probably
>> > >>the
>> > >> size of each zNode. As long as you don¹t have so many partitions
>>that
>> > >> zNode cannot handle, it should be fine.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> Jiangjie (Becket) Qin
>> > >>
>> > >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
>> > >><ac...@flipkart.com>
>> > >> wrote:
>> > >>
>> > >> >Hi,
>> > >> >
>> > >> >Could anyone help with this?
>> > >> >
>> > >> >Thanks.
>> > >> >
>> > >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
>> > >> >achanta.vamsi@flipkart.com> wrote:
>> > >> >
>> > >> >> Hi,
>> > >> >>
>> > >> >> How often does Kafka query zookeeper while producing and
>>consuming?
>> > >> >>
>> > >> >> Ex:
>> > >> >> If there is a single partition to which we produce and a
>>HighLevel
>> > >> >> consumer running on it, how many read/write queries to zookeeper
>> > >>happen.
>> > >> >>
>> > >> >> Extending further, multiple topics with ~100 partitions each,
>>how
>> > >>many
>> > >> >> zookeeper calls will be made (read/write).
>> > >> >>
>> > >> >> What is the max limit of no of partitions / kafka cluster that
>> > >>zookeeper
>> > >> >> can handle?
>> > >> >>
>> > >> >> --
>> > >> >> Regards
>> > >> >> Vamsi Subhash
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> >--
>> > >> >Regards
>> > >> >Vamsi Subhash
>> > >>
>> > >>
>> > >
>> > >
>> > >--
>> > >Regards
>> > >Vamsi Subhash
>> >
>> >
>>
>
>
>
>-- 
>Regards
>Vamsi Subhash

Re: Kafka Zookeeper queries

[Achanta Vamsi Subhash <ac...@flipkart.com>]

@Qin
Thanks for the inputs. Could you point us what we should monitor in
zookeeper if we have 20k partitions (this might go ~1 lakh in the coming 6
months)? We are not running zookeeper on SSDs and we were facing
ZkConnectionLoss in HighLevelConsumer, sometimes for all the topics. What
are the critical things to monitor/optimise wrt. zookeeper.

@Pushkar
Yes. We are using kafka for storing offsets instead of zookeeper.

On Tue, Apr 21, 2015 at 11:55 PM, pushkar priyadarshi <
priyadarshi.pushkar@gmail.com> wrote:

> In my knowledge if you are using 0.8.2.1 which is latest stable you can
> sync up your consumer offsets in kafka itself instead of Zk which further
> brings down write load on ZKs.
>
> Regards,
> Pushkar
>
> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > 20000 partitions should be OK.
> >
> > On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <
> achanta.vamsi@flipkart.com>
> > wrote:
> >
> > >We are planning to have ~20000 partitions. Will it be a bottleneck?
> > >
> > >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin
> <jqin@linkedin.com.invalid
> > >
> > >wrote:
> > >
> > >> Producers usually do not query zookeeper at all.
> > >> Consumers usually query zookeeper at beginning or rebalance. It is
> > >> supposed to be in frequent if you don¹t have consumers come and go all
> > >>the
> > >> time. One exception is that if you are using zookeeper based consumer
> > >> offset commit, it will commit offset to zookeeper frequently.
> > >> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
> > >> listener and they are not fired in a regular frequency.
> > >>
> > >> The limitation of Zookeeper usage for Kafka I am aware of is probably
> > >>the
> > >> size of each zNode. As long as you don¹t have so many partitions that
> > >> zNode cannot handle, it should be fine.
> > >>
> > >> Thanks.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
> > >><ac...@flipkart.com>
> > >> wrote:
> > >>
> > >> >Hi,
> > >> >
> > >> >Could anyone help with this?
> > >> >
> > >> >Thanks.
> > >> >
> > >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
> > >> >achanta.vamsi@flipkart.com> wrote:
> > >> >
> > >> >> Hi,
> > >> >>
> > >> >> How often does Kafka query zookeeper while producing and consuming?
> > >> >>
> > >> >> Ex:
> > >> >> If there is a single partition to which we produce and a HighLevel
> > >> >> consumer running on it, how many read/write queries to zookeeper
> > >>happen.
> > >> >>
> > >> >> Extending further, multiple topics with ~100 partitions each, how
> > >>many
> > >> >> zookeeper calls will be made (read/write).
> > >> >>
> > >> >> What is the max limit of no of partitions / kafka cluster that
> > >>zookeeper
> > >> >> can handle?
> > >> >>
> > >> >> --
> > >> >> Regards
> > >> >> Vamsi Subhash
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> >--
> > >> >Regards
> > >> >Vamsi Subhash
> > >>
> > >>
> > >
> > >
> > >--
> > >Regards
> > >Vamsi Subhash
> >
> >
>



-- 
Regards
Vamsi Subhash

Re: Kafka Zookeeper queries

[pushkar priyadarshi <pr...@gmail.com>]

In my knowledge if you are using 0.8.2.1 which is latest stable you can
sync up your consumer offsets in kafka itself instead of Zk which further
brings down write load on ZKs.

Regards,
Pushkar

On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> 20000 partitions should be OK.
>
> On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
> wrote:
>
> >We are planning to have ~20000 partitions. Will it be a bottleneck?
> >
> >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> >wrote:
> >
> >> Producers usually do not query zookeeper at all.
> >> Consumers usually query zookeeper at beginning or rebalance. It is
> >> supposed to be in frequent if you don¹t have consumers come and go all
> >>the
> >> time. One exception is that if you are using zookeeper based consumer
> >> offset commit, it will commit offset to zookeeper frequently.
> >> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
> >> listener and they are not fired in a regular frequency.
> >>
> >> The limitation of Zookeeper usage for Kafka I am aware of is probably
> >>the
> >> size of each zNode. As long as you don¹t have so many partitions that
> >> zNode cannot handle, it should be fine.
> >>
> >> Thanks.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
> >><ac...@flipkart.com>
> >> wrote:
> >>
> >> >Hi,
> >> >
> >> >Could anyone help with this?
> >> >
> >> >Thanks.
> >> >
> >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
> >> >achanta.vamsi@flipkart.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> How often does Kafka query zookeeper while producing and consuming?
> >> >>
> >> >> Ex:
> >> >> If there is a single partition to which we produce and a HighLevel
> >> >> consumer running on it, how many read/write queries to zookeeper
> >>happen.
> >> >>
> >> >> Extending further, multiple topics with ~100 partitions each, how
> >>many
> >> >> zookeeper calls will be made (read/write).
> >> >>
> >> >> What is the max limit of no of partitions / kafka cluster that
> >>zookeeper
> >> >> can handle?
> >> >>
> >> >> --
> >> >> Regards
> >> >> Vamsi Subhash
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >Regards
> >> >Vamsi Subhash
> >>
> >>
> >
> >
> >--
> >Regards
> >Vamsi Subhash
>
>

Re: Kafka Zookeeper queries

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

20000 partitions should be OK.

On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
wrote:

>We are planning to have ~20000 partitions. Will it be a bottleneck?
>
>On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
>wrote:
>
>> Producers usually do not query zookeeper at all.
>> Consumers usually query zookeeper at beginning or rebalance. It is
>> supposed to be in frequent if you don¹t have consumers come and go all
>>the
>> time. One exception is that if you are using zookeeper based consumer
>> offset commit, it will commit offset to zookeeper frequently.
>> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
>> listener and they are not fired in a regular frequency.
>>
>> The limitation of Zookeeper usage for Kafka I am aware of is probably
>>the
>> size of each zNode. As long as you don¹t have so many partitions that
>> zNode cannot handle, it should be fine.
>>
>> Thanks.
>>
>> Jiangjie (Becket) Qin
>>
>> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
>><ac...@flipkart.com>
>> wrote:
>>
>> >Hi,
>> >
>> >Could anyone help with this?
>> >
>> >Thanks.
>> >
>> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
>> >achanta.vamsi@flipkart.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> How often does Kafka query zookeeper while producing and consuming?
>> >>
>> >> Ex:
>> >> If there is a single partition to which we produce and a HighLevel
>> >> consumer running on it, how many read/write queries to zookeeper
>>happen.
>> >>
>> >> Extending further, multiple topics with ~100 partitions each, how
>>many
>> >> zookeeper calls will be made (read/write).
>> >>
>> >> What is the max limit of no of partitions / kafka cluster that
>>zookeeper
>> >> can handle?
>> >>
>> >> --
>> >> Regards
>> >> Vamsi Subhash
>> >>
>> >
>> >
>> >
>> >--
>> >Regards
>> >Vamsi Subhash
>>
>>
>
>
>-- 
>Regards
>Vamsi Subhash

Re: Kafka Zookeeper queries

[Achanta Vamsi Subhash <ac...@flipkart.com>]

We are planning to have ~20000 partitions. Will it be a bottleneck?

On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> Producers usually do not query zookeeper at all.
> Consumers usually query zookeeper at beginning or rebalance. It is
> supposed to be in frequent if you don¹t have consumers come and go all the
> time. One exception is that if you are using zookeeper based consumer
> offset commit, it will commit offset to zookeeper frequently.
> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
> listener and they are not fired in a regular frequency.
>
> The limitation of Zookeeper usage for Kafka I am aware of is probably the
> size of each zNode. As long as you don¹t have so many partitions that
> zNode cannot handle, it should be fine.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>
> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
> wrote:
>
> >Hi,
> >
> >Could anyone help with this?
> >
> >Thanks.
> >
> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
> >achanta.vamsi@flipkart.com> wrote:
> >
> >> Hi,
> >>
> >> How often does Kafka query zookeeper while producing and consuming?
> >>
> >> Ex:
> >> If there is a single partition to which we produce and a HighLevel
> >> consumer running on it, how many read/write queries to zookeeper happen.
> >>
> >> Extending further, multiple topics with ~100 partitions each, how many
> >> zookeeper calls will be made (read/write).
> >>
> >> What is the max limit of no of partitions / kafka cluster that zookeeper
> >> can handle?
> >>
> >> --
> >> Regards
> >> Vamsi Subhash
> >>
> >
> >
> >
> >--
> >Regards
> >Vamsi Subhash
>
>


-- 
Regards
Vamsi Subhash

Re: Kafka Zookeeper queries

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

Producers usually do not query zookeeper at all.
Consumers usually query zookeeper at beginning or rebalance. It is
supposed to be in frequent if you don¹t have consumers come and go all the
time. One exception is that if you are using zookeeper based consumer
offset commit, it will commit offset to zookeeper frequently.
In Kafka, the most heavily used mechanism for zookeeper is zookeeper
listener and they are not fired in a regular frequency.

The limitation of Zookeeper usage for Kafka I am aware of is probably the
size of each zNode. As long as you don¹t have so many partitions that
zNode cannot handle, it should be fine.

Thanks.

Jiangjie (Becket) Qin

On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash" <ac...@flipkart.com>
wrote:

>Hi,
>
>Could anyone help with this?
>
>Thanks.
>
>On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
>achanta.vamsi@flipkart.com> wrote:
>
>> Hi,
>>
>> How often does Kafka query zookeeper while producing and consuming?
>>
>> Ex:
>> If there is a single partition to which we produce and a HighLevel
>> consumer running on it, how many read/write queries to zookeeper happen.
>>
>> Extending further, multiple topics with ~100 partitions each, how many
>> zookeeper calls will be made (read/write).
>>
>> What is the max limit of no of partitions / kafka cluster that zookeeper
>> can handle?
>>
>> --
>> Regards
>> Vamsi Subhash
>>
>
>
>
>-- 
>Regards
>Vamsi Subhash

Re: Kafka Zookeeper queries

[Achanta Vamsi Subhash <ac...@flipkart.com>]

Hi,

Could anyone help with this?

Thanks.

On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:

> Hi,
>
> How often does Kafka query zookeeper while producing and consuming?
>
> Ex:
> If there is a single partition to which we produce and a HighLevel
> consumer running on it, how many read/write queries to zookeeper happen.
>
> Extending further, multiple topics with ~100 partitions each, how many
> zookeeper calls will be made (read/write).
>
> What is the max limit of no of partitions / kafka cluster that zookeeper
> can handle?
>
> --
> Regards
> Vamsi Subhash
>



-- 
Regards
Vamsi Subhash