You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Laxman <la...@huawei.com> on 2011/09/14 12:23:53 UTC
Impacts of increasing ZooKeeper ticktime
Hi Everyone,
Our customers are recently complaining about zookeeper client session
timeouts.
When analyzed its found that timeouts are due to heavy GC activity on
Clients.
So, they wanted to increase the session timeouts to 3 minutes which requires
the ticktime to be increased to atleast 9 seconds.
I wanted to know the impacts of increasing the ticktime to 10 seconds.
Following is my understanding of impacts. Correct me if my understanding is
wrong.
1) Leader election and syncing on startup may be delayed.
2) Watches on ephemeral nodes will get notifications only after session
timeout period.
Still I have some open questions w.r.to increasing ticktime.
1) Does this impact any performance?
2) Any other impacts?
3) Any other configurations to be tuned along with this?
--
Thanks,
Laxman
Re: Impacts of increasing ZooKeeper ticktime
Posted by Patrick Hunt <ph...@apache.org>.
On Wed, Sep 14, 2011 at 9:30 PM, Laxman <la...@huawei.com> wrote:
> Pat & Camille, Thanks for your response.
>
>> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
>> here:
>> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
>> iguration
>> (hbase has a similar issue, that's why we added this feature initially).
>
> I've tried this maxSessionTimeout. This satisfies my customer's requirement.
> I also agree that, we still need to fix the GC issue. This has been
> discussed with our customer and "maxSessionTimeout" is provided to unblock
> them temporarily.
>
>>It does mean that if you
>> have clients writing directly to the ZK client (instead of through a
>> wrapper you provide), you need to encourage people not to use MAX_INT in
>> their default client setup unless they need the long timeout.
>
> This is clear and in fact we are using the session timeout value of 30
> seconds only for many use cases like Namenode HA.
>
That's true (see Camille's response, I missed highlighting this in my
response, but it's absolutely the case).
> Thanks again for your quick response.
NP. Good Luck.
Patrick
>
>> -----Original Message-----
>> From: Fournier, Camille F. [mailto:Camille.Fournier@gs.com]
>> Sent: Wednesday, September 14, 2011 11:48 PM
>> To: 'dev@zookeeper.apache.org'; 'lakshman_ch@huawei.com'
>> Subject: RE: Impacts of increasing ZooKeeper ticktime
>>
>> Session timeout is negotiated though eh? So if only the clients that need
>> the long GC session expiration time set their client session timeout to
>> MAX_INT, other clients can keep it at a lower number and not have a
>> problem. That's how we have resolved the issue. It does mean that if you
>> have clients writing directly to the ZK client (instead of through a
>> wrapper you provide), you need to encourage people not to use MAX_INT in
>> their default client setup unless they need the long timeout.
>>
>> C
>>
>> -----Original Message-----
>> From: Patrick Hunt [mailto:phunt@apache.org]
>> Sent: Wednesday, September 14, 2011 1:44 PM
>> To: dev@zookeeper.apache.org; lakshman_ch@huawei.com
>> Subject: Re: Impacts of increasing ZooKeeper ticktime
>>
>> On Wed, Sep 14, 2011 at 3:23 AM, Laxman <la...@huawei.com> wrote:
>> > Our customers are recently complaining about zookeeper client session
>> > timeouts.
>> >
>> > When analyzed its found that timeouts are due to heavy GC activity on
>> > Clients.
>> >
>> > So, they wanted to increase the session timeouts to 3 minutes which
>> requires
>> > the ticktime to be increased to atleast 9 seconds.
>> >
>>
>> I assume you've talked to them about fixing their gc issue at some
>> point? Rather than band-aiding it? :-)
>>
>> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
>> here:
>> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
>> iguration
>> (hbase has a similar issue, that's why we added this feature initially).
>>
>> Really though they should fix the gc issue - setting the session
>> timeout higher means that their sessions will be expiring only after a
>> much longer time. Any reliance on this - such as for leader election
>> fail-over, will now take much more time. In your example it would take
>> 3 minutes for the other clients to notice when a client with the
>> particular session has become unavailable. Perhaps it doesn't matter
>> in your use case, but you are now likely to get complains about ZK not
>> being responsive. ;-)
>>
>> Regards,
>>
>> Patrick
>
>
RE: Impacts of increasing ZooKeeper ticktime
Posted by Laxman <la...@huawei.com>.
Pat & Camille, Thanks for your response.
> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
> here:
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
> iguration
> (hbase has a similar issue, that's why we added this feature initially).
I've tried this maxSessionTimeout. This satisfies my customer's requirement.
I also agree that, we still need to fix the GC issue. This has been
discussed with our customer and "maxSessionTimeout" is provided to unblock
them temporarily.
>It does mean that if you
> have clients writing directly to the ZK client (instead of through a
> wrapper you provide), you need to encourage people not to use MAX_INT in
> their default client setup unless they need the long timeout.
This is clear and in fact we are using the session timeout value of 30
seconds only for many use cases like Namenode HA.
Thanks again for your quick response.
> -----Original Message-----
> From: Fournier, Camille F. [mailto:Camille.Fournier@gs.com]
> Sent: Wednesday, September 14, 2011 11:48 PM
> To: 'dev@zookeeper.apache.org'; 'lakshman_ch@huawei.com'
> Subject: RE: Impacts of increasing ZooKeeper ticktime
>
> Session timeout is negotiated though eh? So if only the clients that need
> the long GC session expiration time set their client session timeout to
> MAX_INT, other clients can keep it at a lower number and not have a
> problem. That's how we have resolved the issue. It does mean that if you
> have clients writing directly to the ZK client (instead of through a
> wrapper you provide), you need to encourage people not to use MAX_INT in
> their default client setup unless they need the long timeout.
>
> C
>
> -----Original Message-----
> From: Patrick Hunt [mailto:phunt@apache.org]
> Sent: Wednesday, September 14, 2011 1:44 PM
> To: dev@zookeeper.apache.org; lakshman_ch@huawei.com
> Subject: Re: Impacts of increasing ZooKeeper ticktime
>
> On Wed, Sep 14, 2011 at 3:23 AM, Laxman <la...@huawei.com> wrote:
> > Our customers are recently complaining about zookeeper client session
> > timeouts.
> >
> > When analyzed its found that timeouts are due to heavy GC activity on
> > Clients.
> >
> > So, they wanted to increase the session timeouts to 3 minutes which
> requires
> > the ticktime to be increased to atleast 9 seconds.
> >
>
> I assume you've talked to them about fixing their gc issue at some
> point? Rather than band-aiding it? :-)
>
> I don't think you need to touch the tickTime here. See "maxSessionTimeout"
> here:
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConf
> iguration
> (hbase has a similar issue, that's why we added this feature initially).
>
> Really though they should fix the gc issue - setting the session
> timeout higher means that their sessions will be expiring only after a
> much longer time. Any reliance on this - such as for leader election
> fail-over, will now take much more time. In your example it would take
> 3 minutes for the other clients to notice when a client with the
> particular session has become unavailable. Perhaps it doesn't matter
> in your use case, but you are now likely to get complains about ZK not
> being responsive. ;-)
>
> Regards,
>
> Patrick
RE: Impacts of increasing ZooKeeper ticktime
Posted by "Fournier, Camille F." <Ca...@gs.com>.
Session timeout is negotiated though eh? So if only the clients that need the long GC session expiration time set their client session timeout to MAX_INT, other clients can keep it at a lower number and not have a problem. That's how we have resolved the issue. It does mean that if you have clients writing directly to the ZK client (instead of through a wrapper you provide), you need to encourage people not to use MAX_INT in their default client setup unless they need the long timeout.
C
-----Original Message-----
From: Patrick Hunt [mailto:phunt@apache.org]
Sent: Wednesday, September 14, 2011 1:44 PM
To: dev@zookeeper.apache.org; lakshman_ch@huawei.com
Subject: Re: Impacts of increasing ZooKeeper ticktime
On Wed, Sep 14, 2011 at 3:23 AM, Laxman <la...@huawei.com> wrote:
> Our customers are recently complaining about zookeeper client session
> timeouts.
>
> When analyzed its found that timeouts are due to heavy GC activity on
> Clients.
>
> So, they wanted to increase the session timeouts to 3 minutes which requires
> the ticktime to be increased to atleast 9 seconds.
>
I assume you've talked to them about fixing their gc issue at some
point? Rather than band-aiding it? :-)
I don't think you need to touch the tickTime here. See "maxSessionTimeout" here:
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConfiguration
(hbase has a similar issue, that's why we added this feature initially).
Really though they should fix the gc issue - setting the session
timeout higher means that their sessions will be expiring only after a
much longer time. Any reliance on this - such as for leader election
fail-over, will now take much more time. In your example it would take
3 minutes for the other clients to notice when a client with the
particular session has become unavailable. Perhaps it doesn't matter
in your use case, but you are now likely to get complains about ZK not
being responsive. ;-)
Regards,
Patrick
Re: Impacts of increasing ZooKeeper ticktime
Posted by Patrick Hunt <ph...@apache.org>.
On Wed, Sep 14, 2011 at 3:23 AM, Laxman <la...@huawei.com> wrote:
> Our customers are recently complaining about zookeeper client session
> timeouts.
>
> When analyzed its found that timeouts are due to heavy GC activity on
> Clients.
>
> So, they wanted to increase the session timeouts to 3 minutes which requires
> the ticktime to be increased to atleast 9 seconds.
>
I assume you've talked to them about fixing their gc issue at some
point? Rather than band-aiding it? :-)
I don't think you need to touch the tickTime here. See "maxSessionTimeout" here:
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConfiguration
(hbase has a similar issue, that's why we added this feature initially).
Really though they should fix the gc issue - setting the session
timeout higher means that their sessions will be expiring only after a
much longer time. Any reliance on this - such as for leader election
fail-over, will now take much more time. In your example it would take
3 minutes for the other clients to notice when a client with the
particular session has become unavailable. Perhaps it doesn't matter
in your use case, but you are now likely to get complains about ZK not
being responsive. ;-)
Regards,
Patrick