You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Botond Hejj <Bo...@MorganStanley.com> on 2011/12/12 18:06:09 UTC
leader election length
Hi ZooKeeper users,
I am playing currently with zookeeper and testing what happens if the
leader of an ensemble goes down.
I know that during the leader election zookeeper server won't reply to
any requests and if leader election takes a long time than existing
sessions might expire.
What I see in my tests that each server reads the last snapshot file
to get last zxid for leader election and when the leader is elected
than the leader reads the snapshot again before it syncs the
followers.
This means that the more data we store in zookeeper the longer it
takes to elect a new leader. This is also means as load of the
ensemble increases clients need bigger session timeout to "survive"
the loss of the leader.
Is it possible to do anything about this and have a fast leader
election even if the snapshot is big?
Regards,
Botond Hejj
Re: leader election length
Posted by Botond Hejj <Bo...@MorganStanley.com>.
Thanks Flavio for looking into this.
If the loadData can be removed from lead() that could be improve
leaderElection a lot on slow disk with high amount of data.
Botond
On Tue, Dec 13, 2011 at 14:02, Flavio Junqueira <fp...@yahoo-inc.com> wrote:
> Hi Botond, I'm under the impression that the leader only loads the database
> twice when it is bootstrapping. In QuorumPeer.start(), we call
> loadDataBase(). The second time corresponds to the call to zk.loadData in
> Leader.lead(). There could be a third time if zkDb.isInitialized() is false
> thought the call path lookForLeader()-> getInitLastLoggedZxid() ->
> getLastLoggedZxid(). I don't see how it could be false, though.
>
> After we get into the main QuorumPeer loop (in run()), we only call
> loadDataBase() through lookForLeader() upon calling getInitLastLoggedZxid()
> -> getLastLoggedZxid(). loadDataBase() is only executed if
> zkDb.isInitialized() is false, and I actually don't see a case in which it
> would be true once for the call coming from lookForLeader().
>
> The part I can't remember is why we need to call loadData() in Leader.lead()
> in the case that the data tree has been initialized already. If we are
> somehow skipping it when it has been initialized already, then I missed it.
> I'll keep looking into it...
>
> -Flavio
>
>
> On Dec 13, 2011, at 11:55 AM, Botond Hejj wrote:
>
>> Thanks Camille,
>>
>> Sorry, my assumption was wrong. I've made now a test and indeed the
>> session doesn't expire in this case.
>> This means than the only problem is that the service is down for an
>> increasing amount of time if snapshot is increasing. We use SAN to
>> store the snapshot and reading it back twice during leader election
>> can be a little slow (~30 secs). It is not bad at all but still have
>> anybody tried making the leader election faster in this case?
>>
>> Regards,
>> Botond
>>
>> On Mon, Dec 12, 2011 at 18:35, Camille Fournier <ca...@apache.org>
>> wrote:
>>>
>>> Existing sessions will not expire from the server side during election.
>>> Your client code may choose to close them on its end if you sit in a
>>> DISCONNECTED state for too long, but nothing should be expiring the
>>> sessions while quorum is not available.
>>>
>>> C
>>>
>>> On Mon, Dec 12, 2011 at 12:06 PM, Botond Hejj
>>> <Botond.Hejj@morganstanley.com
>>>>
>>>> wrote:
>>>
>>>
>>>> Hi ZooKeeper users,
>>>>
>>>> I am playing currently with zookeeper and testing what happens if the
>>>> leader of an ensemble goes down.
>>>> I know that during the leader election zookeeper server won't reply to
>>>> any requests and if leader election takes a long time than existing
>>>> sessions might expire.
>>>> What I see in my tests that each server reads the last snapshot file
>>>> to get last zxid for leader election and when the leader is elected
>>>> than the leader reads the snapshot again before it syncs the
>>>> followers.
>>>>
>>>> This means that the more data we store in zookeeper the longer it
>>>> takes to elect a new leader. This is also means as load of the
>>>> ensemble increases clients need bigger session timeout to "survive"
>>>> the loss of the leader.
>>>>
>>>> Is it possible to do anything about this and have a fast leader
>>>> election even if the snapshot is big?
>>>>
>>>> Regards,
>>>> Botond Hejj
>>>>
>>
>>
>>
>> --
>> Botond Hejj
>> Morgan Stanley | Technology
>> Lechner Odon fasor 8 | Floor 07
>> Budapest, 1095
>> Phone: +36 1 881-3962
>> Botond.Hejj@MorganStanley.com
>
>
> flavio
> junqueira
>
> research scientist
>
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300 fax (408) 349 3301
>
--
Botond Hejj
Morgan Stanley | Technology
Lechner Odon fasor 8 | Floor 07
Budapest, 1095
Phone: +36 1 881-3962
Botond.Hejj@MorganStanley.com
Re: leader election length
Posted by Thawan Kooburat <th...@fb.com>.
If your snapshot size wasn't big. You can increase the snapshot frequency
(Java prop: zookeeper.snapCount)
This may help a bit in reducing the size of txnlog, so it take less time
for the server to load the data on leader election.
However, taking more frequent snapshot cause the server to use more disk
bandwidth during normal operation. It may affect request latency if both
txnlog and snapshot shares the same disk.
--
Thawan Kooburat
On 6/13/13 1:35 PM, "ankitoshniwal" <an...@gmail.com> wrote:
>Hello,
>
>We are running into a similar issues. Is there recommended path to take in
>this scenario. We are running version 3.4.4 of zookeeper and the txlog on
>disk is close to 5 - 6 Gb. Currently the leader election sometimes takes
>up
>to 50 seconds.
>
>Any help is greatly appreciated.
>
>Thanks,
>Ankit.
>
>
>
>--
>View this message in context:
>http://zookeeper-user.578899.n2.nabble.com/leader-election-length-tp708686
>8p7578752.html
>Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: leader election length
Posted by ankitoshniwal <an...@gmail.com>.
Hello,
We are running into a similar issues. Is there recommended path to take in
this scenario. We are running version 3.4.4 of zookeeper and the txlog on
disk is close to 5 - 6 Gb. Currently the leader election sometimes takes up
to 50 seconds.
Any help is greatly appreciated.
Thanks,
Ankit.
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/leader-election-length-tp7086868p7578752.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: leader election length
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi Botond, I'm under the impression that the leader only loads the
database twice when it is bootstrapping. In QuorumPeer.start(), we
call loadDataBase(). The second time corresponds to the call to
zk.loadData in Leader.lead(). There could be a third time if
zkDb.isInitialized() is false thought the call path lookForLeader()->
getInitLastLoggedZxid() -> getLastLoggedZxid(). I don't see how it
could be false, though.
After we get into the main QuorumPeer loop (in run()), we only call
loadDataBase() through lookForLeader() upon calling
getInitLastLoggedZxid() -> getLastLoggedZxid(). loadDataBase() is only
executed if zkDb.isInitialized() is false, and I actually don't see a
case in which it would be true once for the call coming from
lookForLeader().
The part I can't remember is why we need to call loadData() in
Leader.lead() in the case that the data tree has been initialized
already. If we are somehow skipping it when it has been initialized
already, then I missed it. I'll keep looking into it...
-Flavio
On Dec 13, 2011, at 11:55 AM, Botond Hejj wrote:
> Thanks Camille,
>
> Sorry, my assumption was wrong. I've made now a test and indeed the
> session doesn't expire in this case.
> This means than the only problem is that the service is down for an
> increasing amount of time if snapshot is increasing. We use SAN to
> store the snapshot and reading it back twice during leader election
> can be a little slow (~30 secs). It is not bad at all but still have
> anybody tried making the leader election faster in this case?
>
> Regards,
> Botond
>
> On Mon, Dec 12, 2011 at 18:35, Camille Fournier <ca...@apache.org>
> wrote:
>> Existing sessions will not expire from the server side during
>> election.
>> Your client code may choose to close them on its end if you sit in a
>> DISCONNECTED state for too long, but nothing should be expiring the
>> sessions while quorum is not available.
>>
>> C
>>
>> On Mon, Dec 12, 2011 at 12:06 PM, Botond Hejj <Botond.Hejj@morganstanley.com
>>> wrote:
>>
>>> Hi ZooKeeper users,
>>>
>>> I am playing currently with zookeeper and testing what happens if
>>> the
>>> leader of an ensemble goes down.
>>> I know that during the leader election zookeeper server won't
>>> reply to
>>> any requests and if leader election takes a long time than existing
>>> sessions might expire.
>>> What I see in my tests that each server reads the last snapshot file
>>> to get last zxid for leader election and when the leader is elected
>>> than the leader reads the snapshot again before it syncs the
>>> followers.
>>>
>>> This means that the more data we store in zookeeper the longer it
>>> takes to elect a new leader. This is also means as load of the
>>> ensemble increases clients need bigger session timeout to "survive"
>>> the loss of the leader.
>>>
>>> Is it possible to do anything about this and have a fast leader
>>> election even if the snapshot is big?
>>>
>>> Regards,
>>> Botond Hejj
>>>
>
>
>
> --
> Botond Hejj
> Morgan Stanley | Technology
> Lechner Odon fasor 8 | Floor 07
> Budapest, 1095
> Phone: +36 1 881-3962
> Botond.Hejj@MorganStanley.com
flavio
junqueira
research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
Re: leader election length
Posted by Botond Hejj <Bo...@MorganStanley.com>.
Thanks Camille,
Sorry, my assumption was wrong. I've made now a test and indeed the
session doesn't expire in this case.
This means than the only problem is that the service is down for an
increasing amount of time if snapshot is increasing. We use SAN to
store the snapshot and reading it back twice during leader election
can be a little slow (~30 secs). It is not bad at all but still have
anybody tried making the leader election faster in this case?
Regards,
Botond
On Mon, Dec 12, 2011 at 18:35, Camille Fournier <ca...@apache.org> wrote:
> Existing sessions will not expire from the server side during election.
> Your client code may choose to close them on its end if you sit in a
> DISCONNECTED state for too long, but nothing should be expiring the
> sessions while quorum is not available.
>
> C
>
> On Mon, Dec 12, 2011 at 12:06 PM, Botond Hejj <Botond.Hejj@morganstanley.com
>> wrote:
>
>> Hi ZooKeeper users,
>>
>> I am playing currently with zookeeper and testing what happens if the
>> leader of an ensemble goes down.
>> I know that during the leader election zookeeper server won't reply to
>> any requests and if leader election takes a long time than existing
>> sessions might expire.
>> What I see in my tests that each server reads the last snapshot file
>> to get last zxid for leader election and when the leader is elected
>> than the leader reads the snapshot again before it syncs the
>> followers.
>>
>> This means that the more data we store in zookeeper the longer it
>> takes to elect a new leader. This is also means as load of the
>> ensemble increases clients need bigger session timeout to "survive"
>> the loss of the leader.
>>
>> Is it possible to do anything about this and have a fast leader
>> election even if the snapshot is big?
>>
>> Regards,
>> Botond Hejj
>>
--
Botond Hejj
Morgan Stanley | Technology
Lechner Odon fasor 8 | Floor 07
Budapest, 1095
Phone: +36 1 881-3962
Botond.Hejj@MorganStanley.com
Re: leader election length
Posted by Camille Fournier <ca...@apache.org>.
Existing sessions will not expire from the server side during election.
Your client code may choose to close them on its end if you sit in a
DISCONNECTED state for too long, but nothing should be expiring the
sessions while quorum is not available.
C
On Mon, Dec 12, 2011 at 12:06 PM, Botond Hejj <Botond.Hejj@morganstanley.com
> wrote:
> Hi ZooKeeper users,
>
> I am playing currently with zookeeper and testing what happens if the
> leader of an ensemble goes down.
> I know that during the leader election zookeeper server won't reply to
> any requests and if leader election takes a long time than existing
> sessions might expire.
> What I see in my tests that each server reads the last snapshot file
> to get last zxid for leader election and when the leader is elected
> than the leader reads the snapshot again before it syncs the
> followers.
>
> This means that the more data we store in zookeeper the longer it
> takes to elect a new leader. This is also means as load of the
> ensemble increases clients need bigger session timeout to "survive"
> the loss of the leader.
>
> Is it possible to do anything about this and have a fast leader
> election even if the snapshot is big?
>
> Regards,
> Botond Hejj
>