You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@curator.apache.org by chao chu <ch...@gmail.com> on 2014/03/10 15:38:45 UTC

Re: Leader Latch recovery after suspended state

Hi,

Just want to see if there is any progress on this?

I also have a related question about not only re-use the znode, but imho,
It would be great that LeaderLatch can survive from teomprary
ConnectionLossException (i.e., due to transient network issue).

I guess in most cases, the context switch due to leader re-election is
quite expensive, we might not want to do that just because of some
transient issue. if the current leader can re-connect within the session
timeout, it should still hold the leadership and no leader change would
happen during between. The similar rational like the differences between
ConnestionLossException (which is recoverable) and SessionExipredException
(which is not recoverable).

what are your thoughts on this? Thanks a lot!

Regards,

On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> Yes, I was suggesting how to patch Curator.
>
> On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:
>
> Currently this is not supported in the Curator library, but the Curator
> library (specifically leader latch's reset method) is the correct/logical
> place to add this feature if I want it?
>
>
> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> On reset() it could check to see if its node still exists. It would make
>> the code a lot more complicated though.
>>
>> -JZ
>>
>> On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:
>>
>> A leader latch enters the suspended state after failing to receive a
>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>> it is successful, it will enter the state reconnected.
>>
>> However, on reconnect, despite the fact the original node it created in
>> ZK is still there, it will create another ephemeral-sequential node (the
>> reset method is called). This means it will relinquish leadership, if there
>> is another machine with a latch in the same path.
>>
>> Is there any way to reconnect and reuse the original ZK node?
>>
>> Thanks!
>>
>>
>>
>
>

-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

sorry for the spam, to really add user@curator.apache.org this time.


On Mon, Mar 10, 2014 at 10:42 PM, chao chu <ch...@gmail.com> wrote:

> + user@curator.apache.org
>
> The original mail thread was updated in the incubator age :)
>
>
> On Mon, Mar 10, 2014 at 10:38 PM, chao chu <ch...@gmail.com> wrote:
>
>> Hi,
>>
>> Just want to see if there is any progress on this?
>>
>> I also have a related question about not only re-use the znode, but imho,
>> It would be great that LeaderLatch can survive from teomprary
>> ConnectionLossException (i.e., due to transient network issue).
>>
>> I guess in most cases, the context switch due to leader re-election is
>> quite expensive, we might not want to do that just because of some
>> transient issue. if the current leader can re-connect within the session
>> timeout, it should still hold the leadership and no leader change would
>> happen during between. The similar rational like the differences between
>> ConnestionLossException (which is recoverable) and SessionExipredException
>> (which is not recoverable).
>>
>> what are your thoughts on this? Thanks a lot!
>>
>> Regards,
>>
>>
>> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>> Yes, I was suggesting how to patch Curator.
>>>
>>> On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>
>>> Currently this is not supported in the Curator library, but the Curator
>>> library (specifically leader latch's reset method) is the correct/logical
>>> place to add this feature if I want it?
>>>
>>>
>>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>> On reset() it could check to see if its node still exists. It would
>>>> make the code a lot more complicated though.
>>>>
>>>> -JZ
>>>>
>>>> On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>>
>>>> A leader latch enters the suspended state after failing to receive a
>>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>>>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>>>> it is successful, it will enter the state reconnected.
>>>>
>>>> However, on reconnect, despite the fact the original node it created in
>>>> ZK is still there, it will create another ephemeral-sequential node (the
>>>> reset method is called). This means it will relinquish leadership, if there
>>>> is another machine with a latch in the same path.
>>>>
>>>> Is there any way to reconnect and reuse the original ZK node?
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> ChuChao
>>
>
>
>
> --
> ChuChao
>



-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

+ user@curator.apache.org

The original mail thread was updated in the incubator age :)


On Mon, Mar 10, 2014 at 10:38 PM, chao chu <ch...@gmail.com> wrote:

> Hi,
>
> Just want to see if there is any progress on this?
>
> I also have a related question about not only re-use the znode, but imho,
> It would be great that LeaderLatch can survive from teomprary
> ConnectionLossException (i.e., due to transient network issue).
>
> I guess in most cases, the context switch due to leader re-election is
> quite expensive, we might not want to do that just because of some
> transient issue. if the current leader can re-connect within the session
> timeout, it should still hold the leadership and no leader change would
> happen during between. The similar rational like the differences between
> ConnestionLossException (which is recoverable) and SessionExipredException
> (which is not recoverable).
>
> what are your thoughts on this? Thanks a lot!
>
> Regards,
>
>
> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> Yes, I was suggesting how to patch Curator.
>>
>> On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:
>>
>> Currently this is not supported in the Curator library, but the Curator
>> library (specifically leader latch's reset method) is the correct/logical
>> place to add this feature if I want it?
>>
>>
>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>> On reset() it could check to see if its node still exists. It would make
>>> the code a lot more complicated though.
>>>
>>> -JZ
>>>
>>> On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>
>>> A leader latch enters the suspended state after failing to receive a
>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>>> it is successful, it will enter the state reconnected.
>>>
>>> However, on reconnect, despite the fact the original node it created in
>>> ZK is still there, it will create another ephemeral-sequential node (the
>>> reset method is called). This means it will relinquish leadership, if there
>>> is another machine with a latch in the same path.
>>>
>>> Is there any way to reconnect and reuse the original ZK node?
>>>
>>> Thanks!
>>>
>>>
>>>
>>
>>
>
>
> --
> ChuChao
>



-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

Just in case anyone interested in this conversion might miss an important
fact not explicitly mentioned here: any EPHEMERAL nodes will be deleted on
session expire (guaranteed by ZK). This makes the 'fact' I mentioned above
reliable, I guess.


On Tue, Mar 11, 2014 at 12:00 AM, chao chu <ch...@gmail.com> wrote:

> Um, just my personal thoughts. As I understand, I think we are relying on
> the fact that: at any time, the latch who created the lowerest number znode
> on the leader election node path is determinted as the leader (this is
> exactly how the checkLeadership identify the current leader).
>
> Otherwise, at least for LeaderLatch, we lose the advantage to have the
> distinguish between ConnectionLossException and SessionExpiredExpection.
>
>
> On Mon, Mar 10, 2014 at 11:44 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  Sorry for being dense here, but my point is that: Suppose the leader
>> latch does NOT setLeadership(false) on receiving SUSPENDED, then what you
>> mentioned below won't happen, right?
>>
>> If the latch does't set false then it is lying. When there is a network
>> partition it is impossible to know the state of the cluster. The only safe
>> thing to do is assume that you are no longer leader (no longer hold the
>> lock, etc.).
>>
>> -JZ
>>
>
>
>
> --
> ChuChao
>



-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

*>>> Theoretically, you could assume you are still leader during SUSPENDED
if the session timeout hasn't elapsed.*

yep, this is exactly what I want to point out, so far, the only drawback I
can think of is that if the latch1 <--> zk1 connection loss was really
caused due to the client of latch1 (process crash, etc.), we have a time
window (up to session timeout value) where no leader can be used actually.

I will try to clone a LeaderLatch and adapt it to my expected behavior
mentioned for our use case. But I am not sure if you are fine to accept it
as an alternative and leave the choice up to the user? If it's 'yes',
what's your suggested way to do this? it's likely there will be a lot of
duplicate code if adding it as another leader election recipe. maybe we can
open a way to let the default 'listener' in leader latch can be overrided,
there we can override the 'handleStateChange'.

Thanks a lot for your time on this anyway :)

On Tue, Mar 11, 2014 at 12:08 AM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> Otherwise, at least for LeaderLatch, we lose the advantage to have the
> distinguish between ConnectionLossException and SessionExpiredExpection.
>
> Session expired is only generated when the client successfully reconnects
> to the cluster. It's not really a very useful bit of information.
> Theoretically, you could assume you are still leader during SUSPENDED if
> the session timeout hasn't elapsed. But, this is really trying to be too
> clever IMO.
>
> -JZ
>

-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Otherwise, at least for LeaderLatch, we lose the advantage to have the distinguish between ConnectionLossException and SessionExpiredExpection.
Session expired is only generated when the client successfully reconnects to the cluster. It’s not really a very useful bit of information. Theoretically, you could assume you are still leader during SUSPENDED if the session timeout hasn’t elapsed. But, this is really trying to be too clever IMO. 

-JZ

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

Um, just my personal thoughts. As I understand, I think we are relying on
the fact that: at any time, the latch who created the lowerest number znode
on the leader election node path is determinted as the leader (this is
exactly how the checkLeadership identify the current leader).

Otherwise, at least for LeaderLatch, we lose the advantage to have the
distinguish between ConnectionLossException and SessionExpiredExpection.

On Mon, Mar 10, 2014 at 11:44 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

>  Sorry for being dense here, but my point is that: Suppose the leader
> latch does NOT setLeadership(false) on receiving SUSPENDED, then what you
> mentioned below won't happen, right?
>
> If the latch does't set false then it is lying. When there is a network
> partition it is impossible to know the state of the cluster. The only safe
> thing to do is assume that you are no longer leader (no longer hold the
> lock, etc.).
>
> -JZ
>

-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Sorry for being dense here, but my point is that: Suppose the leader latch does NOT setLeadership(false) on receiving SUSPENDED, then what you mentioned below won't happen, right?

If the latch does’t set false then it is lying. When there is a network partition it is impossible to know the state of the cluster. The only safe thing to do is assume that you are no longer leader (no longer hold the lock, etc.). 

-JZ

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

Hi JZ,

Sorry for being dense here, but my point is that: Suppose the leader latch
does NOT setLeadership(false) on receiving SUSPENDED, then what you
mentioned below won't happen, right?

*>>> Further, if zk2/zk3 are communicating then they are in Quorum.
Therefore, latch2 will declare that it is leader*

this way we can avoid the unnecessary leader switch due to the transient
latch1 <--> zk1 connection loss (i.e., the reboot of zk1 somehome) if the
latch1 can re-connect to the zk ensemble in time.


On Mon, Mar 10, 2014 at 11:28 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> In step 3, during the time that zk1 is partitioned latch1 cannot know if
> it's leader or not. Further, if zk2/zk3 are communicating then they are in
> Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MUST
> go into a non-leader state. It doesn't matter if latch1 reconnects before
> the session expires. The issue is how it manages its state while it is in
> partition. To your point, latch1 could check to see if it's still leader
> when it reconnects but I'm not sure that there's a huge win here. Any
> clients of the latch will have to have handled the leadership loss during
> the partition.
>
> -JZ
>
>
> From: chao chu chuchao333@gmail.com
> Reply: user@curator.apache.org user@curator.apache.org
> Date: March 10, 2014 at 10:21:29 AM
> To: user user@curator.apache.org
>
> Subject:  Re: Leader Latch recovery after suspended state
>
>  Thanks for your reply first.
>
>  >>> Technically, this is already the case. When a RECONNECTED is
> received, LeaderLatch will attempt to regain leadership. The problem is
> that when there is a network partition there is no way to guarantee that
> you are still the leader. If there is Quorum in another segment of the
> cluster a new leader might be elected there.
>
>  I don't quite understand what you meant here. But let me explain the
> case I mentioned in detail:
>
> 1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two
> participants for the leader electoin {leader latch1, leader latch2}
> 2. latch1 is the current leader, and connected to zk1
> 3. the connection latch1 <--> zk1 broken somehow
> 4. but sooner (within the session timeout), latch1 re-connected (maybe to
> zk2)
>
> I guess the problem is that in LeaderLatch, it setLeadership(false)
> (meaning that there must be leader changes) as long as a SUSPEND state (or
> actually a ZK DISCONNECTED event).
>
> while in this case, ideally (just my personal thinking), since latch1
> re-connected in time, its znode will still be there, no others (latch2 in
> this example) will observe any events due to this, and once re-connected,
> if it detected that its znode still there, checkLeadership should still set
> it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be
> called) during this whole process. Thus, we can avoid the unnecessary
> leader switch (which, as I mentioned, can be very expensive in most of
> cases).
>
> does this make any sense to you? thanks
>
>
>
> On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  Please provide an implementation/fix and submit a pull request on
>> Github.
>>
>>   I also have a related question about not only re-use the znode, but
>> imho, It would be great that LeaderLatch can survive from teomprary
>> ConnectionLossException (i.e., due to transient network issue).
>>
>>  Technically, this is already the case. When a RECONNECTED is received,
>> LeaderLatch will attempt to regain leadership. The problem is that when
>> there is a network partition there is no way to guarantee that you are
>> still the leader. If there is Quorum in another segment of the cluster a
>> new leader might be elected there.
>>
>> -JZ
>>
>> From: chao chu chuchao333@gmail.com
>> Reply: user@curator.apache.org user@curator.apache.org
>> Date: March 10, 2014 at 9:39:50 AM
>> To: user@curator.incubator.apache.org user@curator.incubator.apache.org
>> Subject:  Re: Leader Latch recovery after suspended state
>>
>>   Hi,
>>
>> Just want to see if there is any progress on this?
>>
>> I also have a related question about not only re-use the znode, but imho,
>> It would be great that LeaderLatch can survive from teomprary
>> ConnectionLossException (i.e., due to transient network issue).
>>
>> I guess in most cases, the context switch due to leader re-election is
>> quite expensive, we might not want to do that just because of some
>> transient issue. if the current leader can re-connect within the session
>> timeout, it should still hold the leadership and no leader change would
>> happen during between. The similar rational like the differences between
>> ConnestionLossException (which is recoverable) and SessionExipredException
>> (which is not recoverable).
>>
>> what are your thoughts on this? Thanks a lot!
>>
>> Regards,
>>
>>
>> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  Yes, I was suggesting how to patch Curator.
>>>
>>>  On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>
>>>  Currently this is not supported in the Curator library, but the
>>> Curator library (specifically leader latch's reset method) is the
>>> correct/logical place to add this feature if I want it?
>>>
>>>
>>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>>  On reset() it could check to see if its node still exists. It would
>>>> make the code a lot more complicated though.
>>>>
>>>> -JZ
>>>>
>>>>  On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>>
>>>>  A leader latch enters the suspended state after failing to receive a
>>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>>>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>>>> it is successful, it will enter the state reconnected.
>>>>
>>>> However, on reconnect, despite the fact the original node it created in
>>>> ZK is still there, it will create another ephemeral-sequential node (the
>>>> reset method is called). This means it will relinquish leadership, if there
>>>> is another machine with a latch in the same path.
>>>>
>>>> Is there any way to reconnect and reuse the original ZK node?
>>>>
>>>>  Thanks!
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> ChuChao
>>
>>
>
>
> --
> ChuChao
>
>


-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

In step 3, during the time that zk1 is partitioned latch1 cannot know if it’s leader or not. Further, if zk2/zk3 are communicating then they are in Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MUST go into a non-leader state. It doesn’t matter if latch1 reconnects before the session expires. The issue is how it manages its state while it is in partition. To your point, latch1 could check to see if it’s still leader when it reconnects but I’m not sure that there’s a huge win here. Any clients of the latch will have to have handled the leadership loss during the partition.

-JZ

From: chao chu chuchao333@gmail.com
Reply: user@curator.apache.org user@curator.apache.org
Date: March 10, 2014 at 10:21:29 AM
To: user user@curator.apache.org
Subject: Re: Leader Latch recovery after suspended state

Thanks for your reply first.

>>> Technically, this is already the case. When a RECONNECTED is received, LeaderLatch will attempt to regain leadership. The problem is that when there is a network partition there is no way to guarantee that you are still the leader. If there is Quorum in another segment of the cluster a new leader might be elected there.

I don't quite understand what you meant here. But let me explain the case I mentioned in detail:

1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two participants for the leader electoin {leader latch1, leader latch2}
2. latch1 is the current leader, and connected to zk1
3. the connection latch1 <--> zk1 broken somehow
4. but sooner (within the session timeout), latch1 re-connected (maybe to zk2)

I guess the problem is that in LeaderLatch, it setLeadership(false) (meaning that there must be leader changes) as long as a SUSPEND state (or actually a ZK DISCONNECTED event).

while in this case, ideally (just my personal thinking), since latch1 re-connected in time, its znode will still be there, no others (latch2 in this example) will observe any events due to this, and once re-connected, if it detected that its znode still there, checkLeadership should still set it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be called) during this whole process. Thus, we can avoid the unnecessary leader switch (which, as I mentioned, can be very expensive in most of cases).

does this make any sense to you? thanks

On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
Please provide an implementation/fix and submit a pull request on Github.

I also have a related question about not only re-use the znode, but imho, It would be great that LeaderLatch can survive from teomprary ConnectionLossException (i.e., due to transient network issue).
Technically, this is already the case. When a RECONNECTED is received, LeaderLatch will attempt to regain leadership. The problem is that when there is a network partition there is no way to guarantee that you are still the leader. If there is Quorum in another segment of the cluster a new leader might be elected there.

-JZ

From: chao chu chuchao333@gmail.com
Reply: user@curator.apache.org user@curator.apache.org
Date: March 10, 2014 at 9:39:50 AM
To: user@curator.incubator.apache.org user@curator.incubator.apache.org
Subject: Re: Leader Latch recovery after suspended state

Hi,

Just want to see if there is any progress on this?

I also have a related question about not only re-use the znode, but imho, It would be great that LeaderLatch can survive from teomprary ConnectionLossException (i.e., due to transient network issue).

I guess in most cases, the context switch due to leader re-election is quite expensive, we might not want to do that just because of some transient issue. if the current leader can re-connect within the session timeout, it should still hold the leadership and no leader change would happen during between. The similar rational like the differences between ConnestionLossException (which is recoverable) and SessionExipredException (which is not recoverable).

what are your thoughts on this? Thanks a lot!

Regards,

On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
Yes, I was suggesting how to patch Curator.

On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:

Currently this is not supported in the Curator library, but the Curator library (specifically leader latch's reset method) is the correct/logical place to add this feature if I want it?

On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
On reset() it could check to see if its node still exists. It would make the code a lot more complicated though.

-JZ

On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:

A leader latch enters the suspended state after failing to receive a response from the first ZK machine it heartbeats to (takes 2 thirds of the timeout). For the last 1 third, it tries to contact another ZK machine. If it is successful, it will enter the state reconnected.

However, on reconnect, despite the fact the original node it created in ZK is still there, it will create another ephemeral-sequential node (the reset method is called). This means it will relinquish leadership, if there is another machine with a latch in the same path.

Is there any way to reconnect and reuse the original ZK node?

Thanks!

--
ChuChao

Re: Leader Latch recovery after suspended state

Posted by chao chu <ch...@gmail.com>.

Thanks for your reply first.

*>>> Technically, this is already the case. When a RECONNECTED is received,
LeaderLatch will attempt to regain leadership. The problem is that when
there is a network partition there is no way to guarantee that you are
still the leader. If there is Quorum in another segment of the cluster a
new leader might be elected there.*

I don't quite understand what you meant here. But let me explain the case I
mentioned in detail:

1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two
participants for the leader electoin {leader latch1, leader latch2}
2. latch1 is the current leader, and connected to zk1
3. the connection latch1 <--> zk1 broken somehow
4. but sooner (within the session timeout), latch1 re-connected (maybe to
zk2)

I guess the problem is that in LeaderLatch, it setLeadership(false)
(meaning that there must be leader changes) as long as a SUSPEND state (or
actually a ZK DISCONNECTED event).

while in this case, ideally (just my personal thinking), since latch1
re-connected in time, its znode will still be there, no others (latch2 in
this example) will observe any events due to this, and once re-connected,
if it detected that its znode still there, checkLeadership should still set
it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be
called) during this whole process. Thus, we can avoid the unnecessary
leader switch (which, as I mentioned, can be very expensive in most of
cases).

does this make any sense to you? thanks

On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> Please provide an implementation/fix and submit a pull request on Github.
>
> I also have a related question about not only re-use the znode, but imho,
> It would be great that LeaderLatch can survive from teomprary
> ConnectionLossException (i.e., due to transient network issue).
>
> Technically, this is already the case. When a RECONNECTED is received,
> LeaderLatch will attempt to regain leadership. The problem is that when
> there is a network partition there is no way to guarantee that you are
> still the leader. If there is Quorum in another segment of the cluster a
> new leader might be elected there.
>
> -JZ
>
> From: chao chu chuchao333@gmail.com
> Reply: user@curator.apache.org user@curator.apache.org
> Date: March 10, 2014 at 9:39:50 AM
> To: user@curator.incubator.apache.org user@curator.incubator.apache.org
> Subject:  Re: Leader Latch recovery after suspended state
>
>  Hi,
>
> Just want to see if there is any progress on this?
>
> I also have a related question about not only re-use the znode, but imho,
> It would be great that LeaderLatch can survive from teomprary
> ConnectionLossException (i.e., due to transient network issue).
>
> I guess in most cases, the context switch due to leader re-election is
> quite expensive, we might not want to do that just because of some
> transient issue. if the current leader can re-connect within the session
> timeout, it should still hold the leadership and no leader change would
> happen during between. The similar rational like the differences between
> ConnestionLossException (which is recoverable) and SessionExipredException
> (which is not recoverable).
>
> what are your thoughts on this? Thanks a lot!
>
> Regards,
>
>
> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  Yes, I was suggesting how to patch Curator.
>>
>>  On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:
>>
>>  Currently this is not supported in the Curator library, but the Curator
>> library (specifically leader latch's reset method) is the correct/logical
>> place to add this feature if I want it?
>>
>>
>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  On reset() it could check to see if its node still exists. It would
>>> make the code a lot more complicated though.
>>>
>>> -JZ
>>>
>>>  On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:
>>>
>>>  A leader latch enters the suspended state after failing to receive a
>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>>> it is successful, it will enter the state reconnected.
>>>
>>> However, on reconnect, despite the fact the original node it created in
>>> ZK is still there, it will create another ephemeral-sequential node (the
>>> reset method is called). This means it will relinquish leadership, if there
>>> is another machine with a latch in the same path.
>>>
>>> Is there any way to reconnect and reuse the original ZK node?
>>>
>>>  Thanks!
>>>
>>>
>>>
>>
>>
>
>
> --
> ChuChao
>
>

-- 
ChuChao

Re: Leader Latch recovery after suspended state

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Please provide an implementation/fix and submit a pull request on Github.

-JZ

Hi,

Just want to see if there is any progress on this?

I also have a related question about not only re-use the znode, but imho, It would be great that LeaderLatch can survive from teomprary ConnectionLossException (i.e., due to transient network issue).

what are your thoughts on this? Thanks a lot!

Regards,

On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
Yes, I was suggesting how to patch Curator.

On Aug 20, 2013, at 10:59 AM, Calvin Jia <ji...@gmail.com> wrote:

Currently this is not supported in the Curator library, but the Curator library (specifically leader latch's reset method) is the correct/logical place to add this feature if I want it?

On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
On reset() it could check to see if its node still exists. It would make the code a lot more complicated though.

-JZ

On Aug 20, 2013, at 10:25 AM, Calvin Jia <ji...@gmail.com> wrote:

Is there any way to reconnect and reuse the original ZK node?

Thanks!

--
ChuChao