You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@curator.apache.org by Jozef Vilcek <jo...@gmail.com> on 2014/01/13 15:19:58 UTC

Persistent locks

Hi

I have a question about curator locks. I see that locks are implemented via
znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an
implementation via PERSISTENT_SEQUENTIAL.

Main reason for this are processes with critical sections, where we can not
afford to loose a lock due to session expiration. In such case, others
might acquire a lock and kick in while the previous process is still
running but e.g. experiencing connection issues. To kill this temporally
detached process in favor of others would be too costly.

My thoughts are to have:

* Persistent lock - if something go south and client code does not release
lock, it will stay there until removed by manual or some other intervention

* Persistent ephemeral lock - this would be ephemeral implemented by
persistent lock. For not so much critical stuff but e.g. for unstable
environments. This would install kind of refresh hook on a created lock
node. Other clients waiting to acquire lock could garbage collect locks
which does not received refresh for reasonable long time (scaling beyond
session timeouts).

What I would like to know:

* any wisdom, if this does make sense or if there is a better way out there
* support from curator ... There is a lot of good code in curator I would
have to copy to make this work. I want to avoid this. Would it be possible
of provide either path for making locks core extensible/reusable (or to
contribute implementation of locks if considered worth for framework) ?


Best,
Jozo

Re: Persistent locks

Posted by Jozef Vilcek <jo...@gmail.com>.

Patch proposal created at: https://issues.apache.org/jira/browse/CURATOR-84


On Fri, Jan 17, 2014 at 5:48 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> I’m not sure. You can give it a try. I can’t guarantee that we’ll accept
> the patch. That code is already very complicated. But, if it’s reasonable
> we’ll take it.
>
> -JZ
>
> ------------------------------
> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
> Reply: Jozef Vilcek jozo.vilcek@gmail.com
> Date: January 16, 2014 at 11:20:58 PM
>
> To: Jordan Zimmerman jordan@jordanzimmerman.com
> Subject:  Re: Persistent locks
>
> Jordan,
>
> so, do you think it is feasible to do such changes into Curator locks? Via
> changes into LockInternalsDriver? If yes, should I go ahead and create a
> JIRA and draft a patch?
>
> Best,
> Jozef
>
>
> On Wed, Jan 15, 2014 at 12:06 PM, Jozef Vilcek <jo...@gmail.com>wrote:
>
>> Yes. I was digging a bit in curator code and thinking about how can I
>> implement this. If I would be able to somehow intercept creation of lock
>> nodes, I could build GC/heartbeat logic around the lock by "installing"
>> appropriate hooks.
>>
>> One idea is that lock would accept a kind of LockNodeFactory, which would
>> be responsible for creating actual zookeeper nodes.
>> LockInternals.attemptLock() would delegate construction to the factory.
>> There, in my own factory, I could add posibility to listen to created lock
>> nodes and apply GC/heartbeat logic.
>>
>> As you suggested, same could be done if node creation would be moved to
>> the LockInternalsDriver and made public/reusable. I like this even better.
>> It could be tricky how to reuse e.g. read-write lock, but it would not be
>> so painful to "re-implement" only that part of curator code on my end.
>>
>>
>> Jozo
>>
>>
>> On Tue, Jan 14, 2014 at 6:51 PM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  You might be able to re-use the code in the LockInternals class. It
>>> could easily be modified to do what you want via the LockInternalsDriver.
>>> Of course it would need to be made public. The gc/heartbeat stuff would
>>> have to be coded fresh, though.
>>>
>>>  -Jordan
>>>
>>>   ------------------------------
>>> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
>>> Reply: Jozef Vilcek jozo.vilcek@gmail.com
>>> Date: January 14, 2014 at 12:18:15 AM
>>> To: Jordan Zimmerman jordan@jordanzimmerman.com
>>> Subject:  Re: Persistent locks
>>>
>>>  Yes, you are correct. I tried to sketch this in my post. There will be
>>> possibility to create persistent lock, which will cause deadlock if holder
>>> crashes and one with possibility to to be gc collected after some time if
>>> holder crashes. The mechanism for this has to be implemented.
>>>
>>> Off course that I use "classical" ephemeral locks where possible. But I
>>> have few cases where it is not feasible.
>>>
>>>
>>> On Tue, Jan 14, 2014 at 2:42 AM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>>  How will you avoid dead locks? You’ll need to write some kind of
>>>> heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if
>>>> the lock holder crashes.
>>>>
>>>>  -JZ
>>>>
>>>>  ------------------------------
>>>> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
>>>> Reply: user@curator.apache.org user@curator.apache.org
>>>> Date: January 13, 2014 at 7:24:06 AM
>>>> To: user@curator.apache.org user@curator.apache.org
>>>> Subject:  Persistent locks
>>>>
>>>>   Hi
>>>>
>>>> I have a question about curator locks. I see that locks are implemented
>>>> via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an
>>>> implementation via PERSISTENT_SEQUENTIAL.
>>>>
>>>> Main reason for this are processes with critical sections, where we can
>>>> not afford to loose a lock due to session expiration. In such case, others
>>>> might acquire a lock and kick in while the previous process is still
>>>> running but e.g. experiencing connection issues. To kill this temporally
>>>> detached process in favor of others would be too costly.
>>>>
>>>> My thoughts are to have:
>>>>
>>>> * Persistent lock - if something go south and client code does not
>>>> release lock, it will stay there until removed by manual or some other
>>>> intervention
>>>>
>>>> * Persistent ephemeral lock - this would be ephemeral implemented by
>>>> persistent lock. For not so much critical stuff but e.g. for unstable
>>>> environments. This would install kind of refresh hook on a created lock
>>>> node. Other clients waiting to acquire lock could garbage collect locks
>>>> which does not received refresh for reasonable long time (scaling beyond
>>>> session timeouts).
>>>>
>>>> What I would like to know:
>>>>
>>>> * any wisdom, if this does make sense or if there is a better way out
>>>> there
>>>> * support from curator ... There is a lot of good code in curator I
>>>> would have to copy to make this work. I want to avoid this. Would it be
>>>> possible of provide either path for making locks core extensible/reusable
>>>> (or to contribute implementation of locks if considered worth for
>>>> framework) ?
>>>>
>>>>
>>>> Best,
>>>> Jozo
>>>>
>>>>
>>>
>>
>

Re: Persistent locks

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

I’m not sure. You can give it a try. I can’t guarantee that we’ll accept the patch. That code is already very complicated. But, if it’s reasonable we’ll take it.

-JZ

From: Jozef Vilcek Jozef Vilcek
Reply: Jozef Vilcek jozo.vilcek@gmail.com
Date: January 16, 2014 at 11:20:58 PM
To: Jordan Zimmerman jordan@jordanzimmerman.com
Subject: Re: Persistent locks
Jordan,

so, do you think it is feasible to do such changes into Curator locks? Via changes into LockInternalsDriver? If yes, should I go ahead and create a JIRA and draft a patch?

Best,
Jozef

On Wed, Jan 15, 2014 at 12:06 PM, Jozef Vilcek <jo...@gmail.com> wrote:
Yes. I was digging a bit in curator code and thinking about how can I implement this. If I would be able to somehow intercept creation of lock nodes, I could build GC/heartbeat logic around the lock by "installing" appropriate hooks.

One idea is that lock would accept a kind of LockNodeFactory, which would be responsible for creating actual zookeeper nodes. LockInternals.attemptLock() would delegate construction to the factory. There, in my own factory, I could add posibility to listen to created lock nodes and apply GC/heartbeat logic.

As you suggested, same could be done if node creation would be moved to the LockInternalsDriver and made public/reusable. I like this even better. It could be tricky how to reuse e.g. read-write lock, but it would not be so painful to "re-implement" only that part of curator code on my end.

Jozo

On Tue, Jan 14, 2014 at 6:51 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
You might be able to re-use the code in the LockInternals class. It could easily be modified to do what you want via the LockInternalsDriver. Of course it would need to be made public. The gc/heartbeat stuff would have to be coded fresh, though.

-Jordan

From: Jozef Vilcek Jozef Vilcek
Reply: Jozef Vilcek jozo.vilcek@gmail.com
Date: January 14, 2014 at 12:18:15 AM
To: Jordan Zimmerman jordan@jordanzimmerman.com
Subject: Re: Persistent locks
Yes, you are correct. I tried to sketch this in my post. There will be possibility to create persistent lock, which will cause deadlock if holder crashes and one with possibility to to be gc collected after some time if holder crashes. The mechanism for this has to be implemented.

Off course that I use "classical" ephemeral locks where possible. But I have few cases where it is not feasible.

On Tue, Jan 14, 2014 at 2:42 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
How will you avoid dead locks? You’ll need to write some kind of heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if the lock holder crashes.

-JZ

From: Jozef Vilcek Jozef Vilcek
Reply: user@curator.apache.org user@curator.apache.org
Date: January 13, 2014 at 7:24:06 AM
To: user@curator.apache.org user@curator.apache.org
Subject: Persistent locks
Hi

I have a question about curator locks. I see that locks are implemented via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an implementation via PERSISTENT_SEQUENTIAL.

Main reason for this are processes with critical sections, where we can not afford to loose a lock due to session expiration. In such case, others might acquire a lock and kick in while the previous process is still running but e.g. experiencing connection issues. To kill this temporally detached process in favor of others would be too costly.

My thoughts are to have:

* Persistent lock - if something go south and client code does not release lock, it will stay there until removed by manual or some other intervention

* Persistent ephemeral lock - this would be ephemeral implemented by persistent lock. For not so much critical stuff but e.g. for unstable environments. This would install kind of refresh hook on a created lock node. Other clients waiting to acquire lock could garbage collect locks which does not received refresh for reasonable long time (scaling beyond session timeouts).

What I would like to know:

* any wisdom, if this does make sense or if there is a better way out there
* support from curator ... There is a lot of good code in curator I would have to copy to make this work. I want to avoid this. Would it be possible of provide either path for making locks core extensible/reusable (or to contribute implementation of locks if considered worth for framework) ?

Best,
Jozo

Re: Persistent locks

Posted by Jozef Vilcek <jo...@gmail.com>.

Jordan,

so, do you think it is feasible to do such changes into Curator locks? Via
changes into LockInternalsDriver? If yes, should I go ahead and create a
JIRA and draft a patch?

Best,
Jozef


On Wed, Jan 15, 2014 at 12:06 PM, Jozef Vilcek <jo...@gmail.com>wrote:

> Yes. I was digging a bit in curator code and thinking about how can I
> implement this. If I would be able to somehow intercept creation of lock
> nodes, I could build GC/heartbeat logic around the lock by "installing"
> appropriate hooks.
>
> One idea is that lock would accept a kind of LockNodeFactory, which would
> be responsible for creating actual zookeeper nodes.
> LockInternals.attemptLock() would delegate construction to the factory.
> There, in my own factory, I could add posibility to listen to created lock
> nodes and apply GC/heartbeat logic.
>
> As you suggested, same could be done if node creation would be moved to
> the LockInternalsDriver and made public/reusable. I like this even better.
> It could be tricky how to reuse e.g. read-write lock, but it would not be
> so painful to "re-implement" only that part of curator code on my end.
>
>
> Jozo
>
>
> On Tue, Jan 14, 2014 at 6:51 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> You might be able to re-use the code in the LockInternals class. It could
>> easily be modified to do what you want via the LockInternalsDriver. Of
>> course it would need to be made public. The gc/heartbeat stuff would have
>> to be coded fresh, though.
>>
>> -Jordan
>>
>> ------------------------------
>> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
>> Reply: Jozef Vilcek jozo.vilcek@gmail.com
>> Date: January 14, 2014 at 12:18:15 AM
>> To: Jordan Zimmerman jordan@jordanzimmerman.com
>> Subject:  Re: Persistent locks
>>
>> Yes, you are correct. I tried to sketch this in my post. There will be
>> possibility to create persistent lock, which will cause deadlock if holder
>> crashes and one with possibility to to be gc collected after some time if
>> holder crashes. The mechanism for this has to be implemented.
>>
>> Off course that I use "classical" ephemeral locks where possible. But I
>> have few cases where it is not feasible.
>>
>>
>> On Tue, Jan 14, 2014 at 2:42 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  How will you avoid dead locks? You’ll need to write some kind of
>>> heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if
>>> the lock holder crashes.
>>>
>>>  -JZ
>>>
>>>  ------------------------------
>>> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
>>> Reply: user@curator.apache.org user@curator.apache.org
>>> Date: January 13, 2014 at 7:24:06 AM
>>> To: user@curator.apache.org user@curator.apache.org
>>> Subject:  Persistent locks
>>>
>>>   Hi
>>>
>>> I have a question about curator locks. I see that locks are implemented
>>> via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an
>>> implementation via PERSISTENT_SEQUENTIAL.
>>>
>>> Main reason for this are processes with critical sections, where we can
>>> not afford to loose a lock due to session expiration. In such case, others
>>> might acquire a lock and kick in while the previous process is still
>>> running but e.g. experiencing connection issues. To kill this temporally
>>> detached process in favor of others would be too costly.
>>>
>>> My thoughts are to have:
>>>
>>> * Persistent lock - if something go south and client code does not
>>> release lock, it will stay there until removed by manual or some other
>>> intervention
>>>
>>> * Persistent ephemeral lock - this would be ephemeral implemented by
>>> persistent lock. For not so much critical stuff but e.g. for unstable
>>> environments. This would install kind of refresh hook on a created lock
>>> node. Other clients waiting to acquire lock could garbage collect locks
>>> which does not received refresh for reasonable long time (scaling beyond
>>> session timeouts).
>>>
>>> What I would like to know:
>>>
>>> * any wisdom, if this does make sense or if there is a better way out
>>> there
>>> * support from curator ... There is a lot of good code in curator I
>>> would have to copy to make this work. I want to avoid this. Would it be
>>> possible of provide either path for making locks core extensible/reusable
>>> (or to contribute implementation of locks if considered worth for
>>> framework) ?
>>>
>>>
>>> Best,
>>> Jozo
>>>
>>>
>>
>

Re: Persistent locks

Posted by Jozef Vilcek <jo...@gmail.com>.

Yes. I was digging a bit in curator code and thinking about how can I
implement this. If I would be able to somehow intercept creation of lock
nodes, I could build GC/heartbeat logic around the lock by "installing"
appropriate hooks.

One idea is that lock would accept a kind of LockNodeFactory, which would
be responsible for creating actual zookeeper nodes.
LockInternals.attemptLock() would delegate construction to the factory.
There, in my own factory, I could add posibility to listen to created lock
nodes and apply GC/heartbeat logic.

As you suggested, same could be done if node creation would be moved to the
LockInternalsDriver and made public/reusable. I like this even better. It
could be tricky how to reuse e.g. read-write lock, but it would not be so
painful to "re-implement" only that part of curator code on my end.


Jozo


On Tue, Jan 14, 2014 at 6:51 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> You might be able to re-use the code in the LockInternals class. It could
> easily be modified to do what you want via the LockInternalsDriver. Of
> course it would need to be made public. The gc/heartbeat stuff would have
> to be coded fresh, though.
>
> -Jordan
>
> ------------------------------
> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
> Reply: Jozef Vilcek jozo.vilcek@gmail.com
> Date: January 14, 2014 at 12:18:15 AM
> To: Jordan Zimmerman jordan@jordanzimmerman.com
> Subject:  Re: Persistent locks
>
> Yes, you are correct. I tried to sketch this in my post. There will be
> possibility to create persistent lock, which will cause deadlock if holder
> crashes and one with possibility to to be gc collected after some time if
> holder crashes. The mechanism for this has to be implemented.
>
> Off course that I use "classical" ephemeral locks where possible. But I
> have few cases where it is not feasible.
>
>
> On Tue, Jan 14, 2014 at 2:42 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  How will you avoid dead locks? You’ll need to write some kind of
>> heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if
>> the lock holder crashes.
>>
>>  -JZ
>>
>>  ------------------------------
>> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
>> Reply: user@curator.apache.org user@curator.apache.org
>> Date: January 13, 2014 at 7:24:06 AM
>> To: user@curator.apache.org user@curator.apache.org
>> Subject:  Persistent locks
>>
>>   Hi
>>
>> I have a question about curator locks. I see that locks are implemented
>> via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an
>> implementation via PERSISTENT_SEQUENTIAL.
>>
>> Main reason for this are processes with critical sections, where we can
>> not afford to loose a lock due to session expiration. In such case, others
>> might acquire a lock and kick in while the previous process is still
>> running but e.g. experiencing connection issues. To kill this temporally
>> detached process in favor of others would be too costly.
>>
>> My thoughts are to have:
>>
>> * Persistent lock - if something go south and client code does not
>> release lock, it will stay there until removed by manual or some other
>> intervention
>>
>> * Persistent ephemeral lock - this would be ephemeral implemented by
>> persistent lock. For not so much critical stuff but e.g. for unstable
>> environments. This would install kind of refresh hook on a created lock
>> node. Other clients waiting to acquire lock could garbage collect locks
>> which does not received refresh for reasonable long time (scaling beyond
>> session timeouts).
>>
>> What I would like to know:
>>
>> * any wisdom, if this does make sense or if there is a better way out
>> there
>> * support from curator ... There is a lot of good code in curator I would
>> have to copy to make this work. I want to avoid this. Would it be possible
>> of provide either path for making locks core extensible/reusable (or to
>> contribute implementation of locks if considered worth for framework) ?
>>
>>
>> Best,
>> Jozo
>>
>>
>

Re: Persistent locks

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

You might be able to re-use the code in the LockInternals class. It could easily be modified to do what you want via the LockInternalsDriver. Of course it would need to be made public. The gc/heartbeat stuff would have to be coded fresh, though.

-Jordan

Off course that I use "classical" ephemeral locks where possible. But I have few cases where it is not feasible.

-JZ

I have a question about curator locks. I see that locks are implemented via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an implementation via PERSISTENT_SEQUENTIAL.

My thoughts are to have:

* Persistent lock - if something go south and client code does not release lock, it will stay there until removed by manual or some other intervention

What I would like to know:

Best,
Jozo

Re: Persistent locks

Posted by Jozef Vilcek <jo...@gmail.com>.

Yes, you are correct. I tried to sketch this in my post. There will be
possibility to create persistent lock, which will cause deadlock if holder
crashes and one with possibility to to be gc collected after some time if
holder crashes. The mechanism for this has to be implemented.

Off course that I use "classical" ephemeral locks where possible. But I
have few cases where it is not feasible.


On Tue, Jan 14, 2014 at 2:42 AM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> How will you avoid dead locks? You’ll need to write some kind of
> heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if
> the lock holder crashes.
>
> -JZ
>
> ------------------------------
> From: Jozef Vilcek Jozef Vilcek <jo...@gmail.com>
> Reply: user@curator.apache.org user@curator.apache.org
> Date: January 13, 2014 at 7:24:06 AM
> To: user@curator.apache.org user@curator.apache.org
> Subject:  Persistent locks
>
>  Hi
>
> I have a question about curator locks. I see that locks are implemented
> via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an
> implementation via PERSISTENT_SEQUENTIAL.
>
> Main reason for this are processes with critical sections, where we can
> not afford to loose a lock due to session expiration. In such case, others
> might acquire a lock and kick in while the previous process is still
> running but e.g. experiencing connection issues. To kill this temporally
> detached process in favor of others would be too costly.
>
> My thoughts are to have:
>
> * Persistent lock - if something go south and client code does not release
> lock, it will stay there until removed by manual or some other intervention
>
> * Persistent ephemeral lock - this would be ephemeral implemented by
> persistent lock. For not so much critical stuff but e.g. for unstable
> environments. This would install kind of refresh hook on a created lock
> node. Other clients waiting to acquire lock could garbage collect locks
> which does not received refresh for reasonable long time (scaling beyond
> session timeouts).
>
> What I would like to know:
>
> * any wisdom, if this does make sense or if there is a better way out there
> * support from curator ... There is a lot of good code in curator I would
> have to copy to make this work. I want to avoid this. Would it be possible
> of provide either path for making locks core extensible/reusable (or to
> contribute implementation of locks if considered worth for framework) ?
>
>
> Best,
> Jozo
>
>

Re: Persistent locks

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

How will you avoid dead locks? You’ll need to write some kind of heartbeat/gc mechanism to the lock. Otherwise I don’t see how it works if the lock holder crashes.

-JZ

I have a question about curator locks. I see that locks are implemented via znode type EPHEMERAL_SEQUENTIAL. I am thinking about having an implementation via PERSISTENT_SEQUENTIAL.

My thoughts are to have:

* Persistent lock - if something go south and client code does not release lock, it will stay there until removed by manual or some other intervention

What I would like to know:

Best,
Jozo