You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2011/09/08 04:01:21 UTC
Distributed locks for global barrier synchronization
Hi,
I'm using Zookeeper for global barrier synchronization of Hama BSP
computing engine. Current implementation is based on 'ZooKeeper
Recipes and Solutions'[1] but there's a problem.
The problem is that, before the last process leaving the barrier
completely, other processors are starting to create their node[2]. So,
that last process hangs forever at "2. if no children, exit" step.
This problem intermittently occurs on high-performance environments.
Can anyone advise me?
1. http://zookeeper.apache.org/doc/trunk/recipes.html
2. https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Lock recipes and the lock path
Posted by Jordan Zimmerman <jz...@netflix.com>.
I gave it a vote - thanks.
On 9/20/11 12:44 PM, "Fournier, Camille F." <Ca...@gs.com>
wrote:
>Well, if you are locking member IDS that are generated strings that will
>never be reused, the best you can do right now is clean those up after a
>period of time.
>
>If, on the other hand, your member IDs are meaningful, it's likely you
>will want to reuse these locations in the future when you need to get
>another lock on this member. The overhead for a node in ZK is very very
>low, so unless you are generating a ton of these nodes that you then
>never use again and never clean up, it's not likely to be too much of a
>problem.
>
>We have discussed allowing "cleanup" type paths that will delete
>themselves when there are no child nodes. See:
>https://issues.apache.org/jira/browse/ZOOKEEPER-723
>
>C
RE: Lock recipes and the lock path
Posted by "Fournier, Camille F." <Ca...@gs.com>.
Well, if you are locking member IDS that are generated strings that will never be reused, the best you can do right now is clean those up after a period of time.
If, on the other hand, your member IDs are meaningful, it's likely you will want to reuse these locations in the future when you need to get another lock on this member. The overhead for a node in ZK is very very low, so unless you are generating a ton of these nodes that you then never use again and never clean up, it's not likely to be too much of a problem.
We have discussed allowing "cleanup" type paths that will delete themselves when there are no child nodes. See:
https://issues.apache.org/jira/browse/ZOOKEEPER-723
C
-----Original Message-----
From: Jordan Zimmerman [mailto:jzimmerman@netflix.com]
Sent: Tuesday, September 20, 2011 3:38 PM
To: user@zookeeper.apache.org
Subject: Lock recipes and the lock path
When writing the lock recipes for ZK, there is the potential for path
garbage to build up. For example, imagine you are writing a lock for some
kind go member ID. You'd have a base path for the locks (say
"/locks/members") and then append the member ID to get locks on it. Thus,
you'd end up with "/locks/members" building up paths, one for each member
ID.
How can you safely clean up the path for each member ID lock? There
doesn't seem to be an atomic way to do it. If one process is cleaning up
for example "/locks/members/12345", it can't be sure than another process
might need to get a lock on that member ID. I.e. There's never a safe
point to delete "/locals/members/12345".
-JZ
Re: Lock recipes and the lock path
Posted by Ted Dunning <te...@gmail.com>.
Multi ops might help here although there might be some difficult because all
we have is an existence check and not a non-existence check.
One way to hack that is to have each normal file operation be a multi that
looks like this:
create lock, modify file, delete lock
The lock will never be visible since it is created and deleted in a single
atomic operation. This does guarantee that the modification will never
succeed if the lock file already exists.
The cleanup can proceed like this:
create lock
get children
delete old files with large multi-op
delete lock
This should be pretty darned fast since only for round-trips are required.
On Tue, Sep 20, 2011 at 12:37 PM, Jordan Zimmerman
<jz...@netflix.com>wrote:
> How can you safely clean up the path for each member ID lock? There
> doesn't seem to be an atomic way to do it. If one process is cleaning up
> for example "/locks/members/12345", it can't be sure than another process
> might need to get a lock on that member ID. I.e. There's never a safe
> point to delete "/locals/members/12345".
>
Lock recipes and the lock path
Posted by Jordan Zimmerman <jz...@netflix.com>.
When writing the lock recipes for ZK, there is the potential for path
garbage to build up. For example, imagine you are writing a lock for some
kind go member ID. You'd have a base path for the locks (say
"/locks/members") and then append the member ID to get locks on it. Thus,
you'd end up with "/locks/members" building up paths, one for each member
ID.
How can you safely clean up the path for each member ID lock? There
doesn't seem to be an atomic way to do it. If one process is cleaning up
for example "/locks/members/12345", it can't be sure than another process
might need to get a lock on that member ID. I.e. There's never a safe
point to delete "/locals/members/12345".
-JZ
Re: Distributed locks for global barrier synchronization
Posted by Ted Dunning <te...@gmail.com>.
Conceptually, it might be good to have two barriers, one for the beginning
of a computation and one for the end. The first barrier is guaranteed not
to have the fast finish characteristic because all nodes are already in the
before-start state.
On Mon, Sep 19, 2011 at 6:04 PM, Edward J. Yoon <ed...@apache.org>wrote:
> Thanks.
>
> And, we found this - https://issues.apache.org/jira/browse/ZOOKEEPER-1011
>
> On Thu, Sep 8, 2011 at 3:06 PM, Ted Dunning <te...@gmail.com> wrote:
> > It might help to have a different znode for synchronization at each
> > iteration. That way, if slow nodes are still just getting around to
> > deleting the old node, the fast nodes creating their new nodes will not
> > interfere.
> >
> > On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> Hi,
> >>
> >> I'm using Zookeeper for global barrier synchronization of Hama BSP
> >> computing engine. Current implementation is based on 'ZooKeeper
> >> Recipes and Solutions'[1] but there's a problem.
> >>
> >> The problem is that, before the last process leaving the barrier
> >> completely, other processors are starting to create their node[2]. So,
> >> that last process hangs forever at "2. if no children, exit" step.
> >> This problem intermittently occurs on high-performance environments.
> >>
> >> Can anyone advise me?
> >>
> >> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
> >> 2.
> >>
> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Distributed locks for global barrier synchronization
Posted by Ted Dunning <te...@gmail.com>.
Conceptually, it might be good to have two barriers, one for the beginning
of a computation and one for the end. The first barrier is guaranteed not
to have the fast finish characteristic because all nodes are already in the
before-start state.
On Mon, Sep 19, 2011 at 6:04 PM, Edward J. Yoon <ed...@apache.org>wrote:
> Thanks.
>
> And, we found this - https://issues.apache.org/jira/browse/ZOOKEEPER-1011
>
> On Thu, Sep 8, 2011 at 3:06 PM, Ted Dunning <te...@gmail.com> wrote:
> > It might help to have a different znode for synchronization at each
> > iteration. That way, if slow nodes are still just getting around to
> > deleting the old node, the fast nodes creating their new nodes will not
> > interfere.
> >
> > On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> Hi,
> >>
> >> I'm using Zookeeper for global barrier synchronization of Hama BSP
> >> computing engine. Current implementation is based on 'ZooKeeper
> >> Recipes and Solutions'[1] but there's a problem.
> >>
> >> The problem is that, before the last process leaving the barrier
> >> completely, other processors are starting to create their node[2]. So,
> >> that last process hangs forever at "2. if no children, exit" step.
> >> This problem intermittently occurs on high-performance environments.
> >>
> >> Can anyone advise me?
> >>
> >> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
> >> 2.
> >>
> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Distributed locks for global barrier synchronization
Posted by "Edward J. Yoon" <ed...@apache.org>.
Thanks.
And, we found this - https://issues.apache.org/jira/browse/ZOOKEEPER-1011
On Thu, Sep 8, 2011 at 3:06 PM, Ted Dunning <te...@gmail.com> wrote:
> It might help to have a different znode for synchronization at each
> iteration. That way, if slow nodes are still just getting around to
> deleting the old node, the fast nodes creating their new nodes will not
> interfere.
>
> On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> Hi,
>>
>> I'm using Zookeeper for global barrier synchronization of Hama BSP
>> computing engine. Current implementation is based on 'ZooKeeper
>> Recipes and Solutions'[1] but there's a problem.
>>
>> The problem is that, before the last process leaving the barrier
>> completely, other processors are starting to create their node[2]. So,
>> that last process hangs forever at "2. if no children, exit" step.
>> This problem intermittently occurs on high-performance environments.
>>
>> Can anyone advise me?
>>
>> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
>> 2.
>> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Distributed locks for global barrier synchronization
Posted by "Edward J. Yoon" <ed...@apache.org>.
Thanks.
And, we found this - https://issues.apache.org/jira/browse/ZOOKEEPER-1011
On Thu, Sep 8, 2011 at 3:06 PM, Ted Dunning <te...@gmail.com> wrote:
> It might help to have a different znode for synchronization at each
> iteration. That way, if slow nodes are still just getting around to
> deleting the old node, the fast nodes creating their new nodes will not
> interfere.
>
> On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <ed...@apache.org>wrote:
>
>> Hi,
>>
>> I'm using Zookeeper for global barrier synchronization of Hama BSP
>> computing engine. Current implementation is based on 'ZooKeeper
>> Recipes and Solutions'[1] but there's a problem.
>>
>> The problem is that, before the last process leaving the barrier
>> completely, other processors are starting to create their node[2]. So,
>> that last process hangs forever at "2. if no children, exit" step.
>> This problem intermittently occurs on high-performance environments.
>>
>> Can anyone advise me?
>>
>> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
>> 2.
>> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Distributed locks for global barrier synchronization
Posted by Ted Dunning <te...@gmail.com>.
It might help to have a different znode for synchronization at each
iteration. That way, if slow nodes are still just getting around to
deleting the old node, the fast nodes creating their new nodes will not
interfere.
On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <ed...@apache.org>wrote:
> Hi,
>
> I'm using Zookeeper for global barrier synchronization of Hama BSP
> computing engine. Current implementation is based on 'ZooKeeper
> Recipes and Solutions'[1] but there's a problem.
>
> The problem is that, before the last process leaving the barrier
> completely, other processors are starting to create their node[2]. So,
> that last process hangs forever at "2. if no children, exit" step.
> This problem intermittently occurs on high-performance environments.
>
> Can anyone advise me?
>
> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
> 2.
> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Distributed locks for global barrier synchronization
Posted by Ted Dunning <te...@gmail.com>.
It might help to have a different znode for synchronization at each
iteration. That way, if slow nodes are still just getting around to
deleting the old node, the fast nodes creating their new nodes will not
interfere.
On Wed, Sep 7, 2011 at 7:01 PM, Edward J. Yoon <ed...@apache.org>wrote:
> Hi,
>
> I'm using Zookeeper for global barrier synchronization of Hama BSP
> computing engine. Current implementation is based on 'ZooKeeper
> Recipes and Solutions'[1] but there's a problem.
>
> The problem is that, before the last process leaving the barrier
> completely, other processors are starting to create their node[2]. So,
> that last process hangs forever at "2. if no children, exit" step.
> This problem intermittently occurs on high-performance environments.
>
> Can anyone advise me?
>
> 1. http://zookeeper.apache.org/doc/trunk/recipes.html
> 2.
> https://issues.apache.org/jira/browse/HAMA-387?focusedCommentId=13037785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13037785
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>