You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (Created) (JIRA)" <ji...@apache.org> on 2011/11/25 13:19:40 UTC
[jira] [Created] (HAMA-480) Try out different barrier
implementations
Try out different barrier implementations
-----------------------------------------
Key: HAMA-480
URL: https://issues.apache.org/jira/browse/HAMA-480
Project: Hama
Issue Type: Improvement
Reporter: Thomas Jungblut
We should have a look at different barrier implementations with Zookeeper.
Have a look at the goldenorb stuff for example:
https://github.com/raveldata/goldenorb/blob/master/src/main/java/org/goldenorb/zookeeper/OrbFastBarrier.java
With out new synchronization service class this should be easily testable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-480) Try out different barrier
implementations
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158184#comment-13158184 ]
ChiaHung Lin commented on HAMA-480:
-----------------------------------
{quote}
Can we "emulate" this tree structure by using znodes in different levels?
For example if we can make a znode for a job and in the next level are the groom names and in the very next levels are the tasks
<pre>
root
/ job
/ groom_name_0
/ task0
/ task1
/ groom_name_1
/ task2
/ task3
</pre>
So we can simply sync on grooms in the tasks and they get notified once all grooms have sync'd theirselfs.
It this the idea behind the tree based sync?
{quote}
There should have no problem to simulate the above tree structure, which is similar to our implementation with double barrier, in zookeeper. For tree based barrier sync, it divides processes into subgroup and then synchronize among each other. Taken into an example of 8 processes, ranging from p0 to p7. At first stage, p1 sends message to p0 for sync; p3 to p2; p5 to p4; p7 to p6. At the seconds stage, p6 sends message to p4; p2 to p0. At the third stage, p4 sends message to p0 for reaching the barrier and then reverses notifying for leaving the barrier.
> Try out different barrier implementations
> -----------------------------------------
>
> Key: HAMA-480
> URL: https://issues.apache.org/jira/browse/HAMA-480
> Project: Hama
> Issue Type: Improvement
> Reporter: Thomas Jungblut
> Labels: bsp,, sync
>
> We should have a look at different barrier implementations with Zookeeper.
> Have a look at the goldenorb stuff for example:
> https://github.com/raveldata/goldenorb/blob/master/src/main/java/org/goldenorb/zookeeper/OrbFastBarrier.java
> With out new synchronization service class this should be easily testable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-480) Try out different barrier
implementations
Posted by "wolfgang hoschek (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247481#comment-13247481 ]
wolfgang hoschek commented on HAMA-480:
---------------------------------------
For example, a variety of scalable barrier implementation strategies (some of which may not applicable here) are referred to in "Related Work" and other sections of these papers:
http://www.ziti.uni-heidelberg.de/ziti/uploads/ce_group/2011-hipc.pdf
http://soft.vub.ac.be/~smarr/downloads/hpcc2010-marr-etal-insertion-tree-phasers.pdf
> Try out different barrier implementations
> -----------------------------------------
>
> Key: HAMA-480
> URL: https://issues.apache.org/jira/browse/HAMA-480
> Project: Hama
> Issue Type: Improvement
> Reporter: Thomas Jungblut
> Labels: bsp,, sync
>
> We should have a look at different barrier implementations with Zookeeper.
> Have a look at the goldenorb stuff for example:
> https://github.com/raveldata/goldenorb/blob/master/src/main/java/org/goldenorb/zookeeper/OrbFastBarrier.java
> With out new synchronization service class this should be easily testable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-480) Try out different barrier
implementations
Posted by "wolfgang hoschek (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247359#comment-13247359 ]
wolfgang hoschek commented on HAMA-480:
---------------------------------------
Could also have a look at netflix's barrier implementation using zookeeper:
https://github.com/Netflix/curator/blob/master/curator-recipes/src/main/java/com/netflix/curator/framework/recipes/barriers/DistributedBarrier.java
https://github.com/Netflix/curator/blob/master/curator-recipes/src/main/java/com/netflix/curator/framework/recipes/barriers/DistributedDoubleBarrier.java
It would be great if Hama could demonstrate a barrier impl that has low latency and scales to tens of thousands of peers. This would enable jobs with many short supersteps on large clusters.
> Try out different barrier implementations
> -----------------------------------------
>
> Key: HAMA-480
> URL: https://issues.apache.org/jira/browse/HAMA-480
> Project: Hama
> Issue Type: Improvement
> Reporter: Thomas Jungblut
> Labels: bsp,, sync
>
> We should have a look at different barrier implementations with Zookeeper.
> Have a look at the goldenorb stuff for example:
> https://github.com/raveldata/goldenorb/blob/master/src/main/java/org/goldenorb/zookeeper/OrbFastBarrier.java
> With out new synchronization service class this should be easily testable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-480) Try out different barrier
implementations
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247492#comment-13247492 ]
Thomas Jungblut commented on HAMA-480:
--------------------------------------
Thanks for the papers wolfgang!
We are currently having a double barrier implementation, looks quite like that of Curator. However we had some problems with zookeeper (and I think we still have) so the implementation is quite cluttered and can be improved.
> Try out different barrier implementations
> -----------------------------------------
>
> Key: HAMA-480
> URL: https://issues.apache.org/jira/browse/HAMA-480
> Project: Hama
> Issue Type: Improvement
> Reporter: Thomas Jungblut
> Labels: bsp,, sync
>
> We should have a look at different barrier implementations with Zookeeper.
> Have a look at the goldenorb stuff for example:
> https://github.com/raveldata/goldenorb/blob/master/src/main/java/org/goldenorb/zookeeper/OrbFastBarrier.java
> With out new synchronization service class this should be easily testable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira