You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (Created) (JIRA)" <ji...@apache.org> on 2011/10/21 07:46:33 UTC
[jira] [Created] (HAMA-461) Extract a Message Service from BSPPeer
Extract a Message Service from BSPPeer
--------------------------------------
Key: HAMA-461
URL: https://issues.apache.org/jira/browse/HAMA-461
Project: Hama
Issue Type: Improvement
Affects Versions: 0.3.0
Reporter: Thomas Jungblut
Fix For: 0.4.0
There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
So we should refactor our messaging system from the peer itself.
A hint in architecture could give us HAMA-457.
We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-461) Extract a Message Service from BSPPeer
Posted by "Thomas Jungblut (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut resolved HAMA-461.
----------------------------------
Resolution: Fixed
Committed. Thanks for your fast review Edward!
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch, HAMA-461_v2.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145258#comment-13145258 ]
Thomas Jungblut commented on HAMA-461:
--------------------------------------
I take a look at it. Thanks for testing.
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146252#comment-13146252 ]
Hudson commented on HAMA-461:
-----------------------------
Integrated in Hama-Nightly #346 (See [https://builds.apache.org/job/Hama-Nightly/346/])
[HAMA-461] extract message service
tjungblut :
Files :
* /incubator/hama/trunk/CHANGES.txt
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeer.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/LocalBSPRunner.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManager.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManagerFactory.java
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHadoopMessageManager.java
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch, HAMA-461_v2.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136170#comment-13136170 ]
Thomas Jungblut commented on HAMA-461:
--------------------------------------
Look at what I have seen:
https://issues.apache.org/jira/browse/HADOOP-7773
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Fix For: 0.4.0
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145336#comment-13145336 ]
Edward J. Yoon commented on HAMA-461:
-------------------------------------
Even though exception occurs, job is always success. We have to fix. :)
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch, HAMA-461_v2.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-461) Extract a Message Service from BSPPeer
Posted by "Thomas Jungblut (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut reassigned HAMA-461:
------------------------------------
Assignee: Thomas Jungblut
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145292#comment-13145292 ]
Edward J. Yoon commented on HAMA-461:
-------------------------------------
Patch looks great! +1
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch, HAMA-461_v2.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-461) Extract a Message Service from BSPPeer
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-461:
---------------------------------
Attachment: HAMA-461_v1.patch
Lots of refactorings. I'm still not sure if this is the final design.
But it is a way to go for our current RPC implementation.
I thought of extending this and offer a asynchronous messaging, but I'll do this on github.
I also revised the synchronization, only the queue for the next iteration which is used by the RPC server is synchronized now. So this should be a slightly bit faster now.
BSPPeer is now not a RPC server anymore.
I'm not sure how we can implement other protocols on top of it, but if we have the need, we can change this.
{noformat}
[INFO] Apache Hama parent POM ............................ SUCCESS [1.652s]
[INFO] Apache Hama Core .................................. SUCCESS [57.303s]
[INFO] Apache Hama Graph Package ......................... SUCCESS [0.771s]
[INFO] Apache Hama Examples .............................. SUCCESS [11.148s]
{noformat}
I hope the peer is now less complex.
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145181#comment-13145181 ]
Edward J. Yoon commented on HAMA-461:
-------------------------------------
Awesome.
BTW, I received below errors.
{code}
java.lang.NullPointerException
at org.apache.hadoop.net.NetUtils.getDefaultSocketFactory(NetUtils.java:89)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
at org.apache.hama.bsp.message.HadoopMessageManagerImpl.getBSPPeerConnection(HadoopMessageManagerImpl.java:133)
at org.apache.hama.bsp.message.HadoopMessageManagerImpl.transfer(HadoopMessageManagerImpl.java:145)
at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:244)
at org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:72)
at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:88)
at org.apache.hama.bsp.BSPTask.run(BSPTask.java:66)
at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:907)
11/11/07 10:26:47 INFO zookeeper.ZooKeeper: Session: 0x1337ba1e64d001d closed
{code}
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-461) Extract a Message Service from
BSPPeer
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132612#comment-13132612 ]
Thomas Jungblut commented on HAMA-461:
--------------------------------------
This could have multiple problems.
Hadoop's Writable interface is hacked in all of our classes.
So it would be quite hard for other protocols to deal with this.
Of course you can always open a ByteArrayOutputStream and set it as the DataOutput and get the real byte codes. But this sucks in performance.
Sadly there is no other way, but refactoring the messages.
I won't do that, because I changed it in the last release and it is currently quite useable. But I cannot remember why I used a marker interface "Messagable" and made an abstract class for a BSPMessage.
Instead it should be a MessagableWritable interface, and other services could then make a MessagableOtherFancyStuff interface. I hope you get the gist.
So within this task, I just refactor out the messaging we currently have in our BSPPeerImpl.
The overall design has to be revisted, though. I would suggest the next release.
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Fix For: 0.4.0
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-461) Extract a Message Service from BSPPeer
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-461:
---------------------------------
Attachment: HAMA-461_v2.patch
Should be fixed now, I forgot to call init and therefore the configuration was null.
> Extract a Message Service from BSPPeer
> --------------------------------------
>
> Key: HAMA-461
> URL: https://issues.apache.org/jira/browse/HAMA-461
> Project: Hama
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.4.0
>
> Attachments: HAMA-461_v1.patch, HAMA-461_v2.patch
>
>
> There's a problem, that we have more synchronized Collections than we need. localQueueForNextIteration (or similar name) is the only one which needs to be thread safe. At least only the put method could be synchronized, because reads does not need to be threadsafe.
> So we should refactor our messaging system from the peer itself.
> A hint in architecture could give us HAMA-457.
> We have to add a factory which let's the user choice their protocol.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira