You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2012/10/18 06:20:03 UTC
[jira] [Created] (HAMA-654) JDK 7 and SDP protocol
Edward J. Yoon created HAMA-654:
-----------------------------------
Summary: JDK 7 and SDP protocol
Key: HAMA-654
URL: https://issues.apache.org/jira/browse/HAMA-654
Project: Hama
Issue Type: Task
Reporter: Edward J. Yoon
Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482150#comment-13482150 ]
Edward J. Yoon commented on HAMA-654:
-------------------------------------
Yes I'll.
By the way, SDP protocol is much more slower than 10G. I need to more investigate :/
{code}
SDP enabled:
bondib0 Link encap:InfiniBand HWaddr 80:00:00:4A:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.1 Bcast:192.168.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:781625 errors:0 dropped:0 overruns:0 frame:0
TX packets:726252 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:682192811 (650.5 MiB) TX bytes:841621558 (802.6 MiB)
CONNECT to 192.168.10.5:61005 (socket converted to SDP protocol)
CONNECT to 192.168.10.6:61004 (socket converted to SDP protocol)
CONNECT to 192.168.10.9:61001 (socket converted to SDP protocol)
CONNECT to 192.168.10.2:61008 (socket converted to SDP protocol)
CONNECT to 192.168.10.7:61003 (socket converted to SDP protocol)
CONNECT to 192.168.10.13:61008 (socket converted to SDP protocol)
CONNECT to 192.168.10.16:61005 (socket converted to SDP protocol)
----
[oracle@bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar pagerank /user/oracle/edward/web-Google.txt edward/testout
12/10/22 23:07:39 INFO bsp.FileInputFormat: Total input paths to process : 1
12/10/22 23:07:39 INFO bsp.FileInputFormat: numSplits: 144
12/10/22 23:07:39 INFO bsp.FileInputFormat: computeSplitSize: 523473 (523473, 1, 268435456)
12/10/22 23:07:39 INFO bsp.FileInputFormat: Total # of splits: 144
12/10/22 23:07:40 INFO bsp.BSPJobClient: Running job: job_201210222307_0001
12/10/22 23:07:43 INFO bsp.BSPJobClient: Current supersteps number: 0
12/10/22 23:07:49 INFO bsp.BSPJobClient: Current supersteps number: 1
12/10/22 23:08:04 INFO bsp.BSPJobClient: Current supersteps number: 2
12/10/22 23:08:22 INFO bsp.BSPJobClient: Current supersteps number: 3
12/10/22 23:08:37 INFO bsp.BSPJobClient: Current supersteps number: 4
12/10/22 23:08:55 INFO bsp.BSPJobClient: Current supersteps number: 5
12/10/22 23:09:13 INFO bsp.BSPJobClient: Current supersteps number: 6
12/10/22 23:09:28 INFO bsp.BSPJobClient: Current supersteps number: 7
12/10/22 23:09:43 INFO bsp.BSPJobClient: Current supersteps number: 8
12/10/22 23:09:58 INFO bsp.BSPJobClient: Current supersteps number: 9
12/10/22 23:10:01 INFO bsp.BSPJobClient: Current supersteps number: 10
12/10/22 23:10:16 INFO bsp.BSPJobClient: Current supersteps number: 11
12/10/22 23:10:16 INFO bsp.BSPJobClient: The total number of supersteps: 11
12/10/22 23:10:16 INFO bsp.BSPJobClient: Counters: 8
12/10/22 23:10:16 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/10/22 23:10:16 INFO bsp.BSPJobClient: LAUNCHED_TASKS=144
12/10/22 23:10:16 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/10/22 23:10:16 INFO bsp.BSPJobClient: SUPERSTEPS=11
12/10/22 23:10:16 INFO bsp.BSPJobClient: SUPERSTEP_SUM=1584
12/10/22 23:10:16 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=9343435
12/10/22 23:10:16 INFO bsp.BSPJobClient: IO_BYTES_READ=75380115
12/10/22 23:10:16 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=14690880
12/10/22 23:10:16 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=5105043
12/10/22 23:10:16 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=7345440
Job Finished in 156.749 seconds
SDP disabled:
[oracle@bda01 hama-0.6.0-SNAPSHOT]$ bin/hama jar hama-examples-0.6.0-SNAPSHOT.jar pagerank /user/oracle/edward/web-Google.txt edward/testout
12/10/22 23:11:07 INFO bsp.FileInputFormat: Total input paths to process : 1
12/10/22 23:11:07 INFO bsp.FileInputFormat: numSplits: 144
12/10/22 23:11:07 INFO bsp.FileInputFormat: computeSplitSize: 523473 (523473, 1, 268435456)
12/10/22 23:11:07 INFO bsp.FileInputFormat: Total # of splits: 144
12/10/22 23:11:08 INFO bsp.BSPJobClient: Running job: job_201210222311_0001
12/10/22 23:11:11 INFO bsp.BSPJobClient: Current supersteps number: 0
12/10/22 23:11:17 INFO bsp.BSPJobClient: Current supersteps number: 1
12/10/22 23:11:20 INFO bsp.BSPJobClient: Current supersteps number: 4
12/10/22 23:11:23 INFO bsp.BSPJobClient: Current supersteps number: 8
12/10/22 23:11:26 INFO bsp.BSPJobClient: Current supersteps number: 11
12/10/22 23:11:26 INFO bsp.BSPJobClient: The total number of supersteps: 11
12/10/22 23:11:26 INFO bsp.BSPJobClient: Counters: 8
12/10/22 23:11:26 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
12/10/22 23:11:26 INFO bsp.BSPJobClient: LAUNCHED_TASKS=144
12/10/22 23:11:26 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
12/10/22 23:11:26 INFO bsp.BSPJobClient: SUPERSTEPS=11
12/10/22 23:11:26 INFO bsp.BSPJobClient: SUPERSTEP_SUM=1584
12/10/22 23:11:26 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=681423
12/10/22 23:11:26 INFO bsp.BSPJobClient: IO_BYTES_READ=75380115
12/10/22 23:11:26 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=14690880
12/10/22 23:11:26 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=5105043
12/10/22 23:11:26 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=7345440
Job Finished in 18.91 seconds
{code}
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481237#comment-13481237 ]
Edward J. Yoon commented on HAMA-654:
-------------------------------------
NOTE:
{code}
bind 192.168.100.1 61000-61020
connect 192.168.100.0/24 61000-61020
...
<name>bsp.child.java.opts</name>
<value>-Dcom.sun.sdp.conf=/home/oracle/sdp.conf -Dcom.sun.sdp.debug=/home/oracle/debug.log -Xmx4000m</value>
{code}
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon resolved HAMA-654.
---------------------------------
Resolution: Fixed
Done. http://wiki.apache.org/hama/RunningHamaOverInfiniBand
I had expected (slightly) improved performance but results was quite strange.
SDP protocol enabled:
Finished in 168.32 seconds
Disabled:
Finished in 13.32 seconds
Will check later.
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon reassigned HAMA-654:
-----------------------------------
Assignee: Edward J. Yoon
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481269#comment-13481269 ]
Edward J. Yoon commented on HAMA-654:
-------------------------------------
It works good as I expected.
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-654) JDK 7 and SDP protocol
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478744#comment-13478744 ]
Edward J. Yoon commented on HAMA-654:
-------------------------------------
I have no problem compiling and running a Hama cluster using JDK 7.
- http://docs.oracle.com/javase/tutorial/sdp/sockets/index.html
I guess infiniband can be used by by adding some variables to hama-env.sh and bsp.child.java.opts property. Will test tomorrow.
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-654) JDK 7 and SDP protocol
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481274#comment-13481274 ]
Thomas Jungblut commented on HAMA-654:
--------------------------------------
I would say, you can add this documentation to the wiki and resolve this.
> JDK 7 and SDP protocol
> ----------------------
>
> Key: HAMA-654
> URL: https://issues.apache.org/jira/browse/HAMA-654
> Project: Hama
> Issue Type: Task
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
>
> Research and Documentation about JDK 7 and SDP protocol (infiniBand).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira