You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2015/03/12 03:22:38 UTC
[jira] [Commented] (HAMA-932) Use Kryo Serializer for message and
bundle serializations.
[ https://issues.apache.org/jira/browse/HAMA-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358006#comment-14358006 ]
Hudson commented on HAMA-932:
-----------------------------
FAILURE: Integrated in Hama-trunk #351 (See [https://builds.apache.org/job/Hama-trunk/351/])
HAMA-932: Use of Kryo Serializer (edwardyoon: rev 1666061)
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/math/DenseDoubleMatrix.java
* /hama/trunk/commons/src/main/java/org/apache/hama/commons/math/DenseDoubleVector.java
* /hama/trunk/core/pom.xml
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPMessageBundle.java
* /hama/trunk/core/src/test/java/org/apache/hama/util/TestKryoSerializer.java
* /hama/trunk/graph/pom.xml
* /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobMessage.java
* /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java
* /hama/trunk/graph/src/main/java/org/apache/hama/graph/OutgoingVertexMessageManager.java
* /hama/trunk/ml/src/test/java/org/apache/hama/ml/recommendation/TestOnlineCF.java
* /hama/trunk/pom.xml
> Use Kryo Serializer for message and bundle serializations.
> ----------------------------------------------------------
>
> Key: HAMA-932
> URL: https://issues.apache.org/jira/browse/HAMA-932
> Project: Hama
> Issue Type: Improvement
> Components: bsp core
> Affects Versions: 0.6.4
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.7.0
>
> Attachments: kryo.patch
>
>
> As we already know, kryo serializer shows quite awesome performance. I suggest that we use kryo serializer for everything.
> Attached patch uses kryo for message serialization and below is its results:
> {code}
> 15/03/10 18:08:08 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000
> 15/03/10 18:08:08 INFO bsp.FileInputFormat: Total input paths to process : 3
> 15/03/10 18:08:09 INFO bsp.BSPJobClient: Running job: job_201503101804_0007
> 15/03/10 18:08:12 INFO bsp.BSPJobClient: Current supersteps number: 0
> 15/03/10 18:09:34 INFO bsp.BSPJobClient: Current supersteps number: 1
> 15/03/10 18:11:24 INFO bsp.BSPJobClient: Current supersteps number: 2
> 15/03/10 18:11:27 INFO bsp.BSPJobClient: Current supersteps number: 3
> 15/03/10 18:14:33 INFO bsp.BSPJobClient: Current supersteps number: 4
> 15/03/10 18:14:36 INFO bsp.BSPJobClient: Current supersteps number: 5
> 15/03/10 18:17:20 INFO bsp.BSPJobClient: Current supersteps number: 7
> 15/03/10 18:18:12 INFO bsp.BSPJobClient: Current supersteps number: 9
> 15/03/10 18:18:15 INFO bsp.BSPJobClient: Current supersteps number: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: The total number of supersteps: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: Counters: 11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.graph.GraphJobRunner$GraphJobCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: ITERATIONS=4
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: INPUT_VERTICES=30000
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: SUPERSTEPS=11
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: LAUNCHED_TASKS=3
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: SUPERSTEP_SUM=33
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=596707
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: IO_BYTES_READ=84870046
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=44717766
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGE_BYTES_TRANSFERED=361694719
> 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=270035
> Job Finished in 639.248 seconds
> ....
> Kryo version:
> 15/03/10 18:23:17 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000
> 15/03/10 18:23:17 INFO bsp.FileInputFormat: Total input paths to process : 3
> 15/03/10 18:23:17 INFO bsp.BSPJobClient: Running job: job_201503101822_0001
> 15/03/10 18:23:20 INFO bsp.BSPJobClient: Current supersteps number: 0
> 15/03/10 18:23:35 INFO bsp.BSPJobClient: Current supersteps number: 1
> 15/03/10 18:24:16 INFO bsp.BSPJobClient: Current supersteps number: 2
> 15/03/10 18:24:19 INFO bsp.BSPJobClient: Current supersteps number: 3
> 15/03/10 18:25:23 INFO bsp.BSPJobClient: Current supersteps number: 4
> 15/03/10 18:25:26 INFO bsp.BSPJobClient: Current supersteps number: 5
> 15/03/10 18:26:47 INFO bsp.BSPJobClient: Current supersteps number: 7
> 15/03/10 18:26:53 INFO bsp.BSPJobClient: Current supersteps number: 9
> 15/03/10 18:26:56 INFO bsp.BSPJobClient: Current supersteps number: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: The total number of supersteps: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: Counters: 11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.graph.GraphJobRunner$GraphJobCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: ITERATIONS=4
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: INPUT_VERTICES=30000
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: SUPERSTEPS=11
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: LAUNCHED_TASKS=3
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: SUPERSTEP_SUM=33
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=185464
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: IO_BYTES_READ=84870046
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=44717766
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGE_BYTES_TRANSFERED=573865062
> 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=270039
> Job Finished in 250.217 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)