You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (Created) (JIRA)" <ji...@apache.org> on 2012/04/18 19:30:40 UTC
[jira] [Created] (HAMA-558) Add graph generators from our benchmark
section
Add graph generators from our benchmark section
-----------------------------------------------
Key: HAMA-558
URL: https://issues.apache.org/jira/browse/HAMA-558
Project: Hama
Issue Type: New Feature
Reporter: Thomas Jungblut
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-558) Add graph generators from our benchmark
section
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-558:
---------------------------------
Component/s: graph
examples
bsp
Description:
In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
I have translated it to the new API and made a patch for it.
So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp, examples, graph
> Reporter: Thomas Jungblut
> Attachments: HAMA-558.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-558) Add graph generators from our benchmark
section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-558:
---------------------------------
Fix Version/s: (was: 0.6.0)
Affects Version/s: (was: 0.5.0)
Status: Open (was: Patch Available)
cancel and wait until the blocker is done.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-558) Add graph generators from our benchmark
section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-558:
---------------------------------
Fix Version/s: 0.6.0
Affects Version/s: 0.5.0
Status: Patch Available (was: Open)
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.6.0
>
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281410#comment-13281410 ]
Thomas Jungblut commented on HAMA-558:
--------------------------------------
Sure it is broken.
However you could fix it with setting the Configuration statically.
Put
bq.VertexWritable.CONFIGURATION = context.getConfiguraton()
in your setup method of the mapper and reducer.
In any case, we should get arround these and look how we could make the serialization smarter.
But that's in HAMA-580.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278695#comment-13278695 ]
Edward J. Yoon commented on HAMA-558:
-------------------------------------
In my cluster case, only hdfs and bsp daemons are running.
I really dislike adding M/R program to HAMA project.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.6.0
>
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258146#comment-13258146 ]
Thomas Jungblut commented on HAMA-558:
--------------------------------------
There is a conf property for localhost HDFS in there. This must be removed before committing.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp, examples, graph
> Reporter: Thomas Jungblut
> Attachments: HAMA-558.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-558) Add graph generators from our benchmark
section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-558:
---------------------------------
Attachment: HAMA-558_2.patch
remove the localhost hdfs
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Fix For: 0.6.0
>
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281482#comment-13281482 ]
Edward J. Yoon commented on HAMA-558:
-------------------------------------
There's a bug. I'm fixing directly.
{code}
Index: src/main/java/org/apache/hama/graph/VertexWritable.java
===================================================================
--- src/main/java/org/apache/hama/graph/VertexWritable.java (revision 1341804)
+++ src/main/java/org/apache/hama/graph/VertexWritable.java (working copy)
@@ -144,8 +144,8 @@
@Override
public int compareTo(VertexWritable<VERTEX_ID, VERTEX_VALUE> o) {
VertexWritable<VERTEX_ID, VERTEX_VALUE> that = o;
- return ((Comparable<VertexWritable<VERTEX_ID, VERTEX_VALUE>>) this.vertexId)
- .compareTo((VertexWritable<VERTEX_ID, VERTEX_VALUE>) that.vertexId);
+ return ((Comparable<VERTEX_ID>) this.vertexId)
+ .compareTo((VERTEX_ID) that.vertexId);
}
@Override
{code}
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-558) Add graph generators from our benchmark
section
Posted by "Thomas Jungblut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut updated HAMA-558:
---------------------------------
Attachment: HAMA-558.patch
My translation to graph package and new dates.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Reporter: Thomas Jungblut
> Attachments: HAMA-558.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284037#comment-13284037 ]
Thomas Jungblut commented on HAMA-558:
--------------------------------------
I will host this on my github then, since it just requires Hadoop now.
-> close.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut resolved HAMA-558.
----------------------------------
Resolution: Won't Fix
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284039#comment-13284039 ]
Thomas Jungblut commented on HAMA-558:
--------------------------------------
For SSSP example
https://github.com/thomasjungblut/thomasjungblut-common/blob/master/src/de/jungblut/bsp/RandomGraphGenerator.java
Hf.
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jungblut reassigned HAMA-558:
------------------------------------
Assignee: Thomas Jungblut
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-558) Add graph generators from our
benchmark section
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281353#comment-13281353 ]
Edward J. Yoon commented on HAMA-558:
-------------------------------------
BTW, I'm receiving an below error with this patch. Is this a bug of VertexWritable?
{code}
12/05/23 10:58:57 INFO mapred.JobClient: map 0% reduce 0%
12/05/23 10:59:11 INFO mapred.JobClient: Task Id : attempt_201205231058_0002_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hama.graph.VertexWritable.readFields(VertexWritable.java:100)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:97)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{code}
> Add graph generators from our benchmark section
> -----------------------------------------------
>
> Key: HAMA-558
> URL: https://issues.apache.org/jira/browse/HAMA-558
> Project: Hama
> Issue Type: New Feature
> Components: bsp core, examples, graph
> Reporter: Thomas Jungblut
> Assignee: Thomas Jungblut
> Attachments: HAMA-558.patch, HAMA-558_2.patch
>
>
> In our graph algorithm benchmarks we used an MR based graph generator: http://wiki.apache.org/hama/Benchmarks#Single_Shortest_Path_Problem
> I have translated it to the new API and made a patch for it.
> So people can reproduce the results, create sample graphs and don't have to compile the stuff from a java file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira