You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Jakob Homan (Created) (JIRA)" <ji...@apache.org> on 2011/11/12 05:48:51 UTC
[jira] [Created] (GIRAPH-76) Refactor worker logic from GraphMapper
Refactor worker logic from GraphMapper
--------------------------------------
Key: GIRAPH-76
URL: https://issues.apache.org/jira/browse/GIRAPH-76
Project: Giraph
Issue Type: Improvement
Components: graph
Reporter: Jakob Homan
The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Arun Suresh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151825#comment-13151825 ]
Arun Suresh commented on GIRAPH-76:
-----------------------------------
Yes, this does sound like a good idea. I could take a crack at it you havn't already started.
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Arun Suresh (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun Suresh reassigned GIRAPH-76:
---------------------------------
Assignee: Arun Suresh
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Arun Suresh
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401404#comment-13401404 ]
Alessandro Presta commented on GIRAPH-76:
-----------------------------------------
What functionality exactly should we extract from GraphMapper?
Something vertex-level like:
{code:java}
VertexRunner vertexRunner = new VertexRunner(graphState, context, partitionStats, partitionStatsList);
for (BasicVertex<I, V, E, M> basicVertex : partition.getVertices()) {
vertexRunner.run(basicVertex);
}
{code}
or something that wraps all the configuration (graphState, serviceWorker...) and runs all vertices in all partitions?
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Alessandro Presta
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401264#comment-13401264 ]
Alessandro Presta commented on GIRAPH-76:
-----------------------------------------
Is this still relevant? I could give it a shot.
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Arun Suresh
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402932#comment-13402932 ]
Avery Ching commented on GIRAPH-76:
-----------------------------------
Good start!
So for this (anyone else, please correct me if I'm wrong), I believe the goal is to pull everything out of the map method into something else to make it easier for using something like YARN to be able to run the framework outside of Hadoop.
Btw, perhaps PartitionRunner makes more sense (not creating an object per vertex). Also, I would think about making it open toward supporting multi-threading in the future. I had a patch for multithreading, but it was a little...unclean =).
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Alessandro Presta
> Attachments: Refactor_worker_logic_from_GraphMapper.patch
>
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta reassigned GIRAPH-76:
---------------------------------------
Assignee: Alessandro Presta (was: Arun Suresh)
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Alessandro Presta
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151860#comment-13151860 ]
Jakob Homan commented on GIRAPH-76:
-----------------------------------
I've not. Please go for it.
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-76) Refactor worker logic from GraphMapper
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessandro Presta updated GIRAPH-76:
------------------------------------
Attachment: Refactor_worker_logic_from_GraphMapper.patch
Attached patch is for the per-vertex refactoring.
I can easily modify it to handle all vertices in all partitions in a single run() if that's better.
Also let me know if you have a better name than VertexRunner.
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Alessandro Presta
> Attachments: Refactor_worker_logic_from_GraphMapper.patch
>
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-76) Refactor worker logic from
GraphMapper
Posted by "Claudio Martella (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401287#comment-13401287 ]
Claudio Martella commented on GIRAPH-76:
----------------------------------------
Yes, very much so.
> Refactor worker logic from GraphMapper
> --------------------------------------
>
> Key: GIRAPH-76
> URL: https://issues.apache.org/jira/browse/GIRAPH-76
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Jakob Homan
> Assignee: Arun Suresh
>
> The plumbing around executing vertices is hosted within the mapper, but could be extracted to its own class and executed from the Mapper directly. This would ease testing and make it easier to host in the new YARN infrastructure. There's nothing mapper specific about this code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira