You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Ed Kohlwey (Created) (JIRA)" <ji...@apache.org> on 2011/12/19 22:03:31 UTC
[jira] [Created] (GIRAPH-108) Refactor code to run independently of
Map/Reduce
Refactor code to run independently of Map/Reduce
------------------------------------------------
Key: GIRAPH-108
URL: https://issues.apache.org/jira/browse/GIRAPH-108
Project: Giraph
Issue Type: Improvement
Components: graph
Reporter: Ed Kohlwey
It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (GIRAPH-108) Refactor code to run
independently of Map/Reduce
Posted by Claudio Martella <cl...@gmail.com>.
I agree, I think this might be a good sub-task of the general
re-factoring/abstraction from the mapreduceV1 framework.
What about this patch? does it run on mesos? does it break on MRv1?
infos about the tests?
On Tue, Dec 20, 2011 at 2:11 AM, Ed Kohlwey (Commented) (JIRA)
<ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172813#comment-13172813 ]
>
> Ed Kohlwey commented on GIRAPH-108:
> -----------------------------------
>
> Hi Claudio,
> Thanks for pointing me that way. It seems like this might be appropriate to call a sub-task for GIRAPH-13? My eventual goal with this patch is actually to target Mesos, and 13 seems to be targeting YARN.
>
> It seems like it shouldn't be too hard to maintain a few drivers, one for M/R, one for YARN, and one for Mesos - most of the daemon code is pretty well abstracted and shouldn't take a lot of work to port to the other frameworks.
>
>> Refactor code to run independently of Map/Reduce
>> ------------------------------------------------
>>
>> Key: GIRAPH-108
>> URL: https://issues.apache.org/jira/browse/GIRAPH-108
>> Project: Giraph
>> Issue Type: Improvement
>> Components: graph
>> Reporter: Ed Kohlwey
>> Attachments: GIRAPH-108
>>
>>
>> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
--
Claudio Martella
claudio.martella@gmail.com
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Ed Kohlwey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172813#comment-13172813 ]
Ed Kohlwey commented on GIRAPH-108:
-----------------------------------
Hi Claudio,
Thanks for pointing me that way. It seems like this might be appropriate to call a sub-task for GIRAPH-13? My eventual goal with this patch is actually to target Mesos, and 13 seems to be targeting YARN.
It seems like it shouldn't be too hard to maintain a few drivers, one for M/R, one for YARN, and one for Mesos - most of the daemon code is pretty well abstracted and shouldn't take a lot of work to port to the other frameworks.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-108) Refactor code to run independently of
Map/Reduce
Posted by "Ed Kohlwey (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ed Kohlwey updated GIRAPH-108:
------------------------------
Attachment: GIRAPH-108.patch
This patch uses TaskAttemptContext instead of TaskInputOutputContext, which is a bit cleaner.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108, GIRAPH-108.patch
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173396#comment-13173396 ]
Avery Ching commented on GIRAPH-108:
------------------------------------
But the way, I think this is a duplicate of GIRAPH-76.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173931#comment-13173931 ]
Avery Ching commented on GIRAPH-108:
------------------------------------
Actually, I'll let Jakob take a first crack at looking at this since he's got some expertise in the area.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Claudio Martella (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172690#comment-13172690 ]
Claudio Martella commented on GIRAPH-108:
-----------------------------------------
Hi Ed,
thanks for this. You might want to take a look at GIRAPH-13 which is already open about this issue.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Avery Ching (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173401#comment-13173401 ]
Avery Ching commented on GIRAPH-108:
------------------------------------
Thanks for the code Ed. I'll take a look at it tonight.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-108) Refactor code to run independently of
Map/Reduce
Posted by "Ed Kohlwey (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ed Kohlwey updated GIRAPH-108:
------------------------------
Attachment: GIRAPH-108
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Jakob Homan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173406#comment-13173406 ]
Jakob Homan commented on GIRAPH-108:
------------------------------------
bq. It seems like it shouldn't be too hard to maintain a few drivers, one for M/R, one for YARN, and one for Mesos - most of the daemon code is pretty well abstracted and shouldn't take a lot of work to port to the other frameworks.
Yes, this is reasonable, although
bq. It uses a sort of hack to get around the Map/Reduce input format issue
is probably not going to fly. I'd rather not introduce anything more hacky than is strictly necessary. I'll try to take a look at this code by the end of the week.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently
of Map/Reduce
Posted by "Ed Kohlwey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173497#comment-13173497 ]
Ed Kohlwey commented on GIRAPH-108:
-----------------------------------
{quote}
is probably not going to fly. I'd rather not introduce anything more hacky than is strictly necessary. I'll try to take a look at this code by the end of the week.
{quote}
I agree. My philosophy in this patch is to balance incremental change against dropping a patch bomb. I thought about tearing up the Input and Output format classes but realized that it would probably make the patch gigantic (and thus less likely to be accepted).
I went ahead and started a discussion on GIRAPH-111 for this.
> Refactor code to run independently of Map/Reduce
> ------------------------------------------------
>
> Key: GIRAPH-108
> URL: https://issues.apache.org/jira/browse/GIRAPH-108
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Reporter: Ed Kohlwey
> Attachments: GIRAPH-108
>
>
> It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira