You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Avery Ching (JIRA)" <ji...@apache.org> on 2011/08/30 02:46:38 UTC
[jira] [Created] (GIRAPH-14) Support for the Facebook Hadoop branch
Support for the Facebook Hadoop branch
--------------------------------------
Key: GIRAPH-14
URL: https://issues.apache.org/jira/browse/GIRAPH-14
Project: Giraph
Issue Type: New Feature
Reporter: Avery Ching
I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093348#comment-13093348 ]
Avery Ching commented on GIRAPH-14:
-----------------------------------
Thanks Hyunsik.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Joe Xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094209#comment-13094209 ]
Joe Xie commented on GIRAPH-14:
-------------------------------
thank you Avery! It passed the unittest for me too.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094305#comment-13094305 ]
Avery Ching commented on GIRAPH-14:
-----------------------------------
In theory, I believe that Facebook's distro is online (https://github.com/facebook/hadoop-20-warehouse). The long term story is to factor out the parts into modules and then compile them based on the user profile. Then we don't have to "munge" anything anymore. At least that's what I've thought of for now. I'm open to better solutions. Pre-processing will get unmaintainable if we have to support every version of Hadoop. That being said, we should support the big customers of Giraph and that likely includes Facebook as well.
I'll add instructions to the README and submit a new patch.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094271#comment-13094271 ]
Jakob Homan commented on GIRAPH-14:
-----------------------------------
I'm not up to date on FB's distribution. It's available to the public? If we're going to support this, should instructions be given in the README, as there are for the non-secure build option? What's the long term story for these API support #ifdefs? At the moment it's a clever solution to a vexing problem, but longer term it would be good to have a solution that doesn't leave code as comments.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Joe Xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093828#comment-13093828 ]
Joe Xie commented on GIRAPH-14:
-------------------------------
Great job Avery! I can compile on my side too and it works with examples on the cluster. I still got a few testing errors during mvn package & test, let me know if you have any updates.
Joe Xie
Tests in error:
testSuperstepBalancer(org.apache.giraph.TestVertexRangeBalancer)
testBspCheckpoint(org.apache.giraph.TestManualCheckpoint)
testSingleFault(org.apache.giraph.TestAutoCheckpoint)
testInstantiateVertex(org.apache.giraph.TestBspBasic)
testLocalJobRunnerConfig(org.apache.giraph.TestBspBasic)
testBspFail(org.apache.giraph.TestBspBasic)
testBspSuperStep(org.apache.giraph.TestBspBasic)
testBspMsg(org.apache.giraph.TestBspBasic)
testEmptyVertexInputFormat(org.apache.giraph.TestBspBasic)
testBspCombiner(org.apache.giraph.TestBspBasic)
testBspPageRank(org.apache.giraph.TestBspBasic)
testBspShortestPaths(org.apache.giraph.TestBspBasic)
testMutateGraph(org.apache.giraph.TestMutateGraphVertex)
testContinue(org.apache.giraph.TestJsonBase64Format)
testMatchingType(org.apache.giraph.TestVertexTypes)
testDerivedMatchingType(org.apache.giraph.TestVertexTypes)
testDerivedInputFormatType(org.apache.giraph.TestVertexTypes)
testMismatchingVertex(org.apache.giraph.TestVertexTypes)
testMismatchingCombiner(org.apache.giraph.TestVertexTypes)
testJsonBase64FormatType(org.apache.giraph.TestVertexTypes)
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-14) Support for the Facebook Hadoop branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-14:
------------------------------
Attachment: facebook3.patch
Updated with README instructions for building with the Facebook Hadoop release.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt, facebook3.patch
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Joe Xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093831#comment-13093831 ]
Joe Xie commented on GIRAPH-14:
-------------------------------
Thank you Avery! maybe my test file in the trunk is obsolete, let me know whether you can reproduce it.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching reassigned GIRAPH-14:
---------------------------------
Assignee: Avery Ching
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Hyunsik Choi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093345#comment-13093345 ]
Hyunsik Choi commented on GIRAPH-14:
------------------------------------
The below link is for hudson.
http://wiki.apache.org/general/Hudson
I'll create another issue about it.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094216#comment-13094216 ]
Avery Ching commented on GIRAPH-14:
-----------------------------------
Great to hear it! When one of the committers gets a chance to review, I can commit.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093879#comment-13093879 ]
Avery Ching commented on GIRAPH-14:
-----------------------------------
It's good to hear that you can run it on your cluster. As far as the unittests, that is strange. I was able to repeat the same issues and will look into a fix.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-14) Support for the Facebook Hadoop branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-14:
------------------------------
Attachment: facebook2.txt
Looks like I needed to change the groupId so that the right dependencies are pulled in for hadoop. Please try this one out. The unittests all passed for me.
(i.e. mvn -Dhadoop=facebook -Dhadoop.jar.path=/Users/aching/Desktop/hadoop-0.20.1-core.jar package)
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-14) Support for the Facebook Hadoop branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching updated GIRAPH-14:
------------------------------
Attachment: facebook.txt
Supports the Facebook version of Hadoop with mvn -Dhadoop=facebook -Dhadoop.jar.path=<path to jar> <mvn command>
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Avery Ching resolved GIRAPH-14.
-------------------------------
Resolution: Fixed
Committed, with changelog addition.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt, facebook3.patch
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-14) Support for the Facebook Hadoop
branch
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094321#comment-13094321 ]
Jakob Homan commented on GIRAPH-14:
-----------------------------------
Applied the patch and verified everything still works as expected on the regular build. +1.
> Support for the Facebook Hadoop branch
> --------------------------------------
>
> Key: GIRAPH-14
> URL: https://issues.apache.org/jira/browse/GIRAPH-14
> Project: Giraph
> Issue Type: New Feature
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: facebook.txt, facebook2.txt, facebook3.patch
>
>
> I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira