You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Eli Reisman (JIRA)" <ji...@apache.org> on 2012/06/30 22:28:42 UTC
[jira] [Created] (GIRAPH-227) ArrayWritables and their sub-classes
don't play well with GiraphJob.run()
Eli Reisman created GIRAPH-227:
----------------------------------
Summary: ArrayWritables and their sub-classes don't play well with GiraphJob.run()
Key: GIRAPH-227
URL: https://issues.apache.org/jira/browse/GIRAPH-227
Project: Giraph
Issue Type: Bug
Components: graph, test
Affects Versions: 0.2.0
Reporter: Eli Reisman
Fix For: 0.2.0
Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-227) ArrayWritables and their
sub-classes don't play well with GiraphJob.run()
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404924#comment-13404924 ]
Avery Ching commented on GIRAPH-227:
------------------------------------
I'm no multigraph expert =)
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-227) ArrayWritables and their
sub-classes don't play well with GiraphJob.run()
Posted by "Avery Ching (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404684#comment-13404684 ]
Avery Ching commented on GIRAPH-227:
------------------------------------
Have you tried ArrayListWritable (Giraph)? See MsgList for an example. That should work.
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-227) ArrayWritables and their
sub-classes don't play well with GiraphJob.run()
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404674#comment-13404674 ]
Eli Reisman commented on GIRAPH-227:
------------------------------------
If I have missed something here, perhaps a JIRA to get someone more experienced to put up a practice example or vertex showing best practices for using ArrayWritables successfully with Giraph is in order. Again, my concern is without this, Giraph will not support multigraph type <E> edge values so, basically no multigraph support without heavier code chages/API changes. This seems unneeded when Hadoop already has ArrayWritable right there for us to use. Has anyone tried them on Giraph and just not submitted a patch yet?
Thanks!
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-227) ArrayWritables and their sub-classes
don't play well with GiraphJob.run()
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman resolved GIRAPH-227.
--------------------------------
Resolution: Fixed
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-227) ArrayWritables and their
sub-classes don't play well with GiraphJob.run()
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406079#comment-13406079 ]
Eli Reisman commented on GIRAPH-227:
------------------------------------
To those who follow: DO NOT use ArrayWritable and friends as you will see in the Hadoop Writable docs online.
DO SUBCLASS org.apache.giraph.comm.ArrayListWritable as occurs in org.apache.giraph.comm.MsgList and (when GIRAPH-228 patch is up) org.apache.giraph.examples.SimpleTriangleClosingVertex (and more to come)
Matter closed! Thanks for the advice/clarification, Avery!
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-227) ArrayWritables and their
sub-classes don't play well with GiraphJob.run()
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404875#comment-13404875 ]
Eli Reisman commented on GIRAPH-227:
------------------------------------
Fantastic, thanks! I will! I just posted a fix using Text as values in GIRAPH-228 that works fine, but perhaps I could make an example vertex using ArayListWritable as <E> value to show people how to make them work, and to demonstrate that Giraph already supports multi graphs. Are there any easy multigraph algorithms I could implement to give us a demo for the codebase? Then we get 2 useful code examples for the price of one!
> ArrayWritables and their sub-classes don't play well with GiraphJob.run()
> -------------------------------------------------------------------------
>
> Key: GIRAPH-227
> URL: https://issues.apache.org/jira/browse/GIRAPH-227
> Project: Giraph
> Issue Type: Bug
> Components: graph, test
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Fix For: 0.2.0
>
>
> Twice now (GIRAPH-157 and GIRAPH-217) I have attempted to implement vertices that use ArrayWritables (including subclassing as described in Hadoop docs, and the generic String[] version with its own constructor). Although the code will compile and compute() can successfully be unit tested, a real run breaks the code. Thanks to Sean Choi (also struggling with this) for helping me figure out I'm not crazy, this is a real issue.
> A guick grep in Giraph reveals no one else is using them yet, so maybe thats why this has gotten past us thus far. I am happy to take a hard look at GiraphJob (etc) and see if i can figure a workaround, but this one might be above my pay grade. Help!
> Incidently, I used Sean's solution in 157 to simple output final data tuples as Text vertex values, but this is wasteful and not good for bulk data processing. Could be something about our output formats, but between us we have attempted several with no dice. If I have just been a newbie and missed something, please let me know, otherwise I'm going to change GIRAPH-217 to use Text output as well, and fix it again later when this issue is resolved.
> Thanks for your help!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira