[jira] [Commented] (TINKERPOP-1118) SparkGraphComputer should use StarGraph, not VertexWritable.

["Marko A. Rodriguez (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622698#comment-15622698 ] 

Marko A. Rodriguez commented on TINKERPOP-1118:
-----------------------------------------------

I think we can get rid of the {{VertexWritable}}/{{ObjectWritable}} serialization issues if we solve this ticket. cc/ [~dalaro]

Right now, {{VertexWritable}} and {{ObjectWritable}} have their own serialization logic. This is important as these classes are used outside of just running jobs, but also for reading and writing {{SequenceFiles}}. In Spark, we don't need to have the RDD use these writables and in fact, can just directly reference the objects they wrap. In this way, we could have a better split between {{GryoInput/OutputFormat}} and the internal job serialization (message passing and the like).

> SparkGraphComputer should use StarGraph, not VertexWritable.
> ------------------------------------------------------------
>
>                 Key: TINKERPOP-1118
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP-1118
>             Project: TinkerPop
>          Issue Type: Improvement
>          Components: hadoop
>    Affects Versions: 3.1.1-incubating
>            Reporter: Marko A. Rodriguez
>              Labels: breaking
>             Fix For: 3.3.0
>
>
> {{SparkGraphComputer}} input RDDs are typed as:
> {code}
> JavaPairRDD<Object,VertexWritable>
> {code}
> The {{VertexWritable}} usage is a vestige from Hadoop and Giraph. In Spark, we don't need to have this wrapper and thus, we can reduce the overhead (one less object header) by making the input RDDs typed as:
> {code}
> JavaPairRDD<Object,StarGraph>
> {code}
> This would be a breaking change for graph providers that implement their own {{InputRDD}} and {{OutputRDD}}, however, the fix is trivial. Instead of {{new VertexWritable(vertex)}}, they would simply do {{StarGraph.of(vertex)}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)