[jira] [Issue Comment Deleted] (SPARK-1750) EdgePartition is not serialized properly

["Ankur Dave (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ankur Dave updated SPARK-1750:
------------------------------

    Comment: was deleted

(was: Resolved in PR #742: https://github.com/apache/spark/pull/742)

> EdgePartition is not serialized properly
> ----------------------------------------
>
>                 Key: SPARK-1750
>                 URL: https://issues.apache.org/jira/browse/SPARK-1750
>             Project: Spark
>          Issue Type: Bug
>          Components: GraphX
>    Affects Versions: 0.9.0, 1.0.0, 0.9.1
>            Reporter: Ankur Dave
>             Fix For: 1.0.0
>
>
> The GraphX design attempts to avoid moving edges across the network, instead shipping the vertices to the edge partitions. However, Spark sometimes needs to move the edges, such as for straggler mitigation.
> All EdgePartition fields are currently declared transient, so the edges will not be serialized properly. Even if they are not marked transient, Kryo is unable to serialize the EdgePartition, failing with the following error:
> {code}
> java.lang.IllegalArgumentException: Can not set final org.apache.spark.graphx.util.collection.PrimitiveKeyOpenHashMap field org.apache.spark.graphx.impl.EdgePartition.index to scala.collection.immutable.$colon$colon
> {code}
> A workaround is to discourage Spark from moving the edges by setting {{spark.locality.wait}} to a high value such as 100000.



--
This message was sent by Atlassian JIRA
(v6.2#6252)