[jira] [Resolved] (SPARK-1531) GraphX should have messageRDD to enable OutOfCore messages

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-1531.
------------------------------
    Resolution: Won't Fix

> GraphX should have messageRDD to enable OutOfCore messages
> ----------------------------------------------------------
>
>                 Key: SPARK-1531
>                 URL: https://issues.apache.org/jira/browse/SPARK-1531
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 0.9.1
>            Reporter: Jianfeng Jia
>
> There is no such `messageRDD` in Pregel function.
> Most of the sendMessage is directly sent one Scala Iterator. Like the below one in staticPageRank:
> ```
> def sendMessage(edge: EdgeTriplet[Double, Double]) =
>       Iterator((edge.dstId, edge.srcAttr * edge.attr))
> ```
> For some message intensively computation on some bigger graph, it will throw OOM exceptions. If we have some more general messageRDD, at lease we can set MessageRDD.persist(DISK) to enable it flush onto the disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org