You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takeshi Yamamuro (JIRA)" <ji...@apache.org> on 2014/11/28 09:03:12 UTC
[jira] [Commented] (SPARK-1987) More memory-efficient graph
construction
[ https://issues.apache.org/jira/browse/SPARK-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228136#comment-14228136 ]
Takeshi Yamamuro commented on SPARK-1987:
-----------------------------------------
What is the status of this patch?
This is related to a issue I created (https://issues.apache.org/jira/browse/SPARK-4646).
I refactored this patch based on my patch, it is as follows:
https://github.com/maropu/spark/commit/77e34424a5e6cf2bfd6300ab35f329bdaba6e775
Thanks :)
> More memory-efficient graph construction
> ----------------------------------------
>
> Key: SPARK-1987
> URL: https://issues.apache.org/jira/browse/SPARK-1987
> Project: Spark
> Issue Type: Improvement
> Components: GraphX
> Reporter: Ankur Dave
> Assignee: Ankur Dave
>
> A graph's edges are usually the largest component of the graph. GraphX currently stores edges in parallel primitive arrays, so each edge should only take 20 bytes to store (srcId: Long, dstId: Long, attr: Int). However, the current implementation in EdgePartitionBuilder uses an array of Edge objects as an intermediate representation for sorting, so each edge additionally takes about 40 bytes during graph construction (srcId (8) + dstId (8) + attr (4) + uncompressed pointer (8) + object overhead (8) + padding (4)). This unnecessarily increases GraphX's memory requirements by a factor of 3.
> To save memory, EdgePartitionBuilder should instead use a custom sort routine that operates directly on the three parallel arrays.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org