[jira] [Assigned] (SPARK-12737) Decrease the redundant activeIds sent to remote mirrors in "aggregateMessagesWithActiveSet"

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-12737:
------------------------------------

    Assignee: Apache Spark

> Decrease the redundant activeIds sent to remote mirrors in "aggregateMessagesWithActiveSet"
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-12737
>                 URL: https://issues.apache.org/jira/browse/SPARK-12737
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 1.5.2
>            Reporter: qbwu
>            Assignee: Apache Spark
>              Labels: newbie
>
> Hi, I found that it is not necessary to send the activeIds to all the mirrors of some master in the activeSetOpt which is passed to aggregateMessagesWithActiveSet. Because through the passed EdgeDirection, we can infer what kind of mirrors (classified by their position) will be checked isActive later, we can send the activeIds to the mirrors only at some position. In some cases even no activeId needs to be sent.
> I have implemented it, and did some tests using PageRank and ConnectedComponent, the shuffle size and the running time are decreased. But the number of runing turn is not changed.   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org