[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents

["li xiaosen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

li xiaosen updated SPARK-30432:
-------------------------------
        Fix Version/s:     (was: 2.4.4)
     Target Version/s: 2.4.5, 3.0.0  (was: 2.4.4, 2.4.5)
    Affects Version/s:     (was: 2.4.4)
          Description: 
 

So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true.

I have done a small code proposal, because there is a problem when the pregel executions have done,  the degree no need to be recomputed.

 

for example，the Email-EuAll  data set：[http://snap.stanford.edu/data/email-EuAll.html]

do-while loop execute 10 times，and the reduce logic happend 8 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation.

 

I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]

 

I hope you can consider this small code proposal.

Thank you very much,

Best regards,

xs-li

  was:
It would be helpful when computing StronglyConnectedComponents to reduce degree computation.

I have done a small code proposal, because there is a problem when the pregel executions have done,  the degree no need to be recomputed.

I created a branch in my fork: [https://github.com/xs-li/spark/blob/branch-2.4/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]

I hope you can consider this small code proposal.

Thank you very much,

Best regards,

xs-li

             Priority: Major  (was: Minor)

> reduce degree recomputation in StronglyConnectedComponents
> ----------------------------------------------------------
>
>                 Key: SPARK-30432
>                 URL: https://issues.apache.org/jira/browse/SPARK-30432
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 2.4.5, 3.0.0
>            Reporter: li xiaosen
>            Priority: Major
>
>  
> So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true.
> I have done a small code proposal, because there is a problem when the pregel executions have done,  the degree no need to be recomputed.
>  
> for example，the Email-EuAll  data set：[http://snap.stanford.edu/data/email-EuAll.html]
> do-while loop execute 10 times，and the reduce logic happend 8 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation.
>  
> I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]
>  
> I hope you can consider this small code proposal.
> Thank you very much,
> Best regards,
> xs-li



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org