You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "orangeprince@foxmail.com" <or...@foxmail.com> on 2015/03/25 03:12:05 UTC

Graphx gets slower as the iteration number increases

I'm working with graphx to calculate the pageranks of an extreme large social network with billion verteces.
As iteration number increases, the speed of each iteration becomes slower and unacceptable. Is there any reason of it?
How can I accelerate the ineration process?


orangeprince@foxmail.com

Re: Graphx gets slower as the iteration number increases

Posted by Ankur Dave <an...@gmail.com>.

This might be because partitions are getting dropped from memory and
needing to be recomputed. How much memory is in the cluster, and how large
are the partitions? This information should be in the Executors and Storage
pages in the web UI.

Ankur <http://www.ankurdave.com/>

On Tue, Mar 24, 2015 at 7:12 PM, orangeprince@foxmail.com <
orangeprince@foxmail.com> wrote:

> I'm working with graphx to calculate the pageranks of an extreme large
> social network with billion verteces.
> As iteration number increases, the speed of each iteration becomes slower
> and unacceptable. Is there any reason of it?
>