Scalability issue in Spark with SparkPageRank example

["lokesh.gidra" <lo...@gmail.com>]

Hello, 


I ran SparkPageRank example (the one included in the package) to evaluate 
scale-in capability of Spark. I ran experiments on a 8-node 48-core 
AMD machine with local[N] master. But, for N > 10, the completion time 
of the experiment kept increasing, rather than decreasing. 

When I profiled it using Jprofiler, I observed that it wasn't any lock 
which consumed the CPU time. Instead, the amount of time spent in the 
following functions kept increasing: 

1) java.io.ObjectOutputStream.writeObject0 
2) scala.Tuple2.hashCode 

I confirmed the same with Oprofile as well. The findings are consistent. 

I am attaching the jstack output which I took twice during the whole 
execution with N=48 run. I ran the tests with Spark 1.0.0 and Spark 0.9.0

Can someone please suggest me what is wrong. 


Regards,
Lokesh Gidra

lessoutput3.lessoutput3
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n9521/lessoutput3.lessoutput3>  
lessoutput4.lessoutput4
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n9521/lessoutput4.lessoutput4>  



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Scalability-issue-in-Spark-with-SparkPageRank-example-tp9521.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.