Re: SparkPi performance-3 cluster standalone mode

[Adnan <ns...@gmail.com>]

Hi,

Relatively new on spark and have tried running SparkPi example on a
standalone 12 core three machine cluster. What I'm failing to understand is,
that running this example with a single slice gives better performance as
compared to using 12 slices. Same was the case when I was using parallelize
function. The time is scaling almost linearly with adding each slice. Please
let me know if I'm doing anything wrong. The code snippet is given below:

 

Regards,

Ahsan Ijaz



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkPi-performance-3-cluster-standalone-mode-tp4530p4751.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: SparkPi performance-3 cluster standalone mode

[Matei Zaharia <ma...@gmail.com>]

The problem is that SparkPi uses Math.random(), which is a synchronized method, so it can’t scale to multiple cores. In fact it will be slower on multiple cores due to lock contention. Try another example and you’ll see better scaling. I think we’ll have to update SparkPi to create a new Random in each task to avoid this.

Matei

On Apr 24, 2014, at 4:43 AM, Adnan <ns...@gmail.com> wrote:

> Hi,
> 
> Relatively new on spark and have tried running SparkPi example on a
> standalone 12 core three machine cluster. What I'm failing to understand is,
> that running this example with a single slice gives better performance as
> compared to using 12 slices. Same was the case when I was using parallelize
> function. The time is scaling almost linearly with adding each slice. Please
> let me know if I'm doing anything wrong. The code snippet is given below:
> 
> 
> 
> Regards,
> 
> Ahsan Ijaz
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkPi-performance-3-cluster-standalone-mode-tp4530p4751.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.