You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Deep Pradhan <pr...@gmail.com> on 2015/02/27 10:18:57 UTC

spark.default.parallelism

Hi,
I have a four single core machines as slaves in my cluster. I set the
spark.default.parallelism to 4 and ran SparkTC given in examples. It took
around 26 sec.
Now, I increased the spark.default.parallelism to 8, but my performance
deteriorates. The same application takes 32 sec now.
I have read that usually the best performance is obtained when the
parallelism is set to 2X of the number of cores available. I do not quite
understand this. Could anyone please tell me?


Thank You
Regards,
Deep