You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jimitkr <ji...@softpath.net> on 2016/01/24 21:11:12 UTC
Spark master takes more time with local[8] than local[1]
Hi All,
I have a machine with the following configuration:
32 GB RAM
500 GB HDD
8 CPUs
Following are the parameters i'm starting my Spark context with:
val conf = new
SparkConf().setAppName("MasterApp").setMaster("local[1]").set("spark.executor.memory",
"20g")
I'm reading a 4.3 GB file and counting the no. of characters in it.
When i run my program with:
local[1], the count is returned in 1.8 minutes
local[8], the count is returned in 4.2 minutes
Both the times, Spark uses Storage Memory of 3.7 GB.
Why would my program take more time with local[8]?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark master takes more time with local[8] than local[1]
Posted by nsalian <ns...@cloudera.com>.
Hi,
Thanks for the question.
Is it possible for you to elaborate on your application?
The flow of the application will help to understand what could potentially
cause things to slow down?
Do logs give you any idea what goes on? Have you had a chance to look?
Thank you.
-----
Neelesh S. Salian
Cloudera
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052p26061.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark master takes more time with local[8] than local[1]
Posted by Ted Yu <yu...@gmail.com>.
bq. I'm reading a 4.3 GB file
The contents of the file can be held in one executor.
Can you try files with much larger size ?
Cheers
On Sun, Jan 24, 2016 at 12:11 PM, jimitkr <ji...@softpath.net> wrote:
> Hi All,
>
> I have a machine with the following configuration:
> 32 GB RAM
> 500 GB HDD
> 8 CPUs
>
> Following are the parameters i'm starting my Spark context with:
>
> val conf = new
>
> SparkConf().setAppName("MasterApp").setMaster("local[1]").set("spark.executor.memory",
> "20g")
>
> I'm reading a 4.3 GB file and counting the no. of characters in it.
>
> When i run my program with:
> local[1], the count is returned in 1.8 minutes
> local[8], the count is returned in 4.2 minutes
>
> Both the times, Spark uses Storage Memory of 3.7 GB.
>
> Why would my program take more time with local[8]?
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>