You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by olegshirokikh <ol...@solver.com> on 2015/01/23 23:44:58 UTC
Apache Spark standalone mode: number of cores
I'm trying to understand the basics of Spark internals and Spark
documentation for submitting applications in local mode says for
spark-submit --master setting:
local[K] Run Spark locally with K worker threads (ideally, set this to the
number of cores on your machine).
local[*] Run Spark locally with as many worker threads as logical cores on
your machine.
Since all the data is stored on a single local machine, it does not benefit
from distributed operations on RDDs.
How does it benefit and what internally is going on when Spark utilizes
several logical cores?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-standalone-mode-number-of-cores-tp21342.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Apache Spark standalone mode: number of cores
Posted by Boromir Widas <vc...@gmail.com>.
The local mode still parallelizes calculations and it is useful for
debugging as it goes through the steps of serialization/deserialization as
a cluster would.
On Fri, Jan 23, 2015 at 5:44 PM, olegshirokikh <ol...@solver.com> wrote:
> I'm trying to understand the basics of Spark internals and Spark
> documentation for submitting applications in local mode says for
> spark-submit --master setting:
>
> local[K] Run Spark locally with K worker threads (ideally, set this to the
> number of cores on your machine).
>
> local[*] Run Spark locally with as many worker threads as logical cores on
> your machine.
> Since all the data is stored on a single local machine, it does not benefit
> from distributed operations on RDDs.
>
> How does it benefit and what internally is going on when Spark utilizes
> several logical cores?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-standalone-mode-number-of-cores-tp21342.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>