Yarn not running as many executors as I'd like

[Jon Chase <jo...@gmail.com>]

Running on Amazon EMR w/Yarn and Spark 1.1.1, I have trouble getting Yarn
to use the number of executors that I specify in spark-submit:

--num-executors 2

In a cluster with two core nodes will typically only result in one executor
running at a time.  I can play with the memory settings and
num-cores-per-executor, and sometimes I can get 2 executors running at
once, but I'm not sure what the secret formula is to make this happen
consistently.

Re: Yarn not running as many executors as I'd like

[Marcelo Vanzin <va...@cloudera.com>]

How many cores / memory do you have available per NodeManager, and how
many cores / memory are you requesting for your job?

Remember that in Yarn mode, Spark launches "num executors + 1"
containers. The extra container, by default, reserves 1 core and about
1g of memory (more if running in cluster mode and specifying
"--driver-memory").

On Fri, Dec 19, 2014 at 12:57 PM, Jon Chase <jo...@gmail.com> wrote:
> Running on Amazon EMR w/Yarn and Spark 1.1.1, I have trouble getting Yarn to
> use the number of executors that I specify in spark-submit:
>
> --num-executors 2
>
> In a cluster with two core nodes will typically only result in one executor
> running at a time.  I can play with the memory settings and
> num-cores-per-executor, and sometimes I can get 2 executors running at once,
> but I'm not sure what the secret formula is to make this happen
> consistently.



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org