Spark 1.1.0 does not spawn more than 6 executors in yarn-client mode and ignores --num-executors

[Aniket Bhatnagar <an...@gmail.com>]

I am running spark 1.1.0 on AWS EMR and I am running a batch job that
should seems to be highly parallelizable in yarn-client mode. But spark
stop spawning any more executors after spawning 6 executors even though
YARN cluster has 15 healthy m1.large nodes. I even tried providing
'--num-executors 60' argument during spark-submit but even that doesn't
help. A quick look at spark admin UI suggests there are active stages whose
tasks have not been started yet and even then spark doesn't start more
executors. I am not sure why. Any help on this would be greatly appreciated.

Here is link to screen shots that I took of spark admin and yarn admin -
http://imgur.com/a/ztjr7

Thanks,
Aniket

Re: Spark 1.1.0 does not spawn more than 6 executors in yarn-client mode and ignores --num-executors

[Aniket Bhatnagar <an...@gmail.com>]

Hi guys

I am hoping someone might have a clue on why this is happening. Otherwise I
will have to dwell into YARN module's source code to better understand the
issue.

On Wed, Dec 10, 2014, 11:54 PM Aniket Bhatnagar <an...@gmail.com>
wrote:

> I am running spark 1.1.0 on AWS EMR and I am running a batch job that
> should seems to be highly parallelizable in yarn-client mode. But spark
> stop spawning any more executors after spawning 6 executors even though
> YARN cluster has 15 healthy m1.large nodes. I even tried providing
> '--num-executors 60' argument during spark-submit but even that doesn't
> help. A quick look at spark admin UI suggests there are active stages whose
> tasks have not been started yet and even then spark doesn't start more
> executors. I am not sure why. Any help on this would be greatly appreciated.
>
> Here is link to screen shots that I took of spark admin and yarn admin -
> http://imgur.com/a/ztjr7
>
> Thanks,
> Aniket
>
>