You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Gavin Yue <yu...@gmail.com> on 2015/09/24 22:28:46 UTC

executor-cores setting does not work under Yarn

Running Spark app over Yarn 2.7

Here is my sparksubmit setting:
--master yarn-cluster \
 --num-executors 100 \
 --executor-cores 3 \
 --executor-memory 20g \
 --driver-memory 20g \
 --driver-cores 2 \

But the executor cores setting is not working. It always assigns only one
vcore  to one container based on the cluster metrics from yarn resource
manager website.

And yarn setting for container is
min: <memory:6600, vCores:4>  max: <memory:106473, vCores:15>

I have tried to change num-executors and executor memory. It even ignores
the min cCores setting and always assign one core per container.

Any advice?

Thank you!

Re: executor-cores setting does not work under Yarn

Posted by Gavin Yue <yu...@gmail.com>.

I think I found the problem.

Have to change the yarn capacity scheduler to use

DominantResourceCalculator

Thanks!


On Fri, Sep 25, 2015 at 4:54 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> Which version of spark are you having? Can you also check whats set in
> your conf/spark-defaults.conf file?
>
> Thanks
> Best Regards
>
> On Fri, Sep 25, 2015 at 1:58 AM, Gavin Yue <yu...@gmail.com> wrote:
>
>> Running Spark app over Yarn 2.7
>>
>> Here is my sparksubmit setting:
>> --master yarn-cluster \
>>  --num-executors 100 \
>>  --executor-cores 3 \
>>  --executor-memory 20g \
>>  --driver-memory 20g \
>>  --driver-cores 2 \
>>
>> But the executor cores setting is not working. It always assigns only one
>> vcore  to one container based on the cluster metrics from yarn resource
>> manager website.
>>
>> And yarn setting for container is
>> min: <memory:6600, vCores:4>  max: <memory:106473, vCores:15>
>>
>> I have tried to change num-executors and executor memory. It even ignores
>> the min cCores setting and always assign one core per container.
>>
>> Any advice?
>>
>> Thank you!
>>
>>
>>
>>
>>
>

Re: executor-cores setting does not work under Yarn

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Which version of spark are you having? Can you also check whats set in your
conf/spark-defaults.conf file?

Thanks
Best Regards

On Fri, Sep 25, 2015 at 1:58 AM, Gavin Yue <yu...@gmail.com> wrote:

> Running Spark app over Yarn 2.7
>
> Here is my sparksubmit setting:
> --master yarn-cluster \
>  --num-executors 100 \
>  --executor-cores 3 \
>  --executor-memory 20g \
>  --driver-memory 20g \
>  --driver-cores 2 \
>
> But the executor cores setting is not working. It always assigns only one
> vcore  to one container based on the cluster metrics from yarn resource
> manager website.
>
> And yarn setting for container is
> min: <memory:6600, vCores:4>  max: <memory:106473, vCores:15>
>
> I have tried to change num-executors and executor memory. It even ignores
> the min cCores setting and always assign one core per container.
>
> Any advice?
>
> Thank you!
>
>
>
>
>