You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Xiaoye Sun <su...@gmail.com> on 2017/09/08 22:54:39 UTC

Multiple vcores per container when running Spark applications in Yarn cluster mode

Hi,

I am using Spark 1.6.1 and Yarn 2.7.4.
I want to submit a Spark application to a Yarn cluster. However, I found
that the number of vcores assigned to a container/executor is always 1,
even if I set spark.executor.cores=2. I also found the number of tasks an
executor runs concurrently is 2. So, it seems that Spark knows that an
executor/container has two CPU cores but the request is not correctly sent
to Yarn resource scheduler. I am using
the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
on Yarn.

I am wondering that is it possible to assign multiple vcores to a container
when a Spark job is submitted to a Yarn cluster in yarn-cluster mode.

Thanks!
Best,
Xiaoye

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Posted by Gourav Sengupta <go...@gmail.com>.
Saisai,

thanks a ton :)

Regards,
Gourav

On Mon, Sep 11, 2017 at 11:36 PM, Xiaoye Sun <su...@gmail.com> wrote:

> Hi Jerry,
>
> This solves my problem. 🙏 thanks
>
> On Sun, Sep 10, 2017 at 8:19 PM Saisai Shao <sa...@gmail.com>
> wrote:
>
>> I guess you're using Capacity Scheduler with DefaultResourceCalculator,
>> which doesn't count cpu cores into resource calculation, this "1" you saw
>> is actually meaningless. If you want to also calculate cpu resource, you
>> should choose DominantResourceCalculator.
>>
>> Thanks
>> Jerry
>>
>> On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <su...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using Spark 1.6.1 and Yarn 2.7.4.
>>> I want to submit a Spark application to a Yarn cluster. However, I found
>>> that the number of vcores assigned to a container/executor is always 1,
>>> even if I set spark.executor.cores=2. I also found the number of tasks an
>>> executor runs concurrently is 2. So, it seems that Spark knows that an
>>> executor/container has two CPU cores but the request is not correctly sent
>>> to Yarn resource scheduler. I am using the org.apache.hadoop.yarn.
>>> server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn.
>>>
>>> I am wondering that is it possible to assign multiple vcores to a
>>> container when a Spark job is submitted to a Yarn cluster in yarn-cluster
>>> mode.
>>>
>>> Thanks!
>>> Best,
>>> Xiaoye
>>>
>>
>>

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Posted by Xiaoye Sun <su...@gmail.com>.
Hi Jerry,

This solves my problem. 🙏 thanks

On Sun, Sep 10, 2017 at 8:19 PM Saisai Shao <sa...@gmail.com> wrote:

> I guess you're using Capacity Scheduler with DefaultResourceCalculator,
> which doesn't count cpu cores into resource calculation, this "1" you saw
> is actually meaningless. If you want to also calculate cpu resource, you
> should choose DominantResourceCalculator.
>
> Thanks
> Jerry
>
> On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <su...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using Spark 1.6.1 and Yarn 2.7.4.
>> I want to submit a Spark application to a Yarn cluster. However, I found
>> that the number of vcores assigned to a container/executor is always 1,
>> even if I set spark.executor.cores=2. I also found the number of tasks an
>> executor runs concurrently is 2. So, it seems that Spark knows that an
>> executor/container has two CPU cores but the request is not correctly sent
>> to Yarn resource scheduler. I am using
>> the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
>> on Yarn.
>>
>> I am wondering that is it possible to assign multiple vcores to a
>> container when a Spark job is submitted to a Yarn cluster in yarn-cluster
>> mode.
>>
>> Thanks!
>> Best,
>> Xiaoye
>>
>
>

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Posted by Saisai Shao <sa...@gmail.com>.
I guess you're using Capacity Scheduler with DefaultResourceCalculator,
which doesn't count cpu cores into resource calculation, this "1" you saw
is actually meaningless. If you want to also calculate cpu resource, you
should choose DominantResourceCalculator.

Thanks
Jerry

On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <su...@gmail.com> wrote:

> Hi,
>
> I am using Spark 1.6.1 and Yarn 2.7.4.
> I want to submit a Spark application to a Yarn cluster. However, I found
> that the number of vcores assigned to a container/executor is always 1,
> even if I set spark.executor.cores=2. I also found the number of tasks an
> executor runs concurrently is 2. So, it seems that Spark knows that an
> executor/container has two CPU cores but the request is not correctly sent
> to Yarn resource scheduler. I am using the org.apache.hadoop.yarn.
> server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn.
>
> I am wondering that is it possible to assign multiple vcores to a
> container when a Spark job is submitted to a Yarn cluster in yarn-cluster
> mode.
>
> Thanks!
> Best,
> Xiaoye
>