You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satyajit vegesna <sa...@gmail.com> on 2016/08/03 04:11:10 UTC
Spark on yarn, only 1 or 2 vcores getting allocated to the containers
getting created.
Hi All,
I am trying to run a spark job using yarn, and i specify --executor-cores
value as 20.
But when i go check the "nodes of the cluster" page in
http://hostname:8088/cluster/nodes then i see 4 containers getting created
on each of the node in cluster.
But can only see 1 vcore getting assigned for each containier, even when i
specify --executor-cores 20 while submitting job using spark-submit.
yarn-site.xml
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>60000</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>40</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>70000</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>20</value>
</property>
Did anyone face the same issue??
Regards,
Satyajit.
Re: Spark on yarn, only 1 or 2 vcores getting allocated to the
containers getting created.
Posted by Mungeol Heo <mu...@gmail.com>.
Try to turn yarn.scheduler.capacity.resource-calculator on, then check again.
On Wed, Aug 3, 2016 at 4:53 PM, Saisai Shao <sa...@gmail.com> wrote:
> Use dominant resource calculator instead of default resource calculator will
> get the expected vcores as you wanted. Basically by default yarn does not
> honor cpu cores as resource, so you will always see vcore is 1 no matter
> what number of cores you set in spark.
>
> On Wed, Aug 3, 2016 at 12:11 PM, satyajit vegesna
> <sa...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I am trying to run a spark job using yarn, and i specify --executor-cores
>> value as 20.
>> But when i go check the "nodes of the cluster" page in
>> http://hostname:8088/cluster/nodes then i see 4 containers getting created
>> on each of the node in cluster.
>>
>> But can only see 1 vcore getting assigned for each containier, even when i
>> specify --executor-cores 20 while submitting job using spark-submit.
>>
>> yarn-site.xml
>> <property>
>> <name>yarn.scheduler.maximum-allocation-mb</name>
>> <value>60000</value>
>> </property>
>> <property>
>> <name>yarn.scheduler.minimum-allocation-vcores</name>
>> <value>1</value>
>> </property>
>> <property>
>> <name>yarn.scheduler.maximum-allocation-vcores</name>
>> <value>40</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>70000</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.resource.cpu-vcores</name>
>> <value>20</value>
>> </property>
>>
>>
>> Did anyone face the same issue??
>>
>> Regards,
>> Satyajit.
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Spark on yarn, only 1 or 2 vcores getting allocated to the
containers getting created.
Posted by Mungeol Heo <mu...@gmail.com>.
Try to turn "yarn.scheduler.capacity.resource-calculator" on
On Wed, Aug 3, 2016 at 4:53 PM, Saisai Shao <sa...@gmail.com> wrote:
> Use dominant resource calculator instead of default resource calculator will
> get the expected vcores as you wanted. Basically by default yarn does not
> honor cpu cores as resource, so you will always see vcore is 1 no matter
> what number of cores you set in spark.
>
> On Wed, Aug 3, 2016 at 12:11 PM, satyajit vegesna
> <sa...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I am trying to run a spark job using yarn, and i specify --executor-cores
>> value as 20.
>> But when i go check the "nodes of the cluster" page in
>> http://hostname:8088/cluster/nodes then i see 4 containers getting created
>> on each of the node in cluster.
>>
>> But can only see 1 vcore getting assigned for each containier, even when i
>> specify --executor-cores 20 while submitting job using spark-submit.
>>
>> yarn-site.xml
>> <property>
>> <name>yarn.scheduler.maximum-allocation-mb</name>
>> <value>60000</value>
>> </property>
>> <property>
>> <name>yarn.scheduler.minimum-allocation-vcores</name>
>> <value>1</value>
>> </property>
>> <property>
>> <name>yarn.scheduler.maximum-allocation-vcores</name>
>> <value>40</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>70000</value>
>> </property>
>> <property>
>> <name>yarn.nodemanager.resource.cpu-vcores</name>
>> <value>20</value>
>> </property>
>>
>>
>> Did anyone face the same issue??
>>
>> Regards,
>> Satyajit.
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Spark on yarn, only 1 or 2 vcores getting allocated to the
containers getting created.
Posted by Saisai Shao <sa...@gmail.com>.
Use dominant resource calculator instead of default resource calculator
will get the expected vcores as you wanted. Basically by default yarn does
not honor cpu cores as resource, so you will always see vcore is 1 no
matter what number of cores you set in spark.
On Wed, Aug 3, 2016 at 12:11 PM, satyajit vegesna <
satyajit.apasprk@gmail.com> wrote:
> Hi All,
>
> I am trying to run a spark job using yarn, and i specify --executor-cores
> value as 20.
> But when i go check the "nodes of the cluster" page in
> http://hostname:8088/cluster/nodes then i see 4 containers getting
> created on each of the node in cluster.
>
> But can only see 1 vcore getting assigned for each containier, even when i
> specify --executor-cores 20 while submitting job using spark-submit.
>
> yarn-site.xml
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>60000</value>
> </property>
> <property>
> <name>yarn.scheduler.minimum-allocation-vcores</name>
> <value>1</value>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-vcores</name>
> <value>40</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>70000</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.cpu-vcores</name>
> <value>20</value>
> </property>
>
>
> Did anyone face the same issue??
>
> Regards,
> Satyajit.
>
Re: Spark on yarn, only 1 or 2 vcores getting allocated to the
containers getting created.
Posted by Saisai Shao <sa...@gmail.com>.
Use dominant resource calculator instead of default resource calculator
will get the expected vcores as you wanted. Basically by default yarn does
not honor cpu cores as resource, so you will always see vcore is 1 no
matter what number of cores you set in spark.
On Wed, Aug 3, 2016 at 12:11 PM, satyajit vegesna <
satyajit.apasprk@gmail.com> wrote:
> Hi All,
>
> I am trying to run a spark job using yarn, and i specify --executor-cores
> value as 20.
> But when i go check the "nodes of the cluster" page in
> http://hostname:8088/cluster/nodes then i see 4 containers getting
> created on each of the node in cluster.
>
> But can only see 1 vcore getting assigned for each containier, even when i
> specify --executor-cores 20 while submitting job using spark-submit.
>
> yarn-site.xml
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>60000</value>
> </property>
> <property>
> <name>yarn.scheduler.minimum-allocation-vcores</name>
> <value>1</value>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-vcores</name>
> <value>40</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>70000</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.cpu-vcores</name>
> <value>20</value>
> </property>
>
>
> Did anyone face the same issue??
>
> Regards,
> Satyajit.
>