You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Shushant Arora <sh...@gmail.com> on 2015/07/17 20:00:27 UTC
total vcores per node containers in yarn
In Yarn there is a setting to specify no of vcores that can be allocated to
containers.
yarn.nodemanager.resource.cpu-vcores
In my cluster's nodes yarn-site.xml, this property is not specified. But
total vcores displayed on RM's web page for the nodes are different than
the default(8). Is there any other place to control this number ?
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
That doesn't sound right, if you are indeed making use of the new feature.
First off, this is not the default behaviour in the actual Apache Hadoop
2.5.0 release. Are you using a different distribution based on that release
instead? Secondly, if it were using the auto-calculate feature, it'd only
count primary cores as vcores by default.
Are you certain you've looked at the right yarn-site.xml that pertains to
the NodeManager, and not a client-purpose file which may simply lack the
config mentioned in it?
On Sat, Jul 18, 2015 at 7:50 PM Shushant Arora <sh...@gmail.com>
wrote:
> Its hadoop 2.5.0.
> Whats the logic of default using hardware detection. Say My node has 8
> actual core and 32 virtual cores. Its taking 26 as value of vcores
> available of this node on RM UI.
>
> On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What version of Apache Hadoop are you running? Recent changes have made
>> YARN to auto-compute this via hardware detection, by default (rather than
>> the 8 default).
>>
>> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <
>> shushantarora09@gmail.com> wrote:
>>
>>> In Yarn there is a setting to specify no of vcores that can be allocated
>>> to containers.
>>>
>>> yarn.nodemanager.resource.cpu-vcores
>>>
>>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>>> total vcores displayed on RM's web page for the nodes are different than
>>> the default(8). Is there any other place to control this number ?
>>>
>>
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
That doesn't sound right, if you are indeed making use of the new feature.
First off, this is not the default behaviour in the actual Apache Hadoop
2.5.0 release. Are you using a different distribution based on that release
instead? Secondly, if it were using the auto-calculate feature, it'd only
count primary cores as vcores by default.
Are you certain you've looked at the right yarn-site.xml that pertains to
the NodeManager, and not a client-purpose file which may simply lack the
config mentioned in it?
On Sat, Jul 18, 2015 at 7:50 PM Shushant Arora <sh...@gmail.com>
wrote:
> Its hadoop 2.5.0.
> Whats the logic of default using hardware detection. Say My node has 8
> actual core and 32 virtual cores. Its taking 26 as value of vcores
> available of this node on RM UI.
>
> On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What version of Apache Hadoop are you running? Recent changes have made
>> YARN to auto-compute this via hardware detection, by default (rather than
>> the 8 default).
>>
>> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <
>> shushantarora09@gmail.com> wrote:
>>
>>> In Yarn there is a setting to specify no of vcores that can be allocated
>>> to containers.
>>>
>>> yarn.nodemanager.resource.cpu-vcores
>>>
>>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>>> total vcores displayed on RM's web page for the nodes are different than
>>> the default(8). Is there any other place to control this number ?
>>>
>>
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
That doesn't sound right, if you are indeed making use of the new feature.
First off, this is not the default behaviour in the actual Apache Hadoop
2.5.0 release. Are you using a different distribution based on that release
instead? Secondly, if it were using the auto-calculate feature, it'd only
count primary cores as vcores by default.
Are you certain you've looked at the right yarn-site.xml that pertains to
the NodeManager, and not a client-purpose file which may simply lack the
config mentioned in it?
On Sat, Jul 18, 2015 at 7:50 PM Shushant Arora <sh...@gmail.com>
wrote:
> Its hadoop 2.5.0.
> Whats the logic of default using hardware detection. Say My node has 8
> actual core and 32 virtual cores. Its taking 26 as value of vcores
> available of this node on RM UI.
>
> On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What version of Apache Hadoop are you running? Recent changes have made
>> YARN to auto-compute this via hardware detection, by default (rather than
>> the 8 default).
>>
>> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <
>> shushantarora09@gmail.com> wrote:
>>
>>> In Yarn there is a setting to specify no of vcores that can be allocated
>>> to containers.
>>>
>>> yarn.nodemanager.resource.cpu-vcores
>>>
>>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>>> total vcores displayed on RM's web page for the nodes are different than
>>> the default(8). Is there any other place to control this number ?
>>>
>>
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
That doesn't sound right, if you are indeed making use of the new feature.
First off, this is not the default behaviour in the actual Apache Hadoop
2.5.0 release. Are you using a different distribution based on that release
instead? Secondly, if it were using the auto-calculate feature, it'd only
count primary cores as vcores by default.
Are you certain you've looked at the right yarn-site.xml that pertains to
the NodeManager, and not a client-purpose file which may simply lack the
config mentioned in it?
On Sat, Jul 18, 2015 at 7:50 PM Shushant Arora <sh...@gmail.com>
wrote:
> Its hadoop 2.5.0.
> Whats the logic of default using hardware detection. Say My node has 8
> actual core and 32 virtual cores. Its taking 26 as value of vcores
> available of this node on RM UI.
>
> On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What version of Apache Hadoop are you running? Recent changes have made
>> YARN to auto-compute this via hardware detection, by default (rather than
>> the 8 default).
>>
>> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <
>> shushantarora09@gmail.com> wrote:
>>
>>> In Yarn there is a setting to specify no of vcores that can be allocated
>>> to containers.
>>>
>>> yarn.nodemanager.resource.cpu-vcores
>>>
>>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>>> total vcores displayed on RM's web page for the nodes are different than
>>> the default(8). Is there any other place to control this number ?
>>>
>>
>
Re: total vcores per node containers in yarn
Posted by Shushant Arora <sh...@gmail.com>.
Its hadoop 2.5.0.
Whats the logic of default using hardware detection. Say My node has 8
actual core and 32 virtual cores. Its taking 26 as value of vcores
available of this node on RM UI.
On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
> What version of Apache Hadoop are you running? Recent changes have made
> YARN to auto-compute this via hardware detection, by default (rather than
> the 8 default).
>
> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
> wrote:
>
>> In Yarn there is a setting to specify no of vcores that can be allocated
>> to containers.
>>
>> yarn.nodemanager.resource.cpu-vcores
>>
>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>> total vcores displayed on RM's web page for the nodes are different than
>> the default(8). Is there any other place to control this number ?
>>
>
Re: total vcores per node containers in yarn
Posted by Shushant Arora <sh...@gmail.com>.
Its hadoop 2.5.0.
Whats the logic of default using hardware detection. Say My node has 8
actual core and 32 virtual cores. Its taking 26 as value of vcores
available of this node on RM UI.
On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
> What version of Apache Hadoop are you running? Recent changes have made
> YARN to auto-compute this via hardware detection, by default (rather than
> the 8 default).
>
> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
> wrote:
>
>> In Yarn there is a setting to specify no of vcores that can be allocated
>> to containers.
>>
>> yarn.nodemanager.resource.cpu-vcores
>>
>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>> total vcores displayed on RM's web page for the nodes are different than
>> the default(8). Is there any other place to control this number ?
>>
>
Re: total vcores per node containers in yarn
Posted by Shushant Arora <sh...@gmail.com>.
Its hadoop 2.5.0.
Whats the logic of default using hardware detection. Say My node has 8
actual core and 32 virtual cores. Its taking 26 as value of vcores
available of this node on RM UI.
On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
> What version of Apache Hadoop are you running? Recent changes have made
> YARN to auto-compute this via hardware detection, by default (rather than
> the 8 default).
>
> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
> wrote:
>
>> In Yarn there is a setting to specify no of vcores that can be allocated
>> to containers.
>>
>> yarn.nodemanager.resource.cpu-vcores
>>
>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>> total vcores displayed on RM's web page for the nodes are different than
>> the default(8). Is there any other place to control this number ?
>>
>
Re: total vcores per node containers in yarn
Posted by Shushant Arora <sh...@gmail.com>.
Its hadoop 2.5.0.
Whats the logic of default using hardware detection. Say My node has 8
actual core and 32 virtual cores. Its taking 26 as value of vcores
available of this node on RM UI.
On Sat, Jul 18, 2015 at 7:22 PM, Harsh J <ha...@cloudera.com> wrote:
> What version of Apache Hadoop are you running? Recent changes have made
> YARN to auto-compute this via hardware detection, by default (rather than
> the 8 default).
>
> On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
> wrote:
>
>> In Yarn there is a setting to specify no of vcores that can be allocated
>> to containers.
>>
>> yarn.nodemanager.resource.cpu-vcores
>>
>> In my cluster's nodes yarn-site.xml, this property is not specified. But
>> total vcores displayed on RM's web page for the nodes are different than
>> the default(8). Is there any other place to control this number ?
>>
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
What version of Apache Hadoop are you running? Recent changes have made
YARN to auto-compute this via hardware detection, by default (rather than
the 8 default).
On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
wrote:
> In Yarn there is a setting to specify no of vcores that can be allocated
> to containers.
>
> yarn.nodemanager.resource.cpu-vcores
>
> In my cluster's nodes yarn-site.xml, this property is not specified. But
> total vcores displayed on RM's web page for the nodes are different than
> the default(8). Is there any other place to control this number ?
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
What version of Apache Hadoop are you running? Recent changes have made
YARN to auto-compute this via hardware detection, by default (rather than
the 8 default).
On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
wrote:
> In Yarn there is a setting to specify no of vcores that can be allocated
> to containers.
>
> yarn.nodemanager.resource.cpu-vcores
>
> In my cluster's nodes yarn-site.xml, this property is not specified. But
> total vcores displayed on RM's web page for the nodes are different than
> the default(8). Is there any other place to control this number ?
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
What version of Apache Hadoop are you running? Recent changes have made
YARN to auto-compute this via hardware detection, by default (rather than
the 8 default).
On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
wrote:
> In Yarn there is a setting to specify no of vcores that can be allocated
> to containers.
>
> yarn.nodemanager.resource.cpu-vcores
>
> In my cluster's nodes yarn-site.xml, this property is not specified. But
> total vcores displayed on RM's web page for the nodes are different than
> the default(8). Is there any other place to control this number ?
>
Re: total vcores per node containers in yarn
Posted by Harsh J <ha...@cloudera.com>.
What version of Apache Hadoop are you running? Recent changes have made
YARN to auto-compute this via hardware detection, by default (rather than
the 8 default).
On Fri, Jul 17, 2015 at 11:31 PM Shushant Arora <sh...@gmail.com>
wrote:
> In Yarn there is a setting to specify no of vcores that can be allocated
> to containers.
>
> yarn.nodemanager.resource.cpu-vcores
>
> In my cluster's nodes yarn-site.xml, this property is not specified. But
> total vcores displayed on RM's web page for the nodes are different than
> the default(8). Is there any other place to control this number ?
>