You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shushant Arora <sh...@gmail.com> on 2015/04/30 20:38:13 UTC
max number of application master in YARN
Is there any configuration in MR2 and YARN to limit concurrent max
applications by setting max limit on ApplicationMasters in the cluster?
Re: max number of application master in YARN
Posted by Rajesh Kartha <ka...@gmail.com>.
With Capacity Scheduler, the other useful param would be:
yarn.scheduler.capacity.maximum-applications
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
On Thu, Apr 30, 2015 at 11:52 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Rajesh Kartha <ka...@gmail.com>.
With Capacity Scheduler, the other useful param would be:
yarn.scheduler.capacity.maximum-applications
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
On Thu, Apr 30, 2015 at 11:52 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Rajesh Kartha <ka...@gmail.com>.
With Capacity Scheduler, the other useful param would be:
yarn.scheduler.capacity.maximum-applications
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
On Thu, Apr 30, 2015 at 11:52 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Rajesh Kartha <ka...@gmail.com>.
With Capacity Scheduler, the other useful param would be:
yarn.scheduler.capacity.maximum-applications
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
On Thu, Apr 30, 2015 at 11:52 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Shushant,
If you use fair scheduler, you can restrict number of AM by configuring queue:
* maxRunningApps: limit the number of apps from the queue to run at once
* maxAMShare: limit the fraction of the queue's fair share that can be
used to run application masters. This property can only be used for
leaf queues. For example, if set to 1.0f, then AMs in the leaf queue
can take up to 100% of both the memory and CPU fair share. The value
of -1.0f will disable this feature and the amShare will not be
checked. The default value is 0.5f.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Thanks,
- Tsuyoshi
On Fri, May 1, 2015 at 4:04 AM, Shushant Arora
<sh...@gmail.com> wrote:
> Thanks!!
> Is it for capacity scheduler only. Or applicable to fair scheduler also?
>
> On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
> wrote:
>>
>> Take a look at
>>
>> yarn.scheduler.capacity.maximum-am-resource-percent
>>
>>
>>
>>
>> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora
>> <sh...@gmail.com> wrote:
>>>
>>> Is there any configuration in MR2 and YARN to limit concurrent max
>>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>>
>
Re: max number of application master in YARN
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Shushant,
If you use fair scheduler, you can restrict number of AM by configuring queue:
* maxRunningApps: limit the number of apps from the queue to run at once
* maxAMShare: limit the fraction of the queue's fair share that can be
used to run application masters. This property can only be used for
leaf queues. For example, if set to 1.0f, then AMs in the leaf queue
can take up to 100% of both the memory and CPU fair share. The value
of -1.0f will disable this feature and the amShare will not be
checked. The default value is 0.5f.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Thanks,
- Tsuyoshi
On Fri, May 1, 2015 at 4:04 AM, Shushant Arora
<sh...@gmail.com> wrote:
> Thanks!!
> Is it for capacity scheduler only. Or applicable to fair scheduler also?
>
> On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
> wrote:
>>
>> Take a look at
>>
>> yarn.scheduler.capacity.maximum-am-resource-percent
>>
>>
>>
>>
>> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora
>> <sh...@gmail.com> wrote:
>>>
>>> Is there any configuration in MR2 and YARN to limit concurrent max
>>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>>
>
Re: max number of application master in YARN
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Shushant,
If you use fair scheduler, you can restrict number of AM by configuring queue:
* maxRunningApps: limit the number of apps from the queue to run at once
* maxAMShare: limit the fraction of the queue's fair share that can be
used to run application masters. This property can only be used for
leaf queues. For example, if set to 1.0f, then AMs in the leaf queue
can take up to 100% of both the memory and CPU fair share. The value
of -1.0f will disable this feature and the amShare will not be
checked. The default value is 0.5f.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Thanks,
- Tsuyoshi
On Fri, May 1, 2015 at 4:04 AM, Shushant Arora
<sh...@gmail.com> wrote:
> Thanks!!
> Is it for capacity scheduler only. Or applicable to fair scheduler also?
>
> On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
> wrote:
>>
>> Take a look at
>>
>> yarn.scheduler.capacity.maximum-am-resource-percent
>>
>>
>>
>>
>> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora
>> <sh...@gmail.com> wrote:
>>>
>>> Is there any configuration in MR2 and YARN to limit concurrent max
>>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>>
>
Re: max number of application master in YARN
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Shushant,
If you use fair scheduler, you can restrict number of AM by configuring queue:
* maxRunningApps: limit the number of apps from the queue to run at once
* maxAMShare: limit the fraction of the queue's fair share that can be
used to run application masters. This property can only be used for
leaf queues. For example, if set to 1.0f, then AMs in the leaf queue
can take up to 100% of both the memory and CPU fair share. The value
of -1.0f will disable this feature and the amShare will not be
checked. The default value is 0.5f.
http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Thanks,
- Tsuyoshi
On Fri, May 1, 2015 at 4:04 AM, Shushant Arora
<sh...@gmail.com> wrote:
> Thanks!!
> Is it for capacity scheduler only. Or applicable to fair scheduler also?
>
> On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
> wrote:
>>
>> Take a look at
>>
>> yarn.scheduler.capacity.maximum-am-resource-percent
>>
>>
>>
>>
>> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora
>> <sh...@gmail.com> wrote:
>>>
>>> Is there any configuration in MR2 and YARN to limit concurrent max
>>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>>
>
Re: max number of application master in YARN
Posted by Shushant Arora <sh...@gmail.com>.
Thanks!!
Is it for capacity scheduler only. Or applicable to fair scheduler also?
On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Shushant Arora <sh...@gmail.com>.
Thanks!!
Is it for capacity scheduler only. Or applicable to fair scheduler also?
On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Shushant Arora <sh...@gmail.com>.
Thanks!!
Is it for capacity scheduler only. Or applicable to fair scheduler also?
On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Shushant Arora <sh...@gmail.com>.
Thanks!!
Is it for capacity scheduler only. Or applicable to fair scheduler also?
On Fri, May 1, 2015 at 12:22 AM, Prashant Kommireddi <pr...@gmail.com>
wrote:
> Take a look at
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
>
>
> On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <
> shushantarora09@gmail.com> wrote:
>
>> Is there any configuration in MR2 and YARN to limit concurrent max
>> applications by setting max limit on ApplicationMasters in the cluster?
>>
>
>
Re: max number of application master in YARN
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at
yarn.scheduler.capacity.maximum-am-resource-percent
On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <sh...@gmail.com>
wrote:
> Is there any configuration in MR2 and YARN to limit concurrent max
> applications by setting max limit on ApplicationMasters in the cluster?
>
Re: max number of application master in YARN
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at
yarn.scheduler.capacity.maximum-am-resource-percent
On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <sh...@gmail.com>
wrote:
> Is there any configuration in MR2 and YARN to limit concurrent max
> applications by setting max limit on ApplicationMasters in the cluster?
>
Re: max number of application master in YARN
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at
yarn.scheduler.capacity.maximum-am-resource-percent
On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <sh...@gmail.com>
wrote:
> Is there any configuration in MR2 and YARN to limit concurrent max
> applications by setting max limit on ApplicationMasters in the cluster?
>
Re: max number of application master in YARN
Posted by Prashant Kommireddi <pr...@gmail.com>.
Take a look at
yarn.scheduler.capacity.maximum-am-resource-percent
On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora <sh...@gmail.com>
wrote:
> Is there any configuration in MR2 and YARN to limit concurrent max
> applications by setting max limit on ApplicationMasters in the cluster?
>