You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by mahender bigdata <Ma...@outlook.com> on 2016/04/26 01:16:50 UTC

Is there a way to resolve Fair Scheduler Problem

Hi Team,

Is there way to resolve Fair Queue Scheduler problem. Currently I see 
If  application requires more resources, it fully consumes available 
resources leaving other submitted applications in *pending *or *accepted 
*state. Do i need to modify
set yarn.scheduler.maximum-allocation-mb=5120;
set yarn.scheduler.minimum-allocation-mb=512;

or

Yarn scheduler Queue capacity needs to be modified.

Thanks.

Re: Is there a way to resolve Fair Scheduler Problem

Posted by mahender bigdata <Ma...@outlook.com>.

Thanks Khaja. I will look into it.



On 4/25/2016 4:38 PM, Khaja Hussain wrote:
> Hi
>
> If you need other application in a queue to start you can use 
> pre-emption. Below link has the details. What pre-emption does is it 
> guarantee's capacity, in which case your job will start. If there is 
> no load on the given queue those resources will be used by other 
> queue. Hope this helps.
>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/preemption.html
>
> Regards
> Khaja Hussain
>
> On Mon, Apr 25, 2016 at 7:16 PM, mahender bigdata 
> <Mahender.BigData@outlook.com <ma...@outlook.com>> 
> wrote:
>
>     Hi Team,
>
>     Is there way to resolve Fair Queue Scheduler problem. Currently I
>     see If  application requires more resources, it fully consumes
>     available resources leaving other submitted applications in
>     *pending *or *accepted *state. Do i need to modify
>     set yarn.scheduler.maximum-allocation-mb=5120;
>       set yarn.scheduler.minimum-allocation-mb=512;
>
>     or
>
>     Yarn scheduler Queue capacity needs to be modified.
>
>     Thanks.
>
>

Re: Is there a way to resolve Fair Scheduler Problem

Posted by Khaja Hussain <kh...@gmail.com>.

Hi

If you need other application in a queue to start you can use pre-emption.
Below link has the details. What pre-emption does is it guarantee's
capacity, in which case your job will start. If there is no load on the
given queue those resources will be used by other queue. Hope this helps.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/preemption.html

Regards
Khaja Hussain

On Mon, Apr 25, 2016 at 7:16 PM, mahender bigdata <
Mahender.BigData@outlook.com> wrote:

> Hi Team,
>
> Is there way to resolve Fair Queue Scheduler problem. Currently I see If
> application requires more resources, it fully consumes available resources
> leaving other submitted applications in *pending *or *accepted *state. Do
> i need to modify
>  set yarn.scheduler.maximum-allocation-mb=5120;
>   set yarn.scheduler.minimum-allocation-mb=512;
>
> or
>
> Yarn scheduler Queue capacity needs to be modified.
>
> Thanks.
>
>