You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Ashwin Shankar <as...@gmail.com> on 2014/06/12 02:52:59 UTC
Why resource requests are normalized in RM ?
Hi,
Anyone knows why resource requests from AMs are normalized to
be multiples of yarn.scheduler.minimum-allocation-mb which is 1G
by default ?
Also is there any problem with reducing yarn.scheduler.minimum-allocation-mb
to less
than 1G ?
/**
* Utility method to normalize a list of resource requests, by insuring
that
* the memory for each request is a multiple of minMemory and is not zero.
*/
SchedulerUtils.normalizeRequests()
--
Thanks,
Ashwin
Re: Why resource requests are normalized in RM ?
Posted by Karthik Kambatla <ka...@cloudera.com>.
I believe they are normalized to be multiples of
yarn.scheduler.increment-allocation-mb.
yarn.scheduler.minimum-allocation-mb can be set to as low as zero. Llama
does this.
As to why normalization, I think it is to make sure there is no external
fragmentation. It is similar to why memory is paged.
On Wed, Jun 11, 2014 at 5:52 PM, Ashwin Shankar <as...@gmail.com>
wrote:
> Hi,
> Anyone knows why resource requests from AMs are normalized to
> be multiples of yarn.scheduler.minimum-allocation-mb which is 1G
> by default ?
> Also is there any problem with reducing yarn.scheduler.minimum-allocation-mb
> to less
> than 1G ?
>
> /**
>
> * Utility method to normalize a list of resource requests, by insuring
> that
>
> * the memory for each request is a multiple of minMemory and is not
> zero.
>
> */
>
> SchedulerUtils.normalizeRequests()
> --
> Thanks,
> Ashwin
>
>
>
Re: Why resource requests are normalized in RM ?
Posted by Karthik Kambatla <ka...@cloudera.com>.
I believe they are normalized to be multiples of
yarn.scheduler.increment-allocation-mb.
yarn.scheduler.minimum-allocation-mb can be set to as low as zero. Llama
does this.
As to why normalization, I think it is to make sure there is no external
fragmentation. It is similar to why memory is paged.
On Wed, Jun 11, 2014 at 5:52 PM, Ashwin Shankar <as...@gmail.com>
wrote:
> Hi,
> Anyone knows why resource requests from AMs are normalized to
> be multiples of yarn.scheduler.minimum-allocation-mb which is 1G
> by default ?
> Also is there any problem with reducing yarn.scheduler.minimum-allocation-mb
> to less
> than 1G ?
>
> /**
>
> * Utility method to normalize a list of resource requests, by insuring
> that
>
> * the memory for each request is a multiple of minMemory and is not
> zero.
>
> */
>
> SchedulerUtils.normalizeRequests()
> --
> Thanks,
> Ashwin
>
>
>
Re: Why resource requests are normalized in RM ?
Posted by Karthik Kambatla <ka...@cloudera.com>.
I believe they are normalized to be multiples of
yarn.scheduler.increment-allocation-mb.
yarn.scheduler.minimum-allocation-mb can be set to as low as zero. Llama
does this.
As to why normalization, I think it is to make sure there is no external
fragmentation. It is similar to why memory is paged.
On Wed, Jun 11, 2014 at 5:52 PM, Ashwin Shankar <as...@gmail.com>
wrote:
> Hi,
> Anyone knows why resource requests from AMs are normalized to
> be multiples of yarn.scheduler.minimum-allocation-mb which is 1G
> by default ?
> Also is there any problem with reducing yarn.scheduler.minimum-allocation-mb
> to less
> than 1G ?
>
> /**
>
> * Utility method to normalize a list of resource requests, by insuring
> that
>
> * the memory for each request is a multiple of minMemory and is not
> zero.
>
> */
>
> SchedulerUtils.normalizeRequests()
> --
> Thanks,
> Ashwin
>
>
>
Re: Why resource requests are normalized in RM ?
Posted by Karthik Kambatla <ka...@cloudera.com>.
I believe they are normalized to be multiples of
yarn.scheduler.increment-allocation-mb.
yarn.scheduler.minimum-allocation-mb can be set to as low as zero. Llama
does this.
As to why normalization, I think it is to make sure there is no external
fragmentation. It is similar to why memory is paged.
On Wed, Jun 11, 2014 at 5:52 PM, Ashwin Shankar <as...@gmail.com>
wrote:
> Hi,
> Anyone knows why resource requests from AMs are normalized to
> be multiples of yarn.scheduler.minimum-allocation-mb which is 1G
> by default ?
> Also is there any problem with reducing yarn.scheduler.minimum-allocation-mb
> to less
> than 1G ?
>
> /**
>
> * Utility method to normalize a list of resource requests, by insuring
> that
>
> * the memory for each request is a multiple of minMemory and is not
> zero.
>
> */
>
> SchedulerUtils.normalizeRequests()
> --
> Thanks,
> Ashwin
>
>
>