You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by John Yang <jo...@gmail.com> on 2014/10/29 10:28:37 UTC

Question on receiving less than requested

Hello all,


I have a question on receiving from RM less memory/vcores than requested.

As far as I understand, when the following configurations are set, RM can
allocate more than requested(when less than the minimum):

yarn.scheduler.minimum-allocation-mb,
yarn.scheduler.minimum-allocation-vcores

I wanted to know if the reverse is possible. For example when I request a
container of 500 MB and 2 vcores, is it possible to receive a container of
400 MB and 1 core?


Thanks,
John

Re: Question on receiving less than requested

Posted by John Yang <jo...@gmail.com>.

Hi Brahma,


It turns out that my problem was caused by YARN-2380(
https://issues.apache.org/jira/browse/YARN-2380).

Nevertheless, thank you so much for the information. I'll keep that in mind
when I run MR jobs.


Best regards,
John



On Thu, Oct 30, 2014 at 7:34 PM, Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:

> Hi john,
>
> Is it normal to receive from RM less mb/vcores than requested even if the
> requested mb/vcores are less than the maximum?
>
>
> >> Yes, it will be possible when you map.memory is less than
> yarn.scheduler.maximum-allocation..Ask will go to the schduler based on the
> mapred.map.memory only..
>
>
> E. G :
>
> If max-allocation= 4GB
>    min-allocation = 1GB
> map.memory=1.5 GB
>
> Always child will allocated with 1GB(it will be multiples of
> min-alllocation.) and vice versa...
>
>
>
> Thanks & Regards
> Brahma Reddy Battula
>
> ________________________________________
> From: John Yang [johnyangk@gmail.com]
> Sent: Wednesday, October 29, 2014 3:08 PM
> To: yarn-dev@hadoop.apache.org
> Subject: Re: Question on receiving less than requested
>
> Hi,
>
>
> to be more clear, I understand the following configurations can be set.
>
> yarn.scheduler.maximum-allocation-mb,
> yarn.scheduler.maximum-allocation-vcores
>
> Is it normal to receive from RM less mb/vcores than requested even if the
> requested mb/vcores are less than the maximum?
>
>
> Best regards,
> John
>
>
> On Wed, Oct 29, 2014 at 6:28 PM, John Yang <jo...@gmail.com> wrote:
>
> > Hello all,
> >
> >
> > I have a question on receiving from RM less memory/vcores than requested.
> >
> > As far as I understand, when the following configurations are set, RM can
> > allocate more than requested(when less than the minimum):
> >
> > yarn.scheduler.minimum-allocation-mb,
> > yarn.scheduler.minimum-allocation-vcores
> >
> > I wanted to know if the reverse is possible. For example when I request a
> > container of 500 MB and 2 vcores, is it possible to receive a container
> of
> > 400 MB and 1 core?
> >
> >
> > Thanks,
> > John
> >
> >
> >
>

RE: Question on receiving less than requested

Posted by Brahma Reddy Battula <br...@huawei.com>.

Hi john,

Is it normal to receive from RM less mb/vcores than requested even if the
requested mb/vcores are less than the maximum?


>> Yes, it will be possible when you map.memory is less than yarn.scheduler.maximum-allocation..Ask will go to the schduler based on the mapred.map.memory only..


E. G :

If max-allocation= 4GB
   min-allocation = 1GB
map.memory=1.5 GB

Always child will allocated with 1GB(it will be multiples of min-alllocation.) and vice versa...



Thanks & Regards
Brahma Reddy Battula

________________________________________
From: John Yang [johnyangk@gmail.com]
Sent: Wednesday, October 29, 2014 3:08 PM
To: yarn-dev@hadoop.apache.org
Subject: Re: Question on receiving less than requested

Hi,


to be more clear, I understand the following configurations can be set.

yarn.scheduler.maximum-allocation-mb,
yarn.scheduler.maximum-allocation-vcores

Is it normal to receive from RM less mb/vcores than requested even if the
requested mb/vcores are less than the maximum?


Best regards,
John


On Wed, Oct 29, 2014 at 6:28 PM, John Yang <jo...@gmail.com> wrote:

> Hello all,
>
>
> I have a question on receiving from RM less memory/vcores than requested.
>
> As far as I understand, when the following configurations are set, RM can
> allocate more than requested(when less than the minimum):
>
> yarn.scheduler.minimum-allocation-mb,
> yarn.scheduler.minimum-allocation-vcores
>
> I wanted to know if the reverse is possible. For example when I request a
> container of 500 MB and 2 vcores, is it possible to receive a container of
> 400 MB and 1 core?
>
>
> Thanks,
> John
>
>
>

Re: Question on receiving less than requested

Posted by John Yang <jo...@gmail.com>.

Hi,


to be more clear, I understand the following configurations can be set.

yarn.scheduler.maximum-allocation-mb,
yarn.scheduler.maximum-allocation-vcores

Is it normal to receive from RM less mb/vcores than requested even if the
requested mb/vcores are less than the maximum?


Best regards,
John


On Wed, Oct 29, 2014 at 6:28 PM, John Yang <jo...@gmail.com> wrote:

> Hello all,
>
>
> I have a question on receiving from RM less memory/vcores than requested.
>
> As far as I understand, when the following configurations are set, RM can
> allocate more than requested(when less than the minimum):
>
> yarn.scheduler.minimum-allocation-mb,
> yarn.scheduler.minimum-allocation-vcores
>
> I wanted to know if the reverse is possible. For example when I request a
> container of 500 MB and 2 vcores, is it possible to receive a container of
> 400 MB and 1 core?
>
>
> Thanks,
> John
>
>
>