You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "欧阳晋(欧阳晋)" <ji...@alibaba-inc.com> on 2014/07/19 18:31:07 UTC
About ResourceRequest for a self-written AppMaster to ResourceManager
Hi all:I'm writing my own AppMaster to run in YARN. And I'm confused about resource request between AppMaster and ResourceManager.My situation is that I want total 10 containers, so in my AppMaster . after some initiallization , my AppMaster send a 10 container resource request to ResourceManager, call allocate interface, and wait for allocation in a while loop. My app is such a type that it can run container immediately after allocation happens . So after maybe 2 containers are allocated by ResourceManager, I tell NodeManager to launch it . So my question is, should I send a new ResourceReqeust to ResourceManager that I need 8 Container ? It seems that after the first time I send a resource request and waiting for allocation , I can get containers , but I don't know whether that's suitable.Thanks!OuYang Jin
Re: Re：About ResourceRequest for a self-written AppMaster to ResourceManager
Posted by Devaraj K <de...@apache.org>.
Yes, MRAppMaster also does the same way. It sends total resource requests
for map phase one time and will wait for RM to allocate containers for all
those requests in the further allocate responses. It also starts launching
containers immediately as on it gets from RM without waiting for all
resource requests to be allocated.
On Sun, Jul 20, 2014 at 8:15 AM, 欧阳晋(欧阳晋) <ji...@alibaba-inc.com> wrote:
> Thank you very much Devaraj. It solves my question. And I wonder if this
> is the same thing the MapReduce ApplicationMaster do when it request
> resource to RM? Because MR is the App that can run map immediately after
> some containers for map are allocated, so for Map phase, MR AppMaster just
> send a total resource request for the first time and then just wait for
> allocation response?
>
> Thanks!
> Ouyang Jin
>
> ------------------------------------------------------------------
> 发件人：Devaraj K <de...@apache.org>
> 发送时间：2014年7月20日(星期日) 01:51
> 收件人：yarn-dev@hadoop.apache.org <ya...@hadoop.apache.org>; 欧阳晋(风骖) <
> jin.oyj@alibaba-inc.com>
> 主 题：Re: About ResourceRequest for a self-written AppMaster to
> ResourceManager
>
> Hi Jin,
>
>
> You don't need to request containers for each allocate invocation.
> You can send containers request one time for 10 containers and then keep on
> checking the allocate response for containers allocation, RM will try to
> allocate and gives as part of allocate response whenever they become
> available.
>
>
> For ex: If we request for 10 containers, assume RM can allocate 2
> containers at that time then RM gives two containers and keeps the
> remaining 8 containers request information. In future RM continues to
> allocate remaining requests whenever resources available and gives to
> ApplicationMaster as part of allocate response.
>
>
> On Sat, Jul 19, 2014 at 10:01 PM, 欧阳晋(欧阳晋) <ji...@alibaba-inc.com>
> wrote:
>
>> Hi all:I'm writing my own AppMaster to run in YARN. And I'm confused
>> about resource request between AppMaster and ResourceManager.My situation
>> is that I want total 10 containers, so in my AppMaster . after some
>> initiallization , my AppMaster send a 10 container resource request to
>> ResourceManager, call allocate interface, and wait for allocation in a
>> while loop. My app is such a type that it can run container immediately
>> after allocation happens . So after maybe 2 containers are allocated by
>> ResourceManager, I tell NodeManager to launch it . So my question is,
>> should I send a new ResourceReqeust to ResourceManager that I need 8
>> Container ? It seems that after the first time I send a resource request
>> and waiting for allocation , I can get containers , but I don't know
>> whether that's suitable.Thanks!OuYang Jin
>
>
>
>
> --
>
>
> Thanks
> Devaraj K
>
--
Thanks
Devaraj K
Re：About ResourceRequest for a self-written AppMaster to ResourceManager
Posted by "欧阳晋(欧阳晋)" <ji...@alibaba-inc.com>.
Thank you very much Devaraj. It solves my question. And I wonder if this is the same thing the MapReduce ApplicationMaster do when it request resource to RM? Because MR is the App that can run map immediately after some containers for map are allocated, so for Map phase, MR AppMaster just send a total resource request for the first time and then just wait for allocation response?
Thanks!Ouyang Jin------------------------------------------------------------------
发件人：Devaraj K <de...@apache.org>
发送时间：2014年7月20日(星期日) 01:51
收件人：yarn-dev@hadoop.apache.org <ya...@hadoop.apache.org>; 欧阳晋(风骖) <ji...@alibaba-inc.com>
主 题：Re: About ResourceRequest for a self-written AppMaster to ResourceManager
Hi Jin,
You don't need to request containers for each allocate
invocation. You can send containers request one time for 10 containers and then
keep on checking the allocate response for containers allocation, RM will try to allocate
and gives as part of allocate response whenever they become available.
For ex: If we request for 10 containers, assume RM can
allocate 2 containers at that time then RM gives two containers and keeps the
remaining 8 containers request information. In future RM continues to allocate remaining
requests whenever resources available and gives to ApplicationMaster as part of
allocate response.
On Sat, Jul 19, 2014 at 10:01 PM, 欧阳晋(欧阳晋) <ji...@alibaba-inc.com> wrote:
Hi all:I'm writing my own AppMaster to run in YARN. And I'm confused about resource request between AppMaster and ResourceManager.My situation is that I want total 10 containers, so in my AppMaster . after some initiallization , my AppMaster send a 10 container resource request to ResourceManager, call allocate interface, and wait for allocation in a while loop. My app is such a type that it can run container immediately after allocation happens . So after maybe 2 containers are allocated by ResourceManager, I tell NodeManager to launch it . So my question is, should I send a new ResourceReqeust to ResourceManager that I need 8 Container ? It seems that after the first time I send a resource request and waiting for allocation , I can get containers , but I don't know whether that's suitable.Thanks!OuYang Jin
--
ThanksDevaraj K
Re: About ResourceRequest for a self-written AppMaster to ResourceManager
Posted by Devaraj K <de...@apache.org>.
Hi Jin,
You don't need to request containers for each allocate invocation. You
can send containers request one time for 10 containers and then keep on
checking the allocate response for containers allocation, RM will try to
allocate and gives as part of allocate response whenever they become
available.
For ex: If we request for 10 containers, assume RM can allocate 2
containers at that time then RM gives two containers and keeps the
remaining 8 containers request information. In future RM continues to
allocate remaining requests whenever resources available and gives to
ApplicationMaster as part of allocate response.
On Sat, Jul 19, 2014 at 10:01 PM, 欧阳晋(欧阳晋) <ji...@alibaba-inc.com> wrote:
> Hi all:I'm writing my own AppMaster to run in YARN. And I'm confused about
> resource request between AppMaster and ResourceManager.My situation is that
> I want total 10 containers, so in my AppMaster . after some initiallization
> , my AppMaster send a 10 container resource request to ResourceManager,
> call allocate interface, and wait for allocation in a while loop. My app is
> such a type that it can run container immediately after allocation happens
> . So after maybe 2 containers are allocated by ResourceManager, I tell
> NodeManager to launch it . So my question is, should I send a new
> ResourceReqeust to ResourceManager that I need 8 Container ? It seems that
> after the first time I send a resource request and waiting for allocation ,
> I can get containers , but I don't know whether that's
> suitable.Thanks!OuYang Jin
--
Thanks
Devaraj K