You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by 杨逸帆Steven <me...@legato.ninja> on 2015/02/23 12:37:43 UTC

Why return NULL_ASSIGNMENT when one app can not assign to this leaf queue?

HI,

I am learning Yarn source code, and have this confussion.

In the assignContainers@LeafQueue.java, yarn will iterate all active
applications and try to do the schedule work. however, I notive if one
application's require is over queue limit, the function will return but not
break.

Say I have app#2 needs 500mb, and next app# needs 300mb. The queue now has
only 400mb available left, so when app#2 canAssignToThisQueue fails,
function return and leaves app#3 without schedule. And in this case, app#3
is able to run.

So, what is the reason to Return here?


Thank you.

P.S can I disable reservation? and how to do that?

Sincerely,
Steven

Re: Why return NULL_ASSIGNMENT when one app can not assign to this leaf queue?

Posted by Wangda Tan <wh...@gmail.com>.

I'm not very sure, since it's more like a hack to me, so you can just try
it :).

Regards,
Wangda

On Mon, Feb 23, 2015 at 5:22 PM, 杨逸帆Steven <me...@legato.ninja> wrote:

> Hi Da,
>
> Thanks for your explain. Now I understand it better. As for the
> reservation, can I simply comment out the function 'reserve' in LeafQueue
> for capacity scheduler?
>
> Sincerely,
> Steven
>
>
> On Tue, Feb 24, 2015 at 4:57 AM, Wangda Tan <wh...@gmail.com> wrote:
>
> > Hi Steven,
> > For now, CapacityScheduler trying to solve is large batch job, it
> allocates
> > resource in FIFO manner, and adding capacity limits, etc. So if a job
> > submit later cannot get resource before previous job get satisfied. This
> is
> > to make big batch job can get resource quickly and finish quickly.
> >
> > I don't think there's a way to disable reservation.
> >
> > HTH,
> > Wangda
> >
> > On Mon, Feb 23, 2015 at 3:37 AM, 杨逸帆Steven <me...@legato.ninja> wrote:
> >
> > > HI,
> > >
> > > I am learning Yarn source code, and have this confussion.
> > >
> > > In the assignContainers@LeafQueue.java, yarn will iterate all active
> > > applications and try to do the schedule work. however, I notive if one
> > > application's require is over queue limit, the function will return but
> > not
> > > break.
> > >
> > > Say I have app#2 needs 500mb, and next app# needs 300mb. The queue now
> > has
> > > only 400mb available left, so when app#2 canAssignToThisQueue fails,
> > > function return and leaves app#3 without schedule. And in this case,
> > app#3
> > > is able to run.
> > >
> > > So, what is the reason to Return here?
> > >
> > >
> > > Thank you.
> > >
> > > P.S can I disable reservation? and how to do that?
> > >
> > > Sincerely,
> > > Steven
> > >
> >
>

Re: Why return NULL_ASSIGNMENT when one app can not assign to this leaf queue?

Posted by 杨逸帆Steven <me...@legato.ninja>.

Hi Da,

Thanks for your explain. Now I understand it better. As for the
reservation, can I simply comment out the function 'reserve' in LeafQueue
for capacity scheduler?

Sincerely,
Steven


On Tue, Feb 24, 2015 at 4:57 AM, Wangda Tan <wh...@gmail.com> wrote:

> Hi Steven,
> For now, CapacityScheduler trying to solve is large batch job, it allocates
> resource in FIFO manner, and adding capacity limits, etc. So if a job
> submit later cannot get resource before previous job get satisfied. This is
> to make big batch job can get resource quickly and finish quickly.
>
> I don't think there's a way to disable reservation.
>
> HTH,
> Wangda
>
> On Mon, Feb 23, 2015 at 3:37 AM, 杨逸帆Steven <me...@legato.ninja> wrote:
>
> > HI,
> >
> > I am learning Yarn source code, and have this confussion.
> >
> > In the assignContainers@LeafQueue.java, yarn will iterate all active
> > applications and try to do the schedule work. however, I notive if one
> > application's require is over queue limit, the function will return but
> not
> > break.
> >
> > Say I have app#2 needs 500mb, and next app# needs 300mb. The queue now
> has
> > only 400mb available left, so when app#2 canAssignToThisQueue fails,
> > function return and leaves app#3 without schedule. And in this case,
> app#3
> > is able to run.
> >
> > So, what is the reason to Return here?
> >
> >
> > Thank you.
> >
> > P.S can I disable reservation? and how to do that?
> >
> > Sincerely,
> > Steven
> >
>

Re: Why return NULL_ASSIGNMENT when one app can not assign to this leaf queue?

Posted by Wangda Tan <wh...@gmail.com>.

Hi Steven,
For now, CapacityScheduler trying to solve is large batch job, it allocates
resource in FIFO manner, and adding capacity limits, etc. So if a job
submit later cannot get resource before previous job get satisfied. This is
to make big batch job can get resource quickly and finish quickly.

I don't think there's a way to disable reservation.

HTH,
Wangda

On Mon, Feb 23, 2015 at 3:37 AM, 杨逸帆Steven <me...@legato.ninja> wrote:

> HI,
>
> I am learning Yarn source code, and have this confussion.
>
> In the assignContainers@LeafQueue.java, yarn will iterate all active
> applications and try to do the schedule work. however, I notive if one
> application's require is over queue limit, the function will return but not
> break.
>
> Say I have app#2 needs 500mb, and next app# needs 300mb. The queue now has
> only 400mb available left, so when app#2 canAssignToThisQueue fails,
> function return and leaves app#3 without schedule. And in this case, app#3
> is able to run.
>
> So, what is the reason to Return here?
>
>
> Thank you.
>
> P.S can I disable reservation? and how to do that?
>
> Sincerely,
> Steven
>