YARN Resource Allocation When Memory is Very Small

[Nico Pappagianis <ni...@salesforce.com>]

How does YARN decide if remaining memory is enough AM allocation

Hi all,

Somewhat of a newb here so hopefully this question isn't too trivial.

I want to know how YARN decides if the remaining memory after AM resource
allocation is "enough". Here's a breakdown of what's going on in my
particular case.

We have a YARN queue that has ~1.7GB memory allocated to it. We have a
parent MR job that spawns multiple child jobs. The AM container memory for
the parent job is configured to 1GB. Once the AM container gets allocated
its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB
is enough for the child jobs to succeed.

My question is, what if that remaining amount wasn't 0.7GB but something
much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will
it allocate *any* remaining resources to the child jobs? Will there be a
ton of thrashing if there is only a small amount of memory available?

Thanks for any help, or a pointer in the right direction!

Re: YARN Resource Allocation When Memory is Very Small

[Ravi Prakash <ra...@gmail.com>]

Hi Nico!

The RM is configured with a minimum allocation. Take a look at
"yarn.scheduler.minimum-allocation-mb" . You can also read through code
here:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java#L1295
.
Please note that the code above is for the trunk branch, you should go look
in the branch that you are running. Thanks to great work from some folks,
this code has been evolving a lot recently.

HTH
Ravi

On Tue, Aug 30, 2016 at 12:09 PM, Nico Pappagianis <
nico.pappagianis@salesforce.com> wrote:

> How does YARN decide if remaining memory is enough AM allocation
>
> Hi all,
>
> Somewhat of a newb here so hopefully this question isn't too trivial.
>
> I want to know how YARN decides if the remaining memory after AM resource
> allocation is "enough". Here's a breakdown of what's going on in my
> particular case.
>
> We have a YARN queue that has ~1.7GB memory allocated to it. We have a
> parent MR job that spawns multiple child jobs. The AM container memory for
> the parent job is configured to 1GB. Once the AM container gets allocated
> its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB
> is enough for the child jobs to succeed.
>
> My question is, what if that remaining amount wasn't 0.7GB but something
> much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will
> it allocate *any* remaining resources to the child jobs? Will there be a
> ton of thrashing if there is only a small amount of memory available?
>
> Thanks for any help, or a pointer in the right direction!
>