Questions about Capacity scheduler behavior

[Fabio <an...@gmail.com>]

Hi guys, I'm posting this in the user mailing list since I got no reply 
in the Yarn-dev. I have to model as well as possible the capacity 
scheduler behavior and I have some questions, hope someone can help me 
with this. In the following I will consider all containers to be equal 
for simplicity:

1) In case of multiple queues having the same level of assigned 
resources, what's the policy to decide which comes first in the resource 
allocation?

2) Let's consider this configuration:
We have a cluster hosting a total of 40 containers. We have 3 queues: A 
is configured to get 39% of cluster capacity, B also gets 39% and C gets 
22%. The number of containers is going to be 15.6, 15.6 and 8.8 for A, B 
an C. Since we can't split a container, how does the Capacity scheduler 
round these values in a real case? Who gets the two contended 
containers? I may think they are considered as extra containers, thus 
shared upon need among the three queues. Is this correct?

3) Let's say I have queues A and B. A is configured to get 20% (20 
containers) of the total cluster capacity (100 containers), B gets 80% 
(80 containers). Capacity scheduler gives available resources firstly to 
the most under-served queue.
In case A is using 10 containers and B is using 20, who is going to get 
the first available container? A is already using 50% of it's assigned 
capacity, B just 25%, but A has less containers than B... who is 
considered to be more under-served?

4) Does the previous question make sense at all? Because I have a doubt 
that when I have free containers I will just serve requests as they 
arrive, possibly over-provisioning a queue (that is: if I get a 
container request for an app in A, I will give it a container since I 
don't know that after a few milliseconds I will get a new request from 
B, or vice versa). The previous question may have sense if there was 
some sort of buffer that is filled with incoming requests, due to the 
difficulty of serving them in real time, thus making the scheduler able 
to choose the request from the most under-served queue. Is this what 
happens?

5) According to the example presented in "Apache Hadoop YARN: Moving 
beyond MapReduce and Batch Processing with Apache Hadoop 2" about the 
resource allocation with the capacity scheduler, what I understood is 
that the chance for a leaf queue to get resources above it's assigned 
capacity is always upper-limited by the fraction of cluster capacity 
assigned to its first/closer parent queue. That is: if I am a leaf queue 
A1, I can only get at most the resources dedicated to my parent A, while 
I can't get the ones from B, sibling of A, even if it doesn't have any 
running application. Actually at first I thought this over-provisioning 
was not limited, and regardless of the queue configuration a single 
application could get the whole cluster (excluding per-application 
limits). Did I misunderstood the example?

Thanks a lot

Fabio

Re: Questions about Capacity scheduler behavior

[Wangda Tan <wh...@gmail.com>]

Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.

2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always

3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.

4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?

5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.

root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90

The guaranteed resource of a.a1 is a.capacity% *  a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).

And as you said, a resource of child can use, will always <= resource of
its parent.

If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.

Does this make sense to you?

Thanks,
Wangda

On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:

> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>

Re: Questions about Capacity scheduler behavior

[Wangda Tan <wh...@gmail.com>]

Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.

2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always

3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.

4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?

5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.

root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90

The guaranteed resource of a.a1 is a.capacity% *  a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).

And as you said, a resource of child can use, will always <= resource of
its parent.

If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.

Does this make sense to you?

Thanks,
Wangda

On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:

> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>

Re: Questions about Capacity scheduler behavior

[Wangda Tan <wh...@gmail.com>]

Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.

2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always

3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.

4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?

5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.

root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90

The guaranteed resource of a.a1 is a.capacity% *  a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).

And as you said, a resource of child can use, will always <= resource of
its parent.

If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.

Does this make sense to you?

Thanks,
Wangda

On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:

> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>

Re: Questions about Capacity scheduler behavior

[Wangda Tan <wh...@gmail.com>]

Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.

2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always

3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.

4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?

5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.

root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90

The guaranteed resource of a.a1 is a.capacity% *  a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).

And as you said, a resource of child can use, will always <= resource of
its parent.

If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.

Does this make sense to you?

Thanks,
Wangda

On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:

> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>