You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Fabio <an...@gmail.com> on 2014/11/03 15:30:43 UTC
Questions about Capacity scheduler behavior
Hi guys, I'm posting this in the user mailing list since I got no reply
in the Yarn-dev. I have to model as well as possible the capacity
scheduler behavior and I have some questions, hope someone can help me
with this. In the following I will consider all containers to be equal
for simplicity:
1) In case of multiple queues having the same level of assigned
resources, what's the policy to decide which comes first in the resource
allocation?
2) Let's consider this configuration:
We have a cluster hosting a total of 40 containers. We have 3 queues: A
is configured to get 39% of cluster capacity, B also gets 39% and C gets
22%. The number of containers is going to be 15.6, 15.6 and 8.8 for A, B
an C. Since we can't split a container, how does the Capacity scheduler
round these values in a real case? Who gets the two contended
containers? I may think they are considered as extra containers, thus
shared upon need among the three queues. Is this correct?
3) Let's say I have queues A and B. A is configured to get 20% (20
containers) of the total cluster capacity (100 containers), B gets 80%
(80 containers). Capacity scheduler gives available resources firstly to
the most under-served queue.
In case A is using 10 containers and B is using 20, who is going to get
the first available container? A is already using 50% of it's assigned
capacity, B just 25%, but A has less containers than B... who is
considered to be more under-served?
4) Does the previous question make sense at all? Because I have a doubt
that when I have free containers I will just serve requests as they
arrive, possibly over-provisioning a queue (that is: if I get a
container request for an app in A, I will give it a container since I
don't know that after a few milliseconds I will get a new request from
B, or vice versa). The previous question may have sense if there was
some sort of buffer that is filled with incoming requests, due to the
difficulty of serving them in real time, thus making the scheduler able
to choose the request from the most under-served queue. Is this what
happens?
5) According to the example presented in "Apache Hadoop YARN: Moving
beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
resource allocation with the capacity scheduler, what I understood is
that the chance for a leaf queue to get resources above it's assigned
capacity is always upper-limited by the fraction of cluster capacity
assigned to its first/closer parent queue. That is: if I am a leaf queue
A1, I can only get at most the resources dedicated to my parent A, while
I can't get the ones from B, sibling of A, even if it doesn't have any
running application. Actually at first I thought this over-provisioning
was not limited, and regardless of the queue configuration a single
application could get the whole cluster (excluding per-application
limits). Did I misunderstood the example?
Thanks a lot
Fabio
Re: Questions about Capacity scheduler behavior
Posted by Wangda Tan <wh...@gmail.com>.
Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.
2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always
3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.
4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?
5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.
root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90
The guaranteed resource of a.a1 is a.capacity% * a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).
And as you said, a resource of child can use, will always <= resource of
its parent.
If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.
Does this make sense to you?
Thanks,
Wangda
On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:
> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>
Re: Questions about Capacity scheduler behavior
Posted by Wangda Tan <wh...@gmail.com>.
Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.
2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always
3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.
4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?
5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.
root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90
The guaranteed resource of a.a1 is a.capacity% * a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).
And as you said, a resource of child can use, will always <= resource of
its parent.
If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.
Does this make sense to you?
Thanks,
Wangda
On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:
> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>
Re: Questions about Capacity scheduler behavior
Posted by Wangda Tan <wh...@gmail.com>.
Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.
2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always
3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.
4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?
5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.
root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90
The guaranteed resource of a.a1 is a.capacity% * a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).
And as you said, a resource of child can use, will always <= resource of
its parent.
If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.
Does this make sense to you?
Thanks,
Wangda
On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:
> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>
Re: Questions about Capacity scheduler behavior
Posted by Wangda Tan <wh...@gmail.com>.
Hi Fabio,
To answer your questions:
1)
CS (capacity-scheduler) will do allocation from root to leaf, level by
level, and for queues with same parent, will do allocation in following way:
First will allocate queue with least used resource
If queues with same used resource, say root.qA used 1G, root.qB used 1G as
well, it will allocate resource by name of queue, so qA will go first.
2)
There're two parameters about capacity, one is capacity, another one is
maximum-capacity, queue can only allocate resource with multiple of
minimum-allocation of the cluster, and it will be <= maximum-capacity of
the queue always
3)
I think 1) should already answer your question, queue will go first by used
resource, not percentage of used resource, so in your example, queue-A will
go first.
4)
Sorry, I'm not quite understand this question, could you do some
explanations about it?
5)
You were not misunderstand this example, this is what
capacity/maximum-capacity applied on queues. The maximum-capacity of queue
is used to do such "resource provisioning" for queues.
root.a.capacity = 50
root.a.maximum-capacity = 80
root.a.a1.capacity = 50
root.a.a1.maximum-capacity = 90
The guaranteed resource of a.a1 is a.capacity% * a.a1.capacity% = 25%
The maximum resource of a.a1 is a.maximum-capacity% * a.a1.maximum-capacity
= 72% (0.8 * 0.9).
And as you said, a resource of child can use, will always <= resource of
its parent.
If you want to let all leafQueues can leverage all resource in the cluster,
you can simply set all maximum-capacity of parent queues and leaf queues to
100.
Does this make sense to you?
Thanks,
Wangda
On Mon, Nov 3, 2014 at 6:30 AM, Fabio <an...@gmail.com> wrote:
> Hi guys, I'm posting this in the user mailing list since I got no reply in
> the Yarn-dev. I have to model as well as possible the capacity scheduler
> behavior and I have some questions, hope someone can help me with this. In
> the following I will consider all containers to be equal for simplicity:
>
> 1) In case of multiple queues having the same level of assigned resources,
> what's the policy to decide which comes first in the resource allocation?
>
> 2) Let's consider this configuration:
> We have a cluster hosting a total of 40 containers. We have 3 queues: A is
> configured to get 39% of cluster capacity, B also gets 39% and C gets 22%.
> The number of containers is going to be 15.6, 15.6 and 8.8 for A, B an C.
> Since we can't split a container, how does the Capacity scheduler round
> these values in a real case? Who gets the two contended containers? I may
> think they are considered as extra containers, thus shared upon need among
> the three queues. Is this correct?
>
> 3) Let's say I have queues A and B. A is configured to get 20% (20
> containers) of the total cluster capacity (100 containers), B gets 80% (80
> containers). Capacity scheduler gives available resources firstly to the
> most under-served queue.
> In case A is using 10 containers and B is using 20, who is going to get
> the first available container? A is already using 50% of it's assigned
> capacity, B just 25%, but A has less containers than B... who is considered
> to be more under-served?
>
> 4) Does the previous question make sense at all? Because I have a doubt
> that when I have free containers I will just serve requests as they arrive,
> possibly over-provisioning a queue (that is: if I get a container request
> for an app in A, I will give it a container since I don't know that after a
> few milliseconds I will get a new request from B, or vice versa). The
> previous question may have sense if there was some sort of buffer that is
> filled with incoming requests, due to the difficulty of serving them in
> real time, thus making the scheduler able to choose the request from the
> most under-served queue. Is this what happens?
>
> 5) According to the example presented in "Apache Hadoop YARN: Moving
> beyond MapReduce and Batch Processing with Apache Hadoop 2" about the
> resource allocation with the capacity scheduler, what I understood is that
> the chance for a leaf queue to get resources above it's assigned capacity
> is always upper-limited by the fraction of cluster capacity assigned to its
> first/closer parent queue. That is: if I am a leaf queue A1, I can only get
> at most the resources dedicated to my parent A, while I can't get the ones
> from B, sibling of A, even if it doesn't have any running application.
> Actually at first I thought this over-provisioning was not limited, and
> regardless of the queue configuration a single application could get the
> whole cluster (excluding per-application limits). Did I misunderstood the
> example?
>
> Thanks a lot
>
> Fabio
>